Verifier-Induced Support Reshaping in On-Policy Optimization
On-policy RL with verifiable rewards (RLVR) can optimize the current objective while simultaneously making successful behaviors for other objectives too rare to sample, a phenomenon termed "verifier-induced support reshaping" Math-RLVR improves instruction-following pass@1 by 6.5pp on IFEval with Qwen3-8B-Base but reduces best@32 by 9.8pp, demonstrating a critical divergence between single-sample and multi-sample performance IF-RLVR shifts mathematical reasoning from step-by-step openings toward
Analysis
TL;DR
- On-policy RL with verifiable rewards (RLVR) can optimize the current objective while simultaneously making successful behaviors for other objectives too rare to sample, a phenomenon termed "verifier-induced support reshaping"
- Math-RLVR improves instruction-following pass@1 by 6.5pp on IFEval with Qwen3-8B-Base but reduces best@32 by 9.8pp, demonstrating a critical divergence between single-sample and multi-sample performance
- IF-RLVR shifts mathematical reasoning from step-by-step openings toward direct answers, reducing reward variation and lowering best@k across sampling budgets
- Token-distribution and intervention analyses reveal these effects concentrate in the first few response tokens, with RLVR primarily reranking existing base-policy openings rather than generating new ones
- Reference-policy constraints, routing priors, and on-policy distillation only partially preserve cross-task support, proving that endpoint improvements do not guarantee future trainability or joint capability
Why It Matters
This research reveals a fundamental tension in on-policy RLVR training: optimizing for one capability can actively degrade the model's ability to learn other capabilities by shrinking the effective rewardable support. For practitioners running multi-objective or sequential RLVR pipelines, this means endpoint gains on one benchmark may mask deteriorating trainability on others, requiring careful monitoring of best@k and support diversity rather than relying solely on pass@1 improvements.
Technical Details
- Verifier-induced support reshaping: Defined as the phenomenon where on-policy RLVR improves the current objective while making successful trajectories for later objectives too rare to sample within a fixed rollout budget; effective rewardable support is quantified as successful trajectories reachable within that budget
- Bidirectional training experiments: Conducted across two model families on mathematical reasoning and constrained instruction following, including sequential training where one verifier is applied after the other (Math-RLVR then IF-RLVR and vice versa)
- Key empirical findings: On IFEval with Qwen3-8B-Base, Math-RLVR yields pass@1 +6.5pp but best@32 -9.8pp; IF-RLVR shifts math responses from step-by-step to direct answers and reduces reward variation for subsequent Math-RLVR
- Causal mechanism: Token-distribution analyses and controlled opening interventions show changes concentrate in the first few response tokens; RLVR mainly reranks openings already present in the base policy, and the selected opening causally affects math searchability
- Mitigation attempts: Reference-policy constraints, routing priors, and on-policy distillation were tested but only partially preserved cross-task support; composite benchmarks MathIF and ReasonIF showed marginal gains translate only partly into responses that are both correct and constraint-following
Industry Insight
- Multi-objective RLVR pipelines should monitor best@k and support diversity metrics alongside pass@1, as single-sample improvements can mask shrinking explorability for downstream objectives
- Sequential training strategies that optimize one capability before another risk irreversible support collapse; joint or interleaved optimization with support-preserving regularizers may be necessary for robust multi-task RLVR
- The finding that RLVR primarily reranks existing openings rather than creating new ones suggests that base model diversity and initial rollout budgets are critical bottlenecks—investing in broader pre-training coverage and larger sampling budgets may yield better returns than aggressive on-policy optimization alone
Disclaimer: The above content is generated by AI and is for reference only.