When RLVR Shrinks the Reasoning Boundary: Diagnosing Pass@k Inversion
RLVR training can cause "pass@k inversion," where models improve single-sample accuracy but fail to solve distinct problems under high sampling budgets compared to base models. The failure stems from "boundary prompts" where rare correct trajectories in the base model are lost because they are too sparse to be reinforced during finite RLVR rollouts. Per-Problem Base Anchoring (PBA) is introduced as a solution, anchoring risky prompts to the base distribution to preserve rare verifier-positive tr
Analysis
TL;DR
- RLVR training can cause "pass@k inversion," where models improve single-sample accuracy but fail to solve distinct problems under high sampling budgets compared to base models.
- The failure stems from "boundary prompts" where rare correct trajectories in the base model are lost because they are too sparse to be reinforced during finite RLVR rollouts.
- Per-Problem Base Anchoring (PBA) is introduced as a solution, anchoring risky prompts to the base distribution to preserve rare verifier-positive trajectories.
- Empirical results on Omni-MATH-Test and MATH500 show PBA outperforms standard GRPO in both Pass@1 accuracy and high-budget coverage across multiple seeds.
- The phenomenon extends beyond text-based math reasoning to vision-language agents using external verifiers for visual or spatial tasks.
Why It Matters
This research highlights a critical trade-off in post-training reinforcement learning: optimizing for single-sample correctness can inadvertently degrade robustness and coverage. For practitioners, it underscores the necessity of evaluating models not just on average accuracy, but on their ability to recover solutions through repeated sampling, particularly for complex or edge-case problems.
Technical Details
- Phenomenon Diagnosis: The paper identifies "pass@k inversion" as an absence-of-evidence failure, where rare correct paths in the base policy disappear before RLVR can sample and reinforce them sufficiently.
- Methodology (PBA): Per-Problem Base Anchoring involves sharpening prompts with frozen-base correct evidence and anchoring risky prompts to the base distribution to prevent the loss of sparse correct trajectories.
- Benchmarks & Setup: Evaluated on Omni-MATH-Test with MATH500 as a secondary validation set, using mathematical verifiers as a controlled testbed for verifier-guided optimization.
- Experimental Results: A 3000-prompt regime-controlled study across three training seeds demonstrated that ordinary GRPO loses base-solvable boundary prompts, whereas PBA preserves them and improves overall coverage.
- Generalizability: The diagnostic framework applies to other domains like ECCV-relevant vision-language agents, where repeated attempts are checked by external tools or verifiers.
Industry Insight
- Evaluation Protocols: AI teams should adopt multi-sample evaluation metrics (Pass@k) alongside standard Pass@1 to detect regressions in model robustness and coverage during RLVR fine-tuning.
- Training Strategies: When deploying RLVR, consider hybrid approaches like PBA that anchor to pre-trained capabilities for difficult or rare cases, rather than relying solely on reward maximization which may erase valuable but sparse knowledge.
- Risk Management: Post-training optimization decisions must include a safety assessment of which prompts are "safe" to optimize, ensuring that rare but valid reasoning paths are not inadvertently pruned from the policy.
Disclaimer: The above content is generated by AI and is for reference only.