Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models
Process-only reinforcement learning yields a ~10 percentage point accuracy gain over outcome-only rewards for small language models (Qwen2.5-0.5B) on GSM8K. Hybrid reward structures generally benefit from higher process weights, though extreme outcome bias ($\lambda=0.1$) causes performance degradation due to conflicting optimization signals. Error analysis reveals distinct failure modes: process-supervised models produce structurally inconsistent but arithmetically correct traces, whereas outco
Analysis
TL;DR
- Process-only reinforcement learning yields a ~10 percentage point accuracy gain over outcome-only rewards for small language models (Qwen2.5-0.5B) on GSM8K.
- Hybrid reward structures generally benefit from higher process weights, though extreme outcome bias ($\lambda=0.1$) causes performance degradation due to conflicting optimization signals.
- Error analysis reveals distinct failure modes: process-supervised models produce structurally inconsistent but arithmetically correct traces, whereas outcome-supervised models generate concise but derivation-error-prone chains.
- Reward granularity is identified as a critical design parameter for RLVR, particularly for small models lacking self-correction capabilities under sparse feedback.
Why It Matters
This study challenges the prevailing assumption that final-answer verification is sufficient for training small language models, demonstrating that dense, step-level supervision is essential for improving reasoning fidelity. For practitioners working with resource-constrained models, it provides empirical evidence that investing in process rewards can significantly outperform traditional outcome-only approaches. The findings also highlight the importance of balancing reward signals in hybrid setups to avoid optimization conflicts that degrade performance.
Technical Details
- Model and Method: The study utilizes Qwen2.5-0.5B fine-tuned with Group Relative Policy Optimization (GRPO) on the GSM8K dataset.
- Experimental Setup: Five reward conditions were compared: no-RL baseline, process-only, outcome-only, and three hybrid configurations with process weights $\lambda \in {0.9, 0.5, 0.1}$.
- Performance Metrics: Process-only supervision achieved 63.73% test accuracy compared to 53.75% for outcome-only, with higher step validity and lower deviation from ground-truth chain lengths.
- Error Analysis: Utilized GPT-4o as a judge to categorize failures, distinguishing between structural inconsistencies in process models and derivation errors in outcome models.
Industry Insight
AI developers should prioritize process-level reward modeling when fine-tuning small language models for complex reasoning tasks, as sparse outcome feedback may be insufficient for effective learning. When designing hybrid reward functions, careful tuning of the process-to-outcome ratio is necessary; overly emphasizing outcomes can introduce conflicting gradients that harm performance more than pure outcome supervision. Future research into RLVR for small models must account for the specific failure modes introduced by different reward granularities to optimize trace quality and final accuracy simultaneously.
Disclaimer: The above content is generated by AI and is for reference only.