Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models RLVR中的奖励粒度：比较小语言模型数学推理中的过程和结果奖励结构

TL;DR

• Process-only reinforcement learning yields a ~10 percentage point accuracy gain over outcome-only rewards for small language models (Qwen2.5-0.5B) on GSM8K.
• Hybrid reward structures generally benefit from higher process weights, though extreme outcome bias ($\lambda=0.1$) causes performance degradation due to conflicting optimization signals.
• Error analysis reveals distinct failure modes: process-supervised models produce structurally inconsistent but arithmetically correct traces, whereas outcome-supervised models generate concise but derivation-error-prone chains.
• Reward granularity is identified as a critical design parameter for RLVR, particularly for small models lacking self-correction capabilities under sparse feedback.

Why It Matters

This study challenges the prevailing assumption that final-answer verification is sufficient for training small language models, demonstrating that dense, step-level supervision is essential for improving reasoning fidelity. For practitioners working with resource-constrained models, it provides empirical evidence that investing in process rewards can significantly outperform traditional outcome-only approaches. The findings also highlight the importance of balancing reward signals in hybrid setups to avoid optimization conflicts that degrade performance.

Technical Details

• Model and Method: The study utilizes Qwen2.5-0.5B fine-tuned with Group Relative Policy Optimization (GRPO) on the GSM8K dataset.
• Experimental Setup: Five reward conditions were compared: no-RL baseline, process-only, outcome-only, and three hybrid configurations with process weights $\lambda \in {0.9, 0.5, 0.1}$.
• Performance Metrics: Process-only supervision achieved 63.73% test accuracy compared to 53.75% for outcome-only, with higher step validity and lower deviation from ground-truth chain lengths.
• Error Analysis: Utilized GPT-4o as a judge to categorize failures, distinguishing between structural inconsistencies in process models and derivation errors in outcome models.

Industry Insight

AI developers should prioritize process-level reward modeling when fine-tuning small language models for complex reasoning tasks, as sparse outcome feedback may be insufficient for effective learning. When designing hybrid reward functions, careful tuning of the process-to-outcome ratio is necessary; overly emphasizing outcomes can introduce conflicting gradients that harm performance more than pure outcome supervision. Future research into RLVR for small models must account for the specific failure modes introduced by different reward granularities to optimize trace quality and final accuracy simultaneously.

TL;DR

• 研究对比了过程奖励与结果奖励在小型语言模型数学推理中的效果，发现过程奖励显著优于结果奖励。
• 在Qwen2.5-0.5B模型上使用GRPO算法进行实验，过程奖励仅监督达到63.73%准确率，比结果奖励高近10个百分点。
• 混合奖励策略中，低过程权重配置（$\lambda=0.1$）表现甚至低于纯结果奖励，表明存在冲突的优化信号。
• 错误分析显示，过程奖励模型生成的推理链结构不一致但算术基础扎实，而结果奖励模型推导易出错。
• 结论强调奖励粒度是RLVR设计的一阶决策，过程级监督能大幅提升小模型的准确性和推理保真度。

为什么值得看

本文揭示了在资源受限的小型语言模型中，细粒度的过程监督对于强化学习微调的关键作用，挑战了仅依赖最终答案奖励的主流做法。它为开发者优化小模型推理能力提供了具体的超参数调整方向和理论依据，特别是在处理复杂逻辑任务时。

技术解析

• 实验设置：使用Qwen2.5-0.5B作为基座模型，采用Group Relative Policy Optimization (GRPO)算法，在GSM8K数据集上进行数学推理能力的微调。
• 奖励机制对比：系统比较了五种奖励条件：无RL基线、纯过程奖励、纯结果奖励，以及三种不同过程权重（$\lambda \in {0.9, 0.5, 0.1}$）的混合奖励。
• 性能差异：纯过程奖励获得63.73%的测试准确率，显著高于纯结果奖励的53.75%。过程奖励模型生成的推理步骤有效性更高，且与真实链长度偏差更小。
• 异常发现：当过程权重极低（$\lambda=0.1$）时，模型性能反而不如纯结果奖励，暗示强烈的结果导向可能干扰过程学习的稳定性。
• 错误模式分析：利用GPT-4o作为裁判进行分析，发现过程奖励模型倾向于产生算术正确但结构混乱的推理，而结果奖励模型则产生简洁但包含推导错误的链条。

行业启示

• 重视过程监督：在训练小型或中等规模模型时，应优先考虑引入步骤级的过程奖励，而非仅仅关注最终答案的正确性，以弥补小模型自我纠错能力的不足。
• 谨慎设计混合奖励：混合奖励策略并非线性相关，需仔细平衡过程与结果的权重，避免低过程权重导致的优化信号冲突，从而损害整体性能。
• 优化推理链质量：除了准确率，还应关注推理链的结构完整性和算术一致性，过程奖励有助于生成更可靠、可解释的推理轨迹，这对需要高可信度的应用场景至关重要。