Probing the Origins of Reasoning Performance: Representational Quality for Mathematical Problem-Solving in RL vs. SFT Fine-Tuned Models 探究推理表现的起源：RL与SFT微调模型在数学问题求解中的表征质量

TL;DR

• RL-trained models develop more linearly separable and structured internal representations for mathematical reasoning compared to SFT models, as evidenced by higher accuracy in linear probes predicting answer correctness.
• RL training induces a hierarchical architecture where deeper layers become progressively more critical for reasoning tasks, whereas SFT models distribute importance uniformly across layers.
• Token-count variability under repeated sampling reveals differences in adaptive compute allocation, with some RL-tuned models showing higher variability than SFT counterparts, suggesting token allocation depends more on the overall training pipeline than on RL versus SFT alone.

Why It Matters

This research provides crucial insights into why reinforcement learning (RL) often outperforms supervised fine-tuning (SFT) in mathematical reasoning tasks. By uncovering representational differences and architectural changes induced by RL training, it offers a mechanistic understanding that can guide future model development and optimization strategies for reasoning-intensive applications.

Technical Details

• Linear Probes: Trained on layer-wise hidden states to predict answer correctness, revealing that RL models achieve higher accuracy, indicating more structured and linearly separable representations.
• Mean Ablation Studies: Show that RL models develop a hierarchical architecture with deeper layers becoming progressively more critical, while SFT models distribute importance uniformly across layers.
• Token-Count Variability Analysis: Assesses adaptive compute allocation under repeated sampling across problems, highlighting differences in token allocation between RL and SFT models, which may depend more on the overall training pipeline than on the specific fine-tuning method.

Industry Insight

• Model Fine-Tuning Strategies: The findings suggest that RL fine-tuning should be prioritized for tasks requiring high-level reasoning, as it leads to more structured and efficient internal representations.
• Architecture Design: Understanding the hierarchical nature of RL-trained models can inform the design of architectures that better leverage deep-layer processing for complex reasoning tasks.
• Compute Optimization: Insights into token allocation variability can help optimize computational resources during inference, ensuring stable and reliable performance across different problem types.

TL;DR

• 强化学习（RL）微调的数学推理模型在表现上优于监督微调（SFT）模型，但其内部机制尚不明确。
• 线性探针分析显示，RL模型的隐藏状态具有更高的线性可分性和结构化表示，能更准确地预测答案正确性。
• 平均消融研究表明，RL模型形成层级架构，深层对推理贡献更大，而SFT模型各层重要性分布均匀。
• RL训练从根本上重塑了模型处理推理问题的表征方式。
• 令牌计数变异性分析表明，自适应计算分配可能取决于整体训练管道而非单纯的RL或SFT方法。

为什么值得看

该研究揭示了RL与SFT微调模型在数学推理任务中的性能差异背后的内在机制，为理解大语言模型的推理能力提供了重要视角。对于AI从业者而言，这有助于优化模型训练策略，提升复杂任务的表现。

技术解析

1. 线性探针分析：通过在逐层隐藏状态上训练线性探针，发现RL模型比SFT模型更能准确预测答案的正确性，表明其表征更具线性可分性和结构性。
2. 平均消融研究：通过移除不同层的输出并评估性能变化，结果显示RL模型中深层对推理更为关键，而SFT模型则表现出均匀的重要性分布。
3. 令牌计数变异性：通过对同一问题进行多次采样分析令牌数量的变化，观察到某些RL模型存在较高的变异性，而其他模型则保持稳定，暗示令牌分配受整体训练流程影响较大。
4. 数据集与基准测试：虽然具体使用的数据集未详细列出，但研究聚焦于数学推理任务的评估，可能涉及标准基准如GSM8K等。
5. 实现细节：论文未提供具体的模型规格或实验设置细节，但从上下文推测，比较对象应为同规模下的RL和SFT版本模型。

行业启示

1. 强化学习的优势：对于需要高度逻辑推理的任务，采用RL微调可能是提升模型性能的有效途径，尤其是在构建专业领域应用时。
2. 表征质量的重要性：提高模型内部的表征质量和结构清晰度可以显著增强其在复杂任务上的表现，未来工作可探索如何进一步优化这一方面。
3. 灵活的计算资源管理：根据问题动态调整计算资源（如生成步骤数）的策略值得进一步研究，以实现效率与效果的最佳平衡。