Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR 早期判决，更优预算：用于计算高效RLVR的顺序自适应轮次分配

TL;DR

• The paper introduces SARA (Sequential Adaptive Rollout Allocation), a method to improve compute efficiency in Reinforcement Learning with Verifiable Rewards (RLVR) by dynamically allocating rollout budgets.
• SARA addresses the inefficiency of generating rollouts for prompts that are already saturated (all correct or all incorrect responses) by using a Beta posterior and sequential decision rules to abandon uninformative groups early.
• Compared to Dynamic Sampling (DS), SARA achieves similar performance while using 22% fewer rollouts, and when combined with DPS, it outperforms DS with 67% fewer rollouts.
• The method is validated on mathematical reasoning and planning tasks using 1.5B/3B models on a single GPU, demonstrating its practicality for resource-constrained settings.

Why It Matters

This work directly tackles a critical bottleneck in RLVR—inefficient rollout generation—which limits scalability and adoption in real-world applications. By optimizing budget allocation without sacrificing performance, SARA enables more cost-effective training of large language models for verifiable reward settings, making it highly relevant for practitioners aiming to balance accuracy and computational expense.

Technical Details

• Problem Formulation: The paper frames rollout allocation as a budget-consequential optimal stopping problem, where each prompt group’s effectiveness is assessed sequentially rather than fixed-length sampling.
• Beta Posterior Tracking: SARA maintains a Beta distribution over the success rate of each prompt group, updated incrementally after each rollout to reflect observed outcomes.
• Two-Threshold Decision Rule: Inspired by Sequential Probability Ratio Testing (SPRT), the method uses lower and upper thresholds to decide whether to abandon (low success probability), commit (high success probability), or continue sampling a group.
• Closed-Form Effectiveness Predictor: A computationally efficient metric evaluates whether continuing a group will yield meaningful policy-gradient signals based on current posterior estimates.
• Integration with DPS: When combined with Dynamic Prompt Selection (DPS), SARA further refines candidate pool selection, achieving near-oracle performance at drastically reduced rollout counts.

Industry Insight

Adopting adaptive rollout strategies like SARA can significantly reduce operational costs in RL-based fine-tuning pipelines, especially for teams working under tight GPU budget constraints. This approach encourages a shift from static sampling protocols toward intelligent, real-time resource management—potentially becoming standard practice in production-grade RLHF systems focused on verifiable tasks such as code generation or logical reasoning.

TL;DR

• 提出SARA算法，通过早期停止和预算重分配解决RLVR中因饱和组（全对或全错）导致的梯度信号缺失问题。
• SARA利用Beta后验分布与SPRT风格的双阈值规则，在无需额外预测rollouts的情况下动态调整采样策略。
• 实验表明SARA在数学推理与规划任务中，以22%更少的rollouts达到与动态采样(DPS)相当的性能，组合使用可节省67%成本并提升精度。
• 理论证明包括放弃可靠性、期望rollout节省量、固定预算下的产出优势及有效组产出与GRPO梯度范数的关联。
• 该方法为计算受限环境下的强化学习训练提供高效、可扩展的采样优化方案。

为什么值得看

本文针对RLVR训练中常见的“饱和prompt”浪费计算资源的问题，提出了一种无需预训练难度预测器的高效自适应采样机制，显著降低训练开销同时保持甚至提升模型性能，对工业界部署大模型强化学习具有重要实践价值。其核心思想——基于序贯假设检验的动态预算重分配——可推广至其他高成本采样场景，如多智能体仿真或长序列生成任务。

技术解析

• SARA将每个prompt的success rate建模为Beta分布的后验，实时更新成功/失败计数，避免显式预测prompt难度。
• 采用双阈值SPRT规则：当某prompt的成功率置信区间完全高于上阈则commit；若完全低于下阈则abandon并释放预算；否则继续采样直至达到最大步数或触发阈值。
• 放弃的prompt被立即替换为新prompt，实现滚动式预算再利用，确保总rollout数恒定且集中于高信息量样本。
• 理论分析证明：在固定预算下，SARA的有效组产出严格优于随机采样；且其梯度估计方差更小，收敛更快。
• 在1.5B/3B模型上的单GPU实验中，SARA单独使用时比动态采样少用22% rollouts而性能持平；与DPS联合使用时仅需67% rollouts即超越动态采样上限，接近oracle最优解。

行业启示

• 对于资源敏感的AI团队，应优先采用类似SARA的自适应采样策略替代传统均匀采样或静态难度预估方法，以降低训练成本并加速迭代周期。
• RLVR系统架构设计应内嵌此类在线预算管理机制，而非作为事后优化步骤，从而在训练初期即避免无效计算。
• 未来工作可探索将该框架扩展至非二值奖励场景（如连续评分或多目标权衡），并结合元学习自动调节阈值参数，进一步提升泛化能力。