ERR+: Sequential Entropy Resolution for Efficient and Decisive LLM Reasoning
ERR+ is a two-phase RLVR framework that optimizes the internal reasoning structure of large reasoning models by rewarding entropy resolution during the thinking phase The Entropy Relief Reward (ERR) provides a bonus proportional to cumulative token-level entropy drops, log-normalized by response length, rewarding uncertainty resolution without suppressing exploration The Robust Relative Efficiency Reward uses a tanh-transformed within-group z-score to penalize excessive response length relative
Analysis
TL;DR
- ERR+ is a two-phase RLVR framework that optimizes the internal reasoning structure of large reasoning models by rewarding entropy resolution during the thinking phase
- The Entropy Relief Reward (ERR) provides a bonus proportional to cumulative token-level entropy drops, log-normalized by response length, rewarding uncertainty resolution without suppressing exploration
- The Robust Relative Efficiency Reward uses a tanh-transformed within-group z-score to penalize excessive response length relative to peers, improving conciseness
- Formal analysis reveals gradient conflict between the two objectives in early training, motivating the sequential (rather than joint) optimization design
- Experiments across five datasets show consistent accuracy and conciseness improvements across multiple model backbones
Why It Matters
This work addresses a critical gap in RLVR: while correctness-based rewards optimize final answers, they neglect the quality of the reasoning process itself. By introducing entropy-based process rewards, ERR+ provides a principled way to shape how models think, not just what they conclude. This is directly relevant to anyone building or fine-tuning large reasoning models, as it offers a method to achieve better performance with shorter, more decisive reasoning traces.
Technical Details
- Entropy Relief Reward (ERR): Computes cumulative token-level entropy drops during the thinking phase; rewards resolution of uncertainty (entropy decrease) while leaving exploratory high-entropy states unconstrained, unlike prior methods that suppress entropy entirely. The reward is log-normalized by response length to prevent length bias.
- Robust Relative Efficiency Reward: Scores each response's length against co-generated peers using a tanh-transformed within-group z-score, creating a bounded, robust signal that penalizes unnecessarily long responses without absolute length cutoffs.
- Sequential Two-Phase Design: Formal gradient analysis shows joint optimization of ERR and efficiency rewards induces conflict in early training; the sequential approach (ERR first, then efficiency) avoids this instability.
- Empirical Validation: Tested across five datasets and multiple model families, demonstrating consistent improvements in both accuracy and response conciseness.
- Key Observation: Correct reasoning traces exhibit more frequent and larger token-level entropy drops within the thinking phase compared to incorrect traces, forming the empirical foundation of the approach.
Industry Insight
- Entropy-based process rewards represent an underexplored direction for improving LLM reasoning quality beyond outcome supervision; practitioners should consider monitoring token-level entropy as a diagnostic and training signal.
- The sequential optimization strategy (rather than joint) highlights the importance of understanding gradient interactions in multi-objective RLVR, suggesting that reward scheduling may be as important as reward design.
- The conciseness improvements without accuracy loss are particularly valuable for production deployment, where latency and token costs are significant constraints.
Disclaimer: The above content is generated by AI and is for reference only.