Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learning
Error-penalized scoring rules (+1 correct, -λ wrong, 0 abstain) are widely used to combat hallucination, but when abstention is treated as a discrete action in KL-anchored RL, both the reward gradient and the KL anchor's restoring force are simultaneously throttled by a shared gate-saturation factor, causing catastrophic collapse. Under conditions where blanket answering loses expected score and prompts share a bounded readout, the model drifts toward refusing everything; mean training reward ri
Analysis
TL;DR
- Error-penalized scoring rules (+1 correct, -λ wrong, 0 abstain) are widely used to combat hallucination, but when abstention is treated as a discrete action in KL-anchored RL, both the reward gradient and the KL anchor's restoring force are simultaneously throttled by a shared gate-saturation factor, causing catastrophic collapse.
- Under conditions where blanket answering loses expected score and prompts share a bounded readout, the model drifts toward refusing everything; mean training reward rises toward zero like 1/t while coverage collapses, creating a misleadingly improving reward curve.
- The advantage estimator with group normalization silently replaces every designed penalty with an effective penalty of one, shifting the learned abstention threshold from the theoretically optimal Chow's threshold t* = λ/(1+λ) to 1/2.
- The proposed repair is structural: train a mandatory confidence report using a strictly proper scoring rule plus a correctness reward, and abstain only at deployment by thresholding that report, eliminating the saturable gate entirely.
- Simulations confirm all theoretical predictions, and experiments on language models at two scales demonstrate that the error-penalized rule silences solvable questions within ten optimizer steps, while report-level training simultaneously improves coverage, accuracy, and calibration.
Why It Matters
This work reveals a fundamental flaw in a widely adopted anti-hallucination strategy—error-penalized RL with abstention as an action—showing that it can produce deceptively improving reward curves while the model silently stops answering. For AI practitioners building safety-critical systems, the findings mean that standard reward shaping alone is insufficient and can actively degrade coverage; the proposed confidence-report architecture offers a principled, structurally sound alternative that preserves both performance and calibration.
Technical Details
- Collapse Law: The paper proves that when abstention is a discrete action in KL-anchored policy gradient learning, the reward gradient and the KL anchor's restoring force share an identical gate-saturation factor. When this factor approaches zero (due to high abstention rates), both vanish simultaneously, causing the model to drift toward universal refusal.
- Threshold Distortion via Group Normalization: In the sparse-answer regime, group normalization in the advantage estimator effectively replaces every designed penalty λ with an implicit penalty of 1, shifting the learned optimal threshold from Chow's t* = λ/(1+λ) to 1/2 regardless of the intended penalty schedule.
- Repair Architecture: The fix decouples training from deployment by introducing a mandatory confidence report head trained with a strictly proper scoring rule (e.g., log score) plus a correctness reward. Abstention is applied only at inference time by thresholding this report, ensuring the training signal is never gated.
- Empirical Validation: Experiments on language models at two scales show that error-penalized RL silences questions the model can still solve within ten optimizer steps. An ablation isolates the gate-saturation mechanism as the root cause, and report-level training raises coverage, accuracy, and calibration metrics together.
Industry Insight
- Reward curve vigilance: Practitioners should treat rising mean reward under error-penalized abstention with suspicion; the paper demonstrates this can signal coverage collapse rather than genuine improvement. Monitoring coverage and per-question accuracy alongside aggregate reward is essential.
- Prefer structural over scalar fixes: Simply tuning the penalty parameter λ will not resolve the collapse— the failure is architectural. Teams investing in hallucination mitigation should adopt confidence-report-based training with deployment-time thresholding rather than discrete abstention actions during RL training.
- Group normalization is a silent confounder: The finding that group normalization in advantage estimation silently distorts penalty schedules has broad implications beyond this specific setting; any RL pipeline using group-normalized advantages with sparse positive samples should audit whether effective penalties match design intent.
Disclaimer: The above content is generated by AI and is for reference only.