SMOPD: Selective Token-Entropy Masking for Dirty-History Multi-Turn On-Policy Self-Distillation SMOPD：面向脏历史多轮策略内自蒸馏的选择性Token熵掩码

TL;DR

• SMOPD introduces a loss-only stabilization method for multi-turn on-policy self-distillation (OPSD) that addresses brittleness caused by dirty-history rollouts where erroneous intermediate replies propagate through subsequent turns
• The method ranks token positions by student entropy and removes the lowest-entropy 20% from the clipped generalized Jensen-Shannon distillation loss, targeting token-level uncertainty rather than coarse trajectory outcomes
• SMOPD adds no parameters and has zero inference-time overhead, making it computationally efficient to deploy
• On LiC with Qwen3 models, SMOPD improves SHARDED-view accuracy by 1.0-2.5 percentage points across 1.7B, 4B, and 8B models, with a statistically significant +1.7pp mean gain at 4B (p = 0.022)
• Token-level uncertainty proves to be a more reliable stabilization signal than scalar final-answer correctness in dirty-history OPSD settings, while combining both approaches shows scale-dependent effects

Why It Matters

This research addresses a critical bottleneck in multi-turn on-policy self-distillation, where error propagation through dirty histories can severely degrade model performance. By introducing a parameter-free, zero-overhead stabilization technique, SMOPD offers practitioners a practical solution to improve training stability without architectural changes or computational penalties. The findings also challenge the common assumption that outcome-level correctness signals are optimal for distillation, suggesting token-level uncertainty may be a superior guidance signal.

Technical Details

• Core Mechanism: SMOPD operates on the clipped generalized Jensen-Shannon distillation loss, ranking token positions within each generated middle-turn reply by student entropy and masking out the lowest-entropy 20% before loss computation
• Preservation Strategy: Final-answer losses and FULL-preservation losses remain completely unchanged, ensuring the method only targets intermediate-turn uncertainty without disrupting established training objectives
• Comparison Variant: The authors evaluate a correctness-scaling variant that multiplies a detached reliability proxy based on final-answer correctness, finding it harmful without masking at 1.7B (-4.0pp) and scale-dependent when combined with masking
• Experimental Setup: Evaluated on LiC benchmark using Qwen3 models at 1.7B, 4B, and 8B parameter scales, with both single-seed and multi-seed comparisons
• Statistical Validation: The 4B multi-seed check demonstrates a +1.7pp mean SHARDED gain over baseline with statistical significance (two-tailed p = 0.022)

Industry Insight

• The results suggest that token-level uncertainty metrics should be prioritized over outcome-level correctness signals when designing distillation losses for multi-turn conversational agents, particularly in scenarios prone to error propagation
• SMOPD's parameter-free, zero-overhead design makes it immediately deployable in existing on-policy self-distillation pipelines without infrastructure changes, offering a low-risk optimization for production LLM training
• The scale-dependent behavior observed when combining masking with correctness scaling indicates that hybrid stabilization strategies require careful calibration per model size, and practitioners should validate stabilization methods across their target model scales before deployment

TL;DR

• SMOPD是一种针对多轮on-policy自蒸馏(OPSD)的loss-only稳定化方法，解决"脏历史"rollouts导致的训练不稳定问题
• 核心机制：对中间轮次回复按学生模型熵排序，移除最低熵20%的token从截断广义Jensen-Shannon蒸馏损失中，最终答案和FULL-preservation损失保持不变
• 该方法零参数增加、零推理开销，在Qwen3模型(1.7B/4B/8B)的LiC基准上SHARDED-view准确率提升1.0-2.5pp
• 4B多seed验证显示+1.7pp均值提升(双尾p=0.022)，而添加标量正确性校正无masking时反而有害(-4.0pp)
• 实验表明token级不确定性比标量最终答案正确性更可靠的稳定信号，但因果机制测试和更广泛基准验证留待未来工作

为什么值得看

本文针对多轮对话系统中on-policy自蒸馏的训练稳定性痛点，提出了一种无需额外参数、零推理开销的轻量级解决方案。研究揭示了token级不确定性作为蒸馏信号的价值，为多轮推理场景的模型训练提供了新的技术路径。

技术解析

SMOPD的核心创新在于token-level的熵筛选机制：对每个生成的中间轮次回复，计算学生模型各token位置的熵值并排序，从截断广义Jensen-Shannon蒸馏损失中移除最低熵20%的token，最终答案和FULL-preservation损失不受影响。该方法仅修改损失函数，不引入任何新参数，推理阶段完全透明。

实验基于Qwen3系列模型(1.7B/4B/8B)在LiC基准上的评估，采用SHARDED-view准确率作为主要指标。单seed比较显示SMOPD带来1.0-2.5pp提升，4B模型的多seed验证显示+1.7pp均值提升且统计显著(p=0.022)。对比实验表明，仅使用标量正确性校正无masking时反而导致1.7B模型性能下降4.0pp，而结合masking的效果也呈现规模依赖性。

行业启示

多轮对话系统的训练稳定性优化应从轨迹级粗粒度信号转向token级细粒度不确定性分析，这为on-policy蒸馏方法的设计提供了新的方向。

轻量级loss-only改进方案在保持零额外开销的同时实现显著性能提升，证明了在现有训练框架内进行精细化损失设计的有效性，适合工业界快速落地。

当前研究仍局限于单一基准和有限规模验证，未来需在更广泛的场景和更大模型上进行因果机制测试，以确认方法的泛化能力。