Rethinking Privileged Information in On-Policy Self-Distillation
On-policy self-distillation (OPSD) trains a student model on its own responses using token-level supervision from the same model conditioned on privileged reference information The study finds that performance gains from OPSD may reflect recovery of reasoning behavior already present in the base model rather than genuine learning of privileged reference information Experiments across Qwen3 models (1.7B to 8B) on science and mathematics datasets show the correct reference does not consistently im
Analysis
TL;DR
- On-policy self-distillation (OPSD) trains a student model on its own responses using token-level supervision from the same model conditioned on privileged reference information
- The study finds that performance gains from OPSD may reflect recovery of reasoning behavior already present in the base model rather than genuine learning of privileged reference information
- Experiments across Qwen3 models (1.7B to 8B) on science and mathematics datasets show the correct reference does not consistently improve performance across different teacher generation modes, model sizes, and training datasets
- Students can improve without the correct reference, and solutions from unrelated problems can outperform correct solutions on mathematical reasoning benchmarks
- Student predictions align more strongly with the base model's thinking behavior than with supervision induced by the reference, challenging assumptions about how privileged information contributes to learning
Why It Matters
This research directly challenges a widely-used training paradigm (OPSD) that many practitioners rely on for improving model reasoning through privileged reference information. It forces the AI community to reconsider whether observed performance gains in self-distillation setups genuinely reflect knowledge transfer from references or merely recover pre-existing capabilities, which has significant implications for how training pipelines are designed and evaluated.
Technical Details
- Methodology: The authors developed an analysis framework that separates supervision induced by the privileged reference from supervision provided by the teacher model without the reference, measuring how each aligns with changes in the student's predictions
- Experimental Setup: OPSD experiments conducted on science and mathematics datasets using Qwen3 models ranging from 1.7B to 8B parameters
- Key Finding on Alignment: Student predictions showed stronger alignment with the base model's thinking behavior than with the supervision induced by the reference, though controls constructed from other problems reproduced much of both alignments
- Reference Independence: The study demonstrated that stronger alignment attributable to the correct reference does not reliably coincide with greater performance benefit, suggesting performance gains and distributional alignment alone cannot determine how privileged reference information contributes to student learning
Industry Insight
- Practitioners using OPSD should not assume performance improvements guarantee the student has learned the privileged reference information; rigorous ablation studies separating reference supervision from base model behavior are essential
- The finding that incorrect references can outperform correct ones suggests careful curation and validation of reference materials in self-distillation pipelines is critical, and existing benchmarks may need re-evaluation
- Researchers should develop more sophisticated attribution methods beyond performance gains and distributional alignment to understand what information is actually being transferred during on-policy self-distillation training
Disclaimer: The above content is generated by AI and is for reference only.