OPTD: On-Policy Transition Distillation with Consistency-Guided Adaptive Compression for Few-Step Diffusion Language Models
OPTD introduces on-policy transition distillation for few-step diffusion language models, addressing the distribution mismatch between off-policy supervised trajectories and actual student inference trajectories. The method uses a frozen, question-only teacher to identify outcome-aligned future token candidates and selects the longest prefix whose joint commitment preserves the teacher's rollout outcome. A set-bottleneck objective promotes verified future candidates to the decoder's release thre
Analysis
TL;DR
- OPTD introduces on-policy transition distillation for few-step diffusion language models, addressing the distribution mismatch between off-policy supervised trajectories and actual student inference trajectories.
- The method uses a frozen, question-only teacher to identify outcome-aligned future token candidates and selects the longest prefix whose joint commitment preserves the teacher's rollout outcome.
- A set-bottleneck objective promotes verified future candidates to the decoder's release threshold, while a frozen-teacher KL anchor regularizes all other active positions.
- Training and target construction are gold-response-free, relying entirely on self-generated trajectories and teacher consistency signals.
- OPTD achieves the strongest quality-constrained AUP across four mathematical reasoning and code-generation benchmarks among evaluated few-step baselines.
Why It Matters
Few-step distillation is critical for making diffusion language models practically deployable, as their parallel token prediction is only useful if the number of denoising steps can be aggressively reduced. This work directly addresses the fundamental distribution shift problem that has limited the effectiveness of prior distillation methods, offering a principled on-policy alternative that better aligns training with inference. For practitioners building or deploying dLLMs, OPTD represents a meaningful step toward high-quality, low-latency generation without requiring ground-truth responses.
Technical Details
- On-policy trajectory sampling: Unlike prior off-policy distillation, OPTD samples partial states directly from the few-step student's own generation trajectories, ensuring the training distribution matches inference conditions where early parallel commitments alter subsequent context.
- Frozen question-only teacher: A teacher model conditioned only on the input question (not on student-generated prefixes) identifies future token candidates that are outcome-aligned, then ranks them by current-state confidence to select the longest prefix preserving the teacher's rollout outcome.
- Set-bottleneck objective: Every verified future candidate is promoted to the decoder's release threshold, encouraging the student to commit to longer correct prefixes in a single transition rather than conservative one-step matches.
- KL anchor regularization: A frozen teacher KL divergence term regularizes all non-promoted active positions, preventing distributional collapse while allowing aggressive compression of verified prefixes.
- Gold-response-free training: Neither target construction nor the training objective requires ground-truth responses, making the approach applicable to scenarios where reference outputs are unavailable or unreliable.
Industry Insight
- The shift from off-policy to on-policy distillation in diffusion LMs addresses a fundamental scalability bottleneck; as dLLMs compete with autoregressive models on speed, methods that close the training-inference distribution gap will determine which architectures become practically viable.
- The gold-response-free design is strategically significant for real-world deployment, where reference answers are often unavailable (e.g., open-ended code generation, creative writing), enabling distillation in domains where supervised fine-tuning is infeasible.
- The consistency-guided adaptive compression framework—selecting prefix length based on outcome preservation rather than fixed compression ratios—suggests a broader paradigm shift toward dynamic, outcome-aware distillation that could generalize beyond diffusion models to other parallel-generation architectures.
Disclaimer: The above content is generated by AI and is for reference only.