Weak-to-Strong On-Policy Distillation 弱到强策略蒸馏

TL;DR

• Weak-to-Strong On-Policy Distillation (W2S-OPD) enables improving a strong student model by distilling from multiple weaker models, overcoming limitations of traditional on-policy distillation that require a teacher at least as capable as the student.
• W2S-OPD constructs a proxy teacher in logit space using a contrast pair of smaller, cheaper models (positive and negative), isolating specific capability directions (e.g., RL skills, scale effects, hint-based reasoning) which are added to the student's base model.
• The student distills from this proxy teacher via per-token reverse KL minimization on its own rollouts, allowing it to surpass even domain-expert teachers and continue improving when all supervision sources are weaker than itself.
• Different contrast types yield distinct signals: post-RL and hint contrasts emphasize reasoning frameworks, while scale contrasts emphasize solving procedures.
• Evaluated across four math and three code benchmarks, W2S-OPD outperforms standard OPD and demonstrates robustness in frontier scenarios where no larger teacher exists.

Why It Matters

This work addresses a critical bottleneck in large language model scaling: the inability to transfer knowledge beyond the current frontier when no stronger teacher is available. By enabling weak-to-strong learning through clever logit-space contrast construction, W2S-OPD opens new pathways for efficient model improvement without requiring expensive retraining or access to superior models—making it highly relevant for researchers pushing state-of-the-art boundaries and practitioners seeking cost-effective model refinement strategies.

Technical Details

• Proxy Teacher Construction: A proxy teacher is formed by taking the difference between logits of two weak models (positive minus negative), then adding this delta to the student’s base model logits. This creates a teacher that embodies a specific capability direction while remaining distributionally close to the student.
• Contrast Pair Instantiations: Three key contrast pairs are used: (1) post-RL vs pre-RL expert to isolate reinforcement learning-induced skills; (2) larger vs smaller base model to capture scale-related capabilities; (3) small base model with correct vs wrong hints to extract instance-level solution guidance.
• Distillation Objective: The student minimizes per-token reverse Kullback-Leibler divergence between its own rollout distribution and the proxy teacher’s distribution, ensuring alignment without requiring external data or teacher rollouts.
• On-Policy Constraint: All training occurs on the student’s own generated sequences, preserving policy consistency and avoiding distributional shift issues common in off-policy methods.
• Benchmark Evaluation: Tested on four mathematical reasoning datasets and three coding tasks, showing consistent gains over baseline OPD and ability to exceed performance of domain-specific expert teachers.

Industry Insight

W2S-OPD presents a paradigm shift toward self-improving AI systems that can evolve beyond their initial training limits using only weaker internal resources—a crucial capability for autonomous agents operating in dynamic environments where superior models may not exist. Organizations investing in LLM development should prioritize integrating such weak-to-strong mechanisms into their pipelines to reduce dependency on massive compute for incremental gains and enable continuous model evolution even at the technological frontier. Additionally, the modular nature of contrast design suggests potential for domain-specific tuning (e.g., medical, legal) by crafting tailored weak-model pairs that isolate relevant expertise directions.

TL;DR

• 提出弱到强同策略蒸馏（W2S-OPD）框架，通过多个弱模型构建代理教师提升强学生模型能力。
• 利用对比学习思想，从正负模型的logit差异中提取能力方向，叠加到学生基模形成代理教师。
• 在数学与代码基准测试中超越传统OPD方法，即使所有监督源均弱于学生也能持续优化性能。
• 不同对比组合产生差异化信号：RL后/提示对比侧重推理框架，规模对比侧重解题流程。
• 解决前沿领域无更强教师可用的蒸馏瓶颈问题，降低对昂贵专家训练的依赖。

为什么值得看

该研究突破了传统同策略蒸馏必须依赖强于学生的教师的限制，为模型能力迁移提供新范式。其低成本构建代理教师的方法特别适用于资源受限场景或前沿领域模型优化，具有显著的工业落地价值。

技术解析

1. 核心机制：W2S-OPD在logit空间构造代理教师，通过正负模型（如RL前后模型、大小基模型、带正确/错误提示的小模型）的logit差分离出特定能力方向，再将该方向叠加到学生基模上生成代理教师。
2. 蒸馏过程：学生在自身rollout序列上最小化与代理教师的逐token反向KL散度，实现能力对齐而不改变分布特性。
3. 对比实例设计：①RL专家vs初始化模型提取强化技能；②大基模型vs小基模型捕获规模效应；③小基模型正/负hint获得实例级解题方向。
4. 实验验证：覆盖4个数学和3个代码基准测试，证明该方法不仅优于标准OPD，还能使学生在所有监督源弱于自身时持续提升表现。
5. 信号分析：经分析发现RL和hint对比主要增强推理框架能力，而规模对比更聚焦具体解题步骤执行。

行业启示

1. 模型压缩新路径：当缺乏更大规模教师模型时，可通过弱模型组合构建有效代理教师，降低知识迁移成本。
2. 强化学习应用优化：结合RL前后状态差异进行蒸馏，可高效保留强化学习带来的技能提升，避免重复训练。
3. 混合信号融合策略：同时使用多种对比类型（如规模+提示）能互补不同维度的能力，建议在实际部署中组合使用以提升泛化性。