Frontier post-training recipe review with Finbarr Timbers 前沿后训练方案回顾：与Finbarr Timbers对话

TL;DR

• Post-training recipes fragmented from monolithic pipelines into multi-teacher distillation (MOPD) by 2026.
• MOPD emerged because single RL runs for math, code, and agentic tasks became expensive and conflict-prone.
• DeepSeek V4 scales MOPD to 10+ domain-specialist teachers, a pattern started by MiMo Flash v2.
• RLVR became the centerpiece in 2025 (DeepSeek R1), but 2026 shifted to distillation for consolidation.
• The core engineering trade-off is now between specialist training cost and final model performance.

Key Data

Entity	Key Info	Data/Metrics
MiMo Flash v2 (Jan 2026)	First to introduce MOPD pattern	Trained ~6 domain-specialist teachers
DeepSeek V4 (Apr 2026)	Scaled MOPD from MiMo's pattern	Uses >10 domain experts/teachers
Nemotron 3 Ultra (Jun 2026)	Multi-round MOPD variant	Two iteration rounds, >10 teachers
InstructGPT (Mar 2022)	Canonical three-step pipeline	SFT → Reward Model → PPO
DeepSeek R1 (Jan 2025)	Made large-scale RL the centerpiece	Cold-start SFT → reasoning RL → distillation
Llama 3 (Jul 2024)	Complex multi-stage, no online RL	6 rounds: RM → rejection sampling → SFT → DPO

Deep Analysis

The transcript reveals a fundamental schism in how the field approaches the "last mile" of model training. We're moving past the era where post-training was a simple, sequential polish. It's now the primary battleground for capability, and the engineering has bifurcated into two distinct philosophies: the RL-centric maximalist (exemplified by DeepSeek R1) and the system-centric modularist (exemplified by the 2026 MOPD stack). The article's timeline is a confession: RL, while powerful, hit a scalability wall.

The core problem RL faced wasn't a lack of clever algorithms, but an organizational and computational one. Training a single model with a single RL reward signal to be excellent at math, code, and agentic reasoning simultaneously is a recipe for catastrophic interference. The rewards conflict. The model learns trade-offs, not synergies. This is why MiMo Flash v2's MOPD feels less like a eureka moment and more like an inevitable engineering surrender. It's a direct admission that we cannot yet engineer a single, unified reward function for superhuman multidiscipline competence. Instead, we outsource the problem: create a committee of cheap, focused "expert" models, each mastering one domain via its own tailored RL, then distill their knowledge into a generalist student.

This shift has profound implications for competitive dynamics. The "scaling laws" of pre-training are well-understood. The scaling laws of post-training are now emerging, and they look more like a complex systems problem. MOPD makes post-training organizationally parallelizable. A large lab can now have separate teams for math, code, and agentic RL, iterating independently without breaking the main model. The final MOPD stage is a merger. This is a direct response to the "RL got expensive" line. It's cheaper to have ten small specialist RL runs than one colossal, fraught one. DeepSeek V4 scaling to 10+ teachers isn't just a technical flex; it's an organizational blueprint.

However, I see a critical vulnerability in this pattern. MOPD is, at its heart, a sophisticated form of knowledge distillation. The quality of the student model is strictly bounded by the quality of the teacher ensemble. Are we merely creating a more efficient way to compress the current state-of-art from isolated domains, or are we capping potential breakthroughs? The transcript mentions MAI-Thinking-1 as a notable holdout, preferring a multi-stage RL climb closer to R1. This suggests a philosophical fork: is the goal a perfect amalgamation of existing skills (MOPD), or the emergence of new, synergistic skills from within a single, deep RL process (R1-style)? The former optimizes for predictable, integrated performance. The latter chases a black swan capability jump.

Furthermore, the "generalist student" trained via MOPD is no longer just trained on human data or a single reward. Its behavior is shaped by the output distributions of its specialist teachers. This introduces a new layer of abstraction—and opacity. If the student model hallucinates, is the fault in its own weights, or in a subtle misalignment it inherited from one of a dozen teachers? Debugging this system will be orders of magnitude harder than debugging a traditional RLHF model. The move to MOPD trades the known, acute problem of RL reward hacking for the unknown, systemic problem of multi-teacher distortion.

Ultimately, the 2026 recipe landscape shows a field that has mastered the components of intelligence—reasoning, coding, tool use—but has not yet solved their integration. MOPD is a brilliant, pragmatic patch. It's the microprocessor of model training: taking specialized, existing modules and using a standardized bus (on-policy distillation) to connect them. It will deliver powerful, consistent models. But I suspect the next true frontier leap won't come from a better MOPD architecture. It will come from whoever finally figures out how to write the single, elegant reward function that makes a model want to be a math champion, a coding virtuoso, and a savvy agent all at once. That person won't need a committee of teachers.

Industry Insights

1. Competitive advantage will shift from pre-training scale to post-training pipeline engineering and teacher-model orchestration.
2. Labs must build parallel, domain-specialized RL teams to efficiently produce the teacher models for the new MOPD paradigm.

FAQ

Q: What is MOPD and why did it become the dominant 2026 pattern?
A: Multi-teacher On-Policy Distillation (MOPD) trains many specialist models first, then distills them into one final model via on-policy sampling. It emerged because single RL runs became too expensive and caused capability trade-offs between domains.

Q: How does this differ from traditional RLHF like in InstructGPT?
A: Traditional RLHF uses a single, general reward model to align one model. MOPD uses multiple, domain-specific teachers (each trained with their own RL) to shape the final model, acting like a committee of experts.

Q: What are the potential downsides of the MOPD approach?
A: The final model's capabilities are capped by its teachers, and debugging becomes complex since errors may stem from inherited teacher biases rather than the student's own training. It may also inhibit the emergence of novel, cross-domain synergies.

TL;DR

• 2026年前沿大模型的后训练核心范式，已从单一的RLHF流水线演变为“多专家教师蒸馏”。
• 新模式MOPD（多教师在策略蒸馏）由MiMo Flash v2率先提出，并在DeepSeek V4、Nemotron 3 Ultra上规模化。
• MOPD的出现源于将数学、代码等多能力混合训练在同一RL过程中成本高且易产生冲突。
• 该范式意味着训练流程“从单体走向分布式再合并”，对组织的并行工程能力提出高要求。
• 开源社区复现前沿模型的门槛进一步提高，因需先构建多个领域专家模型。

核心数据

实体	关键信息	数据/指标
MiMo Flash v2	提出MOPD范式的模型	2026年1月，训练约6个领域专家教师
DeepSeek V4	应用MOPD的模型	2026年4月，使用超过10个领域专家教师
Nemotron 3 Ultra	进行了两轮MOPD的模型	2026年6月，使用超过10个教师模型
DeepSeek R1	以大规模RL为核心驱动的模型	2025年1月发布，采用纯RL（GRPO）冷启动
Llama 3	多阶段复杂后训练配方的代表	2024年7月，采用6轮“拒绝采样→SFT→DPO”迭代

深度解读

这篇文章揭示的不仅仅是技术配方的迭代，更是AI实验室组织形态和竞争逻辑的一次深刻变形。从InstructGPT那清晰优雅的“SFT→RM→RL”三部曲，到如今这个由多个专家模型并行训练、最终蒸馏合并的“分布式流水线”，后训练的复杂性已经指数级增长。这不再是研究员在实验室里调优单个模型的游戏，而演变成了一场对实验室工程管理、资源调度和跨领域协作能力的极限考验。

MOPD模式的兴起，本质上是对“RL万能论”的一次修正。文章一针见血地指出：“混合数学、代码和代理能力的RL训练，最终会导致能力间的相互牺牲。” 这意味着，追求一种全能的、单一的强化学习过程已被证明在经济性和效果上难以为继。于是，“专业化分工，再融合集成”成了自然选择。一个模型团队不再需要精通所有领域的专家坐在一起调和一个复杂的奖励函数，而是可以让数学组、代码组、Agent组各自用相对成熟的SFT+RL流程“练级”，最后通过蒸馏让主模型“学习”所有专家的长处。这极大地降低了算法复杂度，但将压力转移到了系统架构和训练流水线上。

这也预示着AI竞赛的一个残酷转折点：资源与规模的门槛被结构性提高。当你需要先训练10个以上的领域专家，再进行多轮蒸馏，你所需要的算力、数据和工程团队规模已经远非初创团队或学术机构所能轻易企及。DeepSeek和Nemotron的实践表明，顶尖选手正在用更复杂的“工业流水线”拉开差距。开源社区虽然能获取最终模型，但要复现这个“生产过程”本身已变得极其困难。未来的开源优势可能在于社区协作构建专家模型，而非单独挑战全流程。

更深远的影响在于，这种范式可能让模型的“个性”和“一致性”面临挑战。一个通过融合众多专家教师而成的学生模型，如何在各种边缘情况下保持行为的连贯和安全？当不同教师的知识可能隐含冲突时，简单的逆KL散度最小化能否解决深层次的价值观对齐问题？这或许是MOPD模式在光鲜的技术效率之下，必须直面的“阿喀琉斯之踵”。

行业启示

1. AI实验室需向“精密制造”转型：后训练已非单纯的算法竞赛，组织需要构建可并行、模块化的多专家训练流水线，工程管理能力变得与算法研究同等重要。
2. 技术路线选择分化：一条是追求极致整合能力的MOPD“分布式融合”路线；另一条是MAI-Thinking-1等坚持的、更接近R1的“多阶段RL强化”路线，后者可能更适用于能力高度统一、对推理连贯性要求极高的场景。
3. 开源社区的协作新契机：社区可以尝试协作构建高质量的领域专家模型（如数学、代码），再通过MOPD框架进行整合，这可能成为对抗闭源实验室巨头的有效组织形式。

FAQ

Q: 什么是MOPD，它和传统的RLHF有什么区别？
A: MOPD（多教师在策略蒸馏）是一种新的后训练范式。它不再用单一奖励模型对主模型进行RL，而是先独立训练多个领域专家教师模型，再让最终的学生模型通过模仿（最小化逆KL散度）来学习所有专家的输出分布。它解决了多能力混合训练时的冲突问题，将单体RL变成了分布式蒸馏。

Q: 为什么RL训练在2026年变得“昂贵且容易冲突”？
A: 因为随着模型能力向数学、代码、工具使用等多维度拓展，试图用一个统一的奖励函数在一次训练中同时提升所有能力，会导致优化目标互相干扰（例如，为代码优化的奖励可能不利于数学推理的泛化），且需要处理海量异构数据和环境交互，计算成本与日俱增。

Q: 这一趋势对AI开源社区意味着什么？
A: 挑战与机遇并存。挑战在于，复现前沿模型的全流程门槛极高。机遇在于，社区可以发挥分布式优势，协作训练并开源各领域的专家模型，然后通过MOPD框架整合，这或许能开辟一条集体突破的路径。