From Pretraining to Agentic K3: How Moonshot Trained the Architecture
Kimi K3 is a 2.8-trillion-parameter multimodal model with native vision training (MoonViT-V2 trained from scratch alongside the language backbone), a million-token context window via Kimi Delta Attention, and agentic reasoning capabilities Moonshot designed training as a unified pipeline rather than bolted stages: Architecture → Pretraining → Base Model → SFT → Reinforcement Learning → MOPD → Unified K3 Nine specialist RL policies were trained across three domains (General Tasks, General Agents,
Analysis
TL;DR
- Kimi K3 is a 2.8-trillion-parameter multimodal model with native vision training (MoonViT-V2 trained from scratch alongside the language backbone), a million-token context window via Kimi Delta Attention, and agentic reasoning capabilities
- Moonshot designed training as a unified pipeline rather than bolted stages: Architecture → Pretraining → Base Model → SFT → Reinforcement Learning → MOPD → Unified K3
- Nine specialist RL policies were trained across three domains (General Tasks, General Agents, Coding Agents) and three reasoning effort levels (Low, High, Max), then consolidated into one model via Multi-Teacher On-Policy Distillation (MOPD)
- Progressive context expansion (8K → 64K → 256K → 1M tokens) and deliberately scattered long-context training examples were critical to teaching the model to actually use its million-token window
- Agentic RL required persistent rollout and sandbox state infrastructure, plus a Unified White-Box RL Environment supporting multiple harnesses (Kimi Code, Claude Code, Codex, OpenClaw, Hermes) to prevent overfitting to a single interface
Why It Matters
This article provides one of the most detailed public breakdowns of how a frontier multimodal reasoning model is trained end-to-end, revealing that architectural innovations like KDA and Gated MLA are only half the story—the real challenge lies in training methodology. For AI practitioners, the nine-policy specialization-then-consolidation approach and cross-harness agentic training offer practical blueprints for building capable agent systems without overfitting to specific tool interfaces.
Technical Details
- Native Multimodal Training: MoonViT-V2 vision encoder was trained from scratch jointly with the language backbone, rather than being pretrained separately and attached later. This produced more stable gradient norms during training compared to the pretrained baseline, achieving comparable visual performance with easier optimization.
- Progressive Context Expansion: K3 did not begin pretraining at million-token length. Context grew progressively from 8K to 64K during main pretraining, then extended to 256K and finally 1M tokens. Long-context training examples deliberately scattered useful information across distant parts of sequences (Clue A → gap → Clue B → gap → Clue C → gap → question requiring all three), forcing the model to learn genuine long-range dependency reasoning rather than local pattern matching.
- Nine RL Specialist Policies: Reinforcement learning was organized along two axes—task domain (General Tasks, General Agents, Coding Agents) and reasoning effort (Low, High, Max)—producing nine distinct policies. These are separate from the 896 MoE experts (which are architectural, not training-level). Each policy specializes in a specific domain-effort combination before consolidation.
- Persistent Rollout & Cross-Harness Agentic Training: Agentic RL infrastructure maintained persistent sandbox state across hundreds-step trajectories, distinguishing between long context (history of events) and persistent environment (consequences of actions). A Unified White-Box RL Environment instantiated multiple agent harnesses (Kimi Code, Claude Code, Codex, OpenClaw, Hermes) to prevent the model from overfitting to any single tool schema or command convention.
- Multi-Teacher On-Policy Distillation (MOPD): The nine specialist RL policies served as teachers to distill their specialized behaviors into a single unified student policy, eliminating the need to serve nine separate 2.8T-parameter checkpoints while preserving the behavioral diversity learned during specialization.
Industry Insight
- The "specialize-then-consolidate" paradigm (nine RL policies → MOPD) offers a scalable alternative to monolithic RL training, suggesting that frontier models may increasingly be built through targeted specialization rather than attempting to optimize a single policy for all capabilities simultaneously.
- Cross-harness agentic training represents a practical solution to a widely ignored overfitting risk: agent models trained in isolated tool environments often fail to generalize when tool interfaces change. The unified white-box environment approach should be adopted as a standard practice for any production agentic system.
- The distinction between architecture (what can happen) and training (what does happen) is critical for practitioners: investing in architectural innovations like recurrent memory and gated global attention is necessary but insufficient—equally rigorous investment in curriculum design, progressive scaling, and training infrastructure is required to realize those architectural possibilities.
Disclaimer: The above content is generated by AI and is for reference only.