Inside Kimi K3: How Moonshot AI Built the Largest Open-Source Model
Kimi K3 is the largest open-weight model as of mid-2026 at 2.8 trillion parameters, with Moonshot AI taking a novel approach to the memory bottleneck in long-context LLMs rather than simply scaling hardware. Kimi Delta Attention (KDA) replaces standard KV-cache-based attention with a fixed-size whiteboard-style memory that selectively forgets unimportant information using a learned Delta Rule with per-head forgetting dials, cutting memory use by ~75%. K3 combines KDA (69 of 93 attention layers)
Analysis
TL;DR
- Kimi K3 is the largest open-weight model as of mid-2026 at 2.8 trillion parameters, with Moonshot AI taking a novel approach to the memory bottleneck in long-context LLMs rather than simply scaling hardware.
- Kimi Delta Attention (KDA) replaces standard KV-cache-based attention with a fixed-size whiteboard-style memory that selectively forgets unimportant information using a learned Delta Rule with per-head forgetting dials, cutting memory use by ~75%.
- K3 combines KDA (69 of 93 attention layers) with Gated Multi-Head Latent Attention (24 layers) for exact recall, plus Attention Residuals, Stable LatentMoE with 896 experts (16 active per token), and several training optimizations including Quantile Balancing and Quantization-aware training in MXFP4/MXFP8.
- The model supports a 1-million-token context window with native multimodal (text, image, video) understanding and achieved top scores on WebDev, Program Bench, SWE Marathon, BrowseComp, and OmniDocBench at launch, though it trails Claude Fable 5 and GPT-5.6 Sol on overall benchmarks.
- KDA is the culmination of Moonshot's long-context research lineage (K1.5 → K2 → K2.5 → Kimi Linear), reflecting a strategic shift from brute-force compute scaling toward intelligent memory management in large models.
Why It Matters
Kimi K3 demonstrates that the next frontier in scaling large language models is not raw parameter count alone but intelligent memory architecture—teaching models to forget selectively rather than hoard everything. For AI practitioners and researchers, this signals a paradigm shift: as models approach trillion-parameter scales, the quadratic cost of standard attention makes architectural innovation in memory management essential for practical deployment. Moonshot's open-weight release and vLLM integration also lower the barrier for the community to experiment with and build upon these techniques.
Technical Details
- Kimi Delta Attention (KDA): A recurrent-style attention mechanism with a fixed-size memory buffer. Each attention head has 128 independent forgetting dials governed by two learned parameters: β (writing strength) and α (forgetting factor). The "erase then write" Delta Rule prevents catastrophic interference from new information overwriting old memories, building on Gated DeltaNet.
- Hybrid Attention Architecture: K3 uses 69 KDA layers interleaved with 24 Gated Multi-Head Latent Attention (Gated MLA) layers at roughly a 3:1 ratio. KDA handles efficient fixed-size memory for long contexts, while Gated MLA (an evolution of DeepSeek's MLA) compresses the KV cache into a latent representation to preserve exact token-by-token recall for critical details.
- Stable LatentMoE: A Mixture-of-Experts design with 896 total experts, activating only 16 per token, yielding ~104 billion active parameters out of 2.8 trillion total. This sparsity enables massive scale while keeping per-token compute tractable.
- Attention Residuals (AttnRes): A drop-in replacement for standard residual connections that mitigates PreNorm dilution by allowing later layers to selectively pull representations from earlier layers, improving training efficiency by ~25% for under 2% additional compute cost.
- Training Optimizations: Quantile Balancing replaces auxiliary loss terms for expert load balancing by deriving routing biases directly from router-score quantiles. Per-Head Muon extends the Muon optimizer for independent attention-head optimization. A custom Sigmoid Tanh Unit (SiTU) activation function replaces GeLU/SwiGLU. Quantization-aware training from SFT onward uses MXFP4 weights with MXFP8 activations, eliminating post-hoc quantization loss.
- Multimodal & Scale Specs: MoonViT-V2 vision encoder (~401M parameters) handles image and video input. Vocabulary size is ~160,000 tokens. Context window is 1 million tokens. Released as ~96 shards totaling ~1.56 TB under a custom Kimi K3 License.
Industry Insight
- The success of KDA suggests that memory efficiency, not just scale, will be the defining competitive advantage in the next generation of open-weight models. Labs that solve the quadratic attention cost problem will unlock practical million-token contexts for trillion-parameter models, opening doors to applications (full-codebase reasoning, long-document analysis, extended agentic workflows) that are currently infeasible.
- Moonshot's hybrid approach—combining efficient forgetting (KDA) with compressed exact recall (Gated MLA)—offers a practical blueprint for other labs: pure linear attention sacrifices too much fidelity, while pure standard attention is too expensive. The 3:1 interleaving ratio in K3 is a design choice worth studying and potentially adapting.
- The open-weight release with vLLM integration and quantization-aware training in MXFP4/MXFP8 lowers deployment barriers significantly. This could accelerate community adoption of trillion-parameter models and spur further innovation in efficient attention mechanisms, particularly as hardware constraints continue to limit access to the largest proprietary models.
Disclaimer: The above content is generated by AI and is for reference only.