Cursor Open-Sources Mixture-of-Kittens (MoK): A Deterministic MoE Training Megakernel for GB300 NVL72 Racks
Cursor Research open-sourced Mixture-of-Kittens (MoK), a deterministic MoE training megakernel that fuses all communication and computation steps into a single kernel, powering Composer models across tens of thousands of GPUs MoK achieves up to 2.37x higher throughput than the strongest public baseline (MXFP8 forward), with 1.41x end-to-end gain across 512 GPUs on GB300 NVL72 racks Three key innovations: pull-based dispatch (5.8x faster signaling than push at 18 µs vs 103 µs), mid-granularity ov
Analysis
TL;DR
- Cursor Research open-sourced Mixture-of-Kittens (MoK), a deterministic MoE training megakernel that fuses all communication and computation steps into a single kernel, powering Composer models across tens of thousands of GPUs
- MoK achieves up to 2.37x higher throughput than the strongest public baseline (MXFP8 forward), with 1.41x end-to-end gain across 512 GPUs on GB300 NVL72 racks
- Three key innovations: pull-based dispatch (5.8x faster signaling than push at 18 µs vs 103 µs), mid-granularity overlap heuristic targeting 2+ SM waves per expert GEMM, and a ring token buffer eliminating CPU-GPU synchronization entirely
- Requires NVIDIA Blackwell SM100/SM103 GPUs (GB200/GB300 NVL72 racks), Python 3.12+, PyTorch 2.10+, CUDA 13.0+, and PyTorch symmetric memory for inter-GPU buffers
- Apache-2.0 licensed; targets pretraining/post-training of DeepSeek-V3-style MoE models, on-policy RL post-training, and internal ablations
Why It Matters
MoK addresses the critical bottleneck in MoE training—inter-GPU communication, which can consume over half of end-to-end training time—by eliminating CPU involvement and maximizing NVLink utilization through a fully fused megakernel. For AI practitioners scaling MoE models on NVL72-scale infrastructure, this represents a significant throughput improvement that directly reduces training costs and time-to-train for frontier model development.
Technical Details
- Architecture: MoK fuses all MoE communication (dispatch/combine) and computation (GEMM, router gradient) into a single deterministic megakernel. Scheduling leverages Blackwell's Cluster Launch Control to avoid RDMA serialization. Router weight gradients use a SonicMoE-style calculation fused into the SwiGLU backward pass.
- Communication strategy: Asymmetric pull/push design—pull-based forward dispatch (18 µs signaling vs 103 µs for push, ~5.8x faster) with up to 29% higher NVLink bandwidth utilization under expert imbalance, combined with push-based forward combine. Backward mirrors this with pull reverse-combine and push reverse-dispatch, all served by one schedule table costing <3% of MoE runtime.
- Overlap granularity: Targets a middle ground between Comet (fine-grained) and DeepEP (coarse-grained), using a heuristic that ensures at least two full SM waves per expert-grouped GEMM. For Kimi 2.5 shapes, the token floor is 2,368 tokens.
- Ring token buffer: A fixed ring buffer of a few hundred megabytes cycles at minibatch granularity, interleaving dispatch and combine at macrobatch boundaries. The ring is walked in reverse during backward to minimize forward activation replay. Zero tokens are dropped, and CPU-GPU synchronization is eliminated.
- Benchmarks: Tested on single NVL72 rack (EP degree 64, 2,048 tokens/GPU) against NCCL+PyTorch, DeepEP+PyTorch, DeepEP+TransformerEngine, and HybridEP+Megatron across Kimi K2.7 Code, GLM-5.2, Qwen3.5-397B-A17B, and DeepSeek-V4-Pro shapes. MXFP8 forward: 2.37x, MXFP8 backward: 1.78x, BF16 forward: 1.92x, BF16 backward: 1.58x. End-to-end on 512 GPUs: 760.9 → 1,070.2 tokens/sec/GPU (1.41x).
Industry Insight
- MoK's hardware requirements (Blackwell SM100/SM103, NVL72 racks) create a high barrier to entry, concentrating competitive advantage among frontier labs, well-funded startups, and GPU neoclouds—smaller teams with 8-GPU setups cannot realistically adopt this optimization.
- The pull-based dispatch design that eliminates cross-GPU completion signals and reduces signaling overhead by 5.8x could become a reference pattern for future MoE communication libraries, potentially influencing the broader ecosystem beyond the NVL72 form factor.
- As MoE architectures continue to dominate frontier model development (DeepSeek-V3, Kimi, etc.), fused megakernel approaches that remove CPU bottlenecks will likely become standard practice for large-scale training, pushing the industry toward more integrated software-hardware co-design.
Disclaimer: The above content is generated by AI and is for reference only.