Thinking Machines Lab Releases Inkling-Small: A 276B Total, 12B Active Open Weights Multimodal MoE Model
Inkling-Small is a 276B total / 12B active sparse Mixture-of-Experts model released under Apache 2.0, roughly a quarter the size of its 975B teacher model Inkling It outperforms Inkling on reasoning and agentic coding benchmarks (HLE 31.6% vs 29.7%, SWE-bench Verified 80.2% vs 77.6%, ARC-AGI-2 40.1% vs 36.5%) despite being significantly smaller The NVFP4 quantized checkpoint enables deployment on a single NVIDIA B300 GPU with only 180 GB VRAM, dramatically lowering the barrier for self-hosting f
Analysis
TL;DR
- Inkling-Small is a 276B total / 12B active sparse Mixture-of-Experts model released under Apache 2.0, roughly a quarter the size of its 975B teacher model Inkling
- It outperforms Inkling on reasoning and agentic coding benchmarks (HLE 31.6% vs 29.7%, SWE-bench Verified 80.2% vs 77.6%, ARC-AGI-2 40.1% vs 36.5%) despite being significantly smaller
- The NVFP4 quantized checkpoint enables deployment on a single NVIDIA B300 GPU with only 180 GB VRAM, dramatically lowering the barrier for self-hosting frontier-class MoE models
- The model is natively multimodal (text, image, audio) with a 1M token context window and adjustable thinking effort, trained on NVIDIA GB300 NVL72 systems
- Factual recall regressed compared to the teacher: SimpleQA Verified dropped to 20.6% from Inkling's 43.9%, and the AA Omniscience index fell to -9.0 from 2.1
Why It Matters
Inkling-Small demonstrates that a significantly smaller MoE model can surpass its larger teacher on complex reasoning and agentic coding tasks, challenging the assumption that scale alone drives capability. The availability of an NVFP4 checkpoint running on a single B300 makes frontier-class multimodal reasoning accessible to startups and regulated enterprises that previously could only access such models through expensive API endpoints. This shifts the economics of self-hosting large reasoning models from data-center-scale deployments to single-GPU instances.
Technical Details
- Architecture: 42-layer decoder-only transformer with a sparse MoE feed-forward backbone; each token routes to 6 of 256 experts plus 2 shared experts active on every token. Hybrid local and global attention layers. Encoder-free native multimodal design.
- Multimodal processing: Images are split into 40×40-pixel patches and transformed via a four-layer hMLP; audio is represented as dMel spectrograms. Both modalities pass through lightweight embedding layers and are processed jointly with text tokens within the same decoder. Audio input is WAV at 16 kHz (ideally under 2 minutes); output is text only.
- Training pipeline: Trained after the larger Inkling, allowing revisions to the pre-training data mix and ML recipe. Post-training used on-policy distillation from Inkling as teacher, followed by two weeks of agentic coding RL scaling.
- Quantization & deployment: Supports BF16, MXFP8, and NVFP4 numerics. BF16 requires 600 GB aggregated VRAM (4× B300 or 8× H200); NVFP4 drops this to 180 GB. W4A4 runs on a single B300 (SM100+ required); W4A16 runs on 2× H200. Compatible runtimes include SGLang, vLLM, TokenSpeed, Unsloth, and Hugging Face.
- Benchmark highlights: HLE 31.6%, SWE-bench Verified 80.2%, Terminal-Bench 2.1 64.7%, Toolathlon Verified 54.4%, GPQA Diamond 89.5%, AIME 2026 95.5%, IFBench 82.2%, ARC-AGI-2 40.1%. Multimodal: MMMU Pro 74.0%, CharXiv RQ 77.4% (81.3% with Python tool use), Audio MC 54.9%, MMAU 77.0%, VoiceBench 90.1%. ForecastBench Brier Index 61.3. Safety: StrongREJECT 98.4%, FORTRESS adversarial 71.6%, FORTRESS benign 96.9%.
Industry Insight
- The single-GPU deployment path for a 276B-parameter model effectively eliminates the infrastructure moat that frontier labs previously held, enabling startups and mid-size enterprises to self-host reasoning-capable models without multi-GPU clusters. This will accelerate private-deployment adoption in regulated sectors (finance, healthcare, telecom, public sector) where data sovereignty is critical.
- The performance regression in factual recall (SimpleQA) alongside gains in reasoning and coding suggests a trade-off in the training recipe that prioritizes agentic and reasoning capabilities over factual grounding. Practitioners should be aware that this model may require additional factual grounding layers or retrieval augmentation for knowledge-intensive applications.
- The adjustable "thinking effort" parameter combined with native multimodal input (text, image, audio) positions this model as a strong candidate for autonomous agent workflows—particularly coding agents, terminal automation, and call-center analytics—where dynamic compute allocation and multi-modal perception are essential.
Disclaimer: The above content is generated by AI and is for reference only.