DeepSeek AI Released DeepSeek-V4.1-Flash with 1M Context, FP4 KV Cache, and Cross-Layer Attention Reuse
DeepSeek-V4.1-Flash achieves a global KV cache footprint of 890 bytes per token — roughly 1/4 of V4-Flash and 437x smaller than V1 — enabling 1M-token context windows without HBM/SSD bottlenecks The Causal Encoder-Decoder (CED) architecture splits the 40-layer backbone into 20 encoder + 20 decoder layers, halving prefill compute by having the decoder derive its global KV from the encoder's final hidden state rather than computing it independently Compressed Sparse Attention 2 (CSA2) introduces t
Analysis
TL;DR
- DeepSeek-V4.1-Flash achieves a global KV cache footprint of 890 bytes per token — roughly 1/4 of V4-Flash and 437x smaller than V1 — enabling 1M-token context windows without HBM/SSD bottlenecks
- The Causal Encoder-Decoder (CED) architecture splits the 40-layer backbone into 20 encoder + 20 decoder layers, halving prefill compute by having the decoder derive its global KV from the encoder's final hidden state rather than computing it independently
- Compressed Sparse Attention 2 (CSA2) introduces three layer modes (Full, Reindex, Reuse) that share main KV, indexer K, and Top-K indices across layers, with a Hierarchical Sparse Indexer capping candidate pools at 16,384 positions
- FP4 (E2M1) KV cache quantization with per-16-channel E4M3 scales, combined with SWA Bounded Replay and DRAM-resident sliding-window storage, reduces persistent cache to approximately 1/8 of V4-Flash
- The model matches DeepSeek-V4-Pro-Base on world knowledge and coding while using only 1/3 total and 1/4 activated parameters, and beats Opus-5 and GPT-5.6 Sol on Terminal-Bench 2.1 and DeepSWE v1.1 with open MIT-licensed weights
Why It Matters
Long-horizon agentic workflows are turning LLM serving into an input-heavy, KV-cache-bound workload, and DeepSeek-V4.1-Flash directly targets this emerging bottleneck with architectural and systems-level innovations that make million-token contexts practically deployable. For AI practitioners, the combination of open weights, aggressive compression, and competitive benchmark performance against closed models like Opus-5 and GPT-5.6 Sol demonstrates that high-quality, cost-effective agent infrastructure is becoming accessible without proprietary API dependency.
Technical Details
- Architecture: Multimodal Mixture-of-Experts with a 552B backbone + 196B Engram parameters, 1M-token context window, 8B activated params per token during prefill (20 encoder layers) and 16B during decode (20 decoder layers). The Causal Encoder-Decoder derives decoder global KV via per-layer projection from the encoder's final hidden state, eliminating redundant KV computation.
- CSA2 (Compressed Sparse Attention 2): Each of the 40 layers is statically assigned one of three modes — Full (computes own KV + indexer K, selects Top-512 indices), Reindex (reuses KV/indexer K, rescores with own indexer Q), or Reuse (skips indexer entirely, reuses latest Top-K). Encoder uses a 2:1 compression ratio (1 Full + 5 Reuse per 6-layer group); decoder uses 1:1 (Full + 3 Reuse or Reindex + 3 Reuse per 4-layer group). A Hierarchical Sparse Indexer bounds candidate pools to 16,384 positions (2,048 blocks × 8).
- FP4 KV Cache + SWA Bounded Replay: Main KV quantized to E2M1 with one E4M3 scale per 16 channels (NVFP4-style without global scale), introduced via quantization-aware post-training. SWA KV (128-token window) is no longer persisted to SSD but lives in a distributed DRAM pool (10% host DRAM, minute-scale TTL), while global KV maintains a 72-hour lifetime. On cache miss, only 128 tokens are recomputed instead of layers × window.
- Additional optimizations: Single-Pass mHC shifts input-mixing coefficients to halve activation memory traffic via a fused Mega-mHC kernel; Engram conditional memory modules at layers 1 and 14; DSpark speculative decoding (backbone frozen, trained post-pretraining); head-wise Muon optimizer. Single-token decode FLOPs increase by only 25% when scaling from 4K to 1M context.
- Training: 45T multimodal tokens (7:1 text-to-multimodal ratio), sparse attention trained from scratch at 64K with no dense warmup, context extended to 1M at 34T tokens. Post-training uses large-scale verifiable agent task synthesis, RL across heterogeneous scaffolds (Claude Code, Codex, OpenCode, Pi, mini-SWE, DeepSeek Harness), and on-policy distillation from 40+ teachers.
Industry Insight
- The 437x KV cache reduction per token signals a fundamental shift: million-token contexts are moving from research demos to production realities, and serving infrastructure (vLLM, SGLang) will need to prioritize sparse attention and DRAM-tiered caching over raw HBM capacity.
- Open-weight models matching or exceeding closed competitors on agent benchmarks (Terminal-Bench, DeepSWE) under MIT licensing compress the advantage of proprietary APIs, accelerating the commoditization of high-capability agent backends and pushing providers to compete on infrastructure and tooling rather than model exclusivity.
- The layered Full/Reindex/Reuse compression strategy demonstrates that static, per-layer attention mode assignment is a viable design space — future work may explore dynamic mode selection or learned compression ratios, but the current approach proves that aggressive KV sharing across layers can preserve quality while slashing memory by orders of magnitude.
Disclaimer: The above content is generated by AI and is for reference only.