The KV Cache Explained: Why Long Conversations Get Expensive
The KV cache stores precomputed keys and values from every token in a sequence, eliminating the need to reprocess the entire context for each new token generated during inference. Cache memory routinely exceeds model weights in production deployments, making it the primary bottleneck for GPU concurrency rather than compute capacity. PagedAttention (introduced by vLLM) eliminates severe memory waste from contiguous allocation by using fixed-size blocks allocated on demand, dramatically increasing
Analysis
TL;DR
- The KV cache stores precomputed keys and values from every token in a sequence, eliminating the need to reprocess the entire context for each new token generated during inference.
- Cache memory routinely exceeds model weights in production deployments, making it the primary bottleneck for GPU concurrency rather than compute capacity.
- PagedAttention (introduced by vLLM) eliminates severe memory waste from contiguous allocation by using fixed-size blocks allocated on demand, dramatically increasing serving throughput.
- When the cache pool is exhausted, the engine preempts running requests via recomputation or CPU swap, causing mid-generation stalls that signal overcommitment.
- GQA, KV cache quantization, and context limits are the main levers for reducing cache footprint, while prefix caching across requests remains an open optimization frontier.
Why It Matters
LLM serving systems are fundamentally constrained by memory, not compute—understanding the KV cache is essential for anyone deploying models at scale, as it directly determines how many concurrent users a GPU can serve. The shift from contiguous to paged allocation represents one of the most impactful engineering advances in inference infrastructure, and the emerging challenge of cross-request prefix caching points to the next frontier in reducing inference costs.
Technical Details
- KV cache composition: For each token, the model computes Query (Q), Key (K), and Value (V) vectors. Q is transient (computed fresh per token and discarded), while K and V are immutable once computed and must be retained for all subsequent attention operations across all layers.
- Memory formula: Per-token cache size = 2 × layers × KV heads × head dimension × bytes per value. A 70B-class model (80 layers, 8 KV heads, 128 head dim, FP16) consumes ~320 KB per token, meaning 8,000 tokens per user requires ~2.5 GB—exceeding the model's 140 GB weight footprint when serving ~100 users.
- PagedAttention: Replaces contiguous pre-allocation with fixed-size blocks (commonly 16 tokens each), managed via a per-sequence block table. This eliminates over-reservation, external fragmentation, and bounds internal fragmentation to at most one partial block per sequence.
- Eviction strategies: When the block pool is exhausted, the engine either recomputes (discards cache, rebuilds on resume via prefill) or swaps (copies blocks to/from CPU RAM over PCIe), both visible as mid-generation stalls to the user.
- Optimization levers: Grouped-query attention (GQA) reduces KV heads (e.g., 8 vs. 64), cutting cache proportionally; KV cache quantization to 8-bit halves memory at some quality cost; context length caps bound per-request consumption; and architectural research into sliding-window, sparse, and linear attention aims to change the growth curve itself.
Industry Insight
- Concurrency planning must account for context length distribution, not just model size—deployments that handle hundreds of short conversations may collapse under dozens of long-context ones on identical hardware.
- PagedAttention has become a de facto standard in modern serving engines (vLLM, TGI, etc.); any production LLM deployment should verify its use, as naive contiguous allocators waste the majority of available VRAM.
- Prefix caching across requests (reusing KV cache for shared system prompts and document contexts) represents the next major cost-reduction opportunity, as production traffic patterns reveal massive redundancy that current per-request isolation ignores.
Disclaimer: The above content is generated by AI and is for reference only.