The Ultimate Guide to LLM Inference Optimization - Part 2
LLM inference is split into two phases: prefill (compute-bound, measured by TTFT) and decode (memory-bound, measured by TPOT), with the KV cache being a major memory bottleneck during decode Grouped Query Attention (GQA) balances memory efficiency and modeling capacity by grouping query heads to share key-value heads, significantly reducing KV cache size without the accuracy drop of Multi-Query Attention PagedAttention treats KV cache memory management like OS paging, storing cache in fixed-size
Analysis
TL;DR
- LLM inference is split into two phases: prefill (compute-bound, measured by TTFT) and decode (memory-bound, measured by TPOT), with the KV cache being a major memory bottleneck during decode
- Grouped Query Attention (GQA) balances memory efficiency and modeling capacity by grouping query heads to share key-value heads, significantly reducing KV cache size without the accuracy drop of Multi-Query Attention
- PagedAttention treats KV cache memory management like OS paging, storing cache in fixed-size non-contiguous blocks and enabling block sharing across requests with identical prefixes, boosting GPU utilization from ~20-40% to ~96%
- Mixture of Experts (MoE) activates only a subset of parameters per token via gating, allowing trillion-parameter models to run efficiently with far fewer active parameters (e.g., 4B active out of 26B total)
- TurboQuant achieves ~6× KV cache compression without retraining by using PolarQuant for compression and Johnson-Lindenstrauss transforms to correct accuracy drift via sign-bit residuals
Why It Matters
These techniques directly address the two most critical bottlenecks in production LLM deployment: memory bandwidth during decoding and GPU memory fragmentation from KV cache management. For AI practitioners, understanding and implementing these optimizations can yield 2x-4x throughput improvements and dramatically reduce infrastructure costs, making large-scale LLM serving economically viable.
Technical Details
- Prefill vs. Decode Phases: Prefill processes all context tokens in parallel (compute-bound, limited by FLOPs), measuring Time To First Token (TTFT). Decode generates tokens sequentially (memory-bound, limited by bandwidth), measuring Time per Output Token (TPOT). The KV cache stores key-value pairs from prefill to avoid recomputation during decode.
- Efficient Attention Variants: Multi-Query Attention (MQA) shares a single key-value projection across all query heads, reducing KV cache but risking accuracy loss. Grouped Query Attention (GQA) groups query heads to share KV projections (num_heads % num_kv_groups == 0), offering a practical middle ground with significant memory and bandwidth savings.
- Mixture of Experts (MoE): Uses a gating network to route each token to top-k experts (e.g., k=2), processing tokens expert-by-expert rather than token-by-token. Output is a weighted sum: output[T] = Σ(gate_prob_i · E_i(T)). Models like Mixtral and Gemma-4-26B-A4B use this architecture for sparse activation.
- PagedAttention: Stores KV cache in fixed-size blocks (e.g., 16 tokens per block) with a block table mapping logical sequences to physical GPU addresses. Enables non-contiguous memory allocation, on-demand block allocation, block reuse for shared prefixes, and block freeing after request completion—reducing memory fragmentation from 60-80% to under 4%.
- TurboQuant Compression: PolarQuant rotates KV vectors into polar coordinates mapped to a fixed grid, eliminating stored quantization scales. Johnson-Lindenstrauss transform (QJL) converts residual errors to sign bits, providing unbiased accuracy correction. Achieves ~6× compression with no retraining, implemented in llama.cpp and VLLM.
Industry Insight
- The shift from model-level optimizations (quantization, pruning, distillation) to systems-level optimizations (PagedAttention, KV cache management) represents a maturation in the field—frameworks like VLLM and SGLang are becoming critical infrastructure that practitioners must master rather than optional tools.
- MoE architectures are becoming the default for production-scale models, but they introduce routing complexity and load-balancing challenges that require careful engineering; practitioners should evaluate token-level vs. batch-level routing tradeoffs when deploying such models.
- KV cache compression techniques like TurboQuant will likely become standard in inference frameworks within 12-18 months, enabling longer context windows (millions of tokens) on existing hardware without proportional memory cost increases—this will unlock new use cases in document processing and long-context reasoning.
Disclaimer: The above content is generated by AI and is for reference only.