Start Here: The Words Everyone Uses About LLM Inference
LLM inference is fundamentally constrained by memory bandwidth, not arithmetic speed; producing one token from a 70B model requires reading ~70GB of weights, making decode inherently bandwidth-bound Batching is the primary optimization: multiple requests share the same weight fetch, so arithmetic intensity in decode equals the batch size, but attention/KV cache remains per-user and cannot be batched Mixture-of-Experts (MoE) models reduce active parameters but do not solve the serving bandwidth p
Analysis
TL;DR
- LLM inference is fundamentally constrained by memory bandwidth, not arithmetic speed; producing one token from a 70B model requires reading ~70GB of weights, making decode inherently bandwidth-bound
- Batching is the primary optimization: multiple requests share the same weight fetch, so arithmetic intensity in decode equals the batch size, but attention/KV cache remains per-user and cannot be batched
- Mixture-of-Experts (MoE) models reduce active parameters but do not solve the serving bandwidth problem at scale, since all experts must reside in GPU memory and high concurrency reactivates full-weight traffic
- The "ridge point" (arithmetic speed ÷ memory bandwidth) varies significantly by GPU: H100 = 296, A100 = 153, L40S = 419, meaning cheaper cards like L40S require larger batches to break even
- KV cache is the critical bottleneck for long-context serving; optimizations must target cache size reduction rather than arithmetic speed, since cache traffic is inherently per-user and unbatchable
Why It Matters
This article provides the foundational mental model for understanding LLM serving performance, explaining why certain optimizations work and others don't. For AI practitioners, it clarifies that tokens-per-second metrics are meaningless without batch size context, and that hardware selection decisions must account for the arithmetic-to-bandwidth ratio, not just peak FLOPS or hourly cost.
Technical Details
- Roofline model applied to inference: The ridge point (ops/byte ratio) determines whether a GPU is compute-bound or bandwidth-bound; prefill operates right of the ridge (compute-bound) while decode operates far left (bandwidth-bound)
- Prefill vs. Decode asymmetry: Prefill processes all prompt tokens in parallel (one weight read for N tokens), while decode generates tokens sequentially (one full weight read per token), creating a fundamental performance imbalance
- Batching mechanics: Weight matrices are request-agnostic, so a single 70GB fetch can serve up to 32+ concurrent requests; however, KV cache is conversation-specific and cannot be shared across requests
- MoE serving dynamics: Active parameters scale with experts fired per token (e.g., 8/256 for Qwen3.6-35B-A3B), but total parameters still occupy GPU memory; batching recovers efficiency only at impractically large concurrency (~9,500 users)
- Hardware-specific ridge points: H100 SXM = 296 ops/byte, A100 80GB = 153, L40S = 419; the L40S's low bandwidth-to-arithmetic ratio makes it harder to reach compute-bound operation despite lower cost
Industry Insight
- When evaluating GPU hardware for LLM serving, prioritize the arithmetic-to-bandwidth ratio over peak FLOPS or hourly pricing; the L40S appears economical but requires significantly larger batches to achieve comparable efficiency to H100
- Monitor actual running batch size (via vLLM metrics like
num_requests_running) rather than configured limits; most production systems operate at 3-10% of available arithmetic capacity, indicating severe underutilization - For long-context features, invest in KV cache optimization (compression, eviction policies, quantization) rather than seeking arithmetic speedups, since cache traffic is the unbatchable bottleneck that batching cannot resolve
Disclaimer: The above content is generated by AI and is for reference only.