Your Second GPU Is Bought for the Cache, Not the Model
Adding GPUs requires diagnosing the actual problem: model too large to fit, insufficient throughput, or KV cache memory exhaustion—each demands a different hardware strategy Disaggregated prefill/decode (splitting prompt processing and token generation across separate GPU pools) dramatically improves goodput under tight latency targets, with reported gains of 2-7x requests per GPU and up to 498% capacity increase at 100ms inter-token targets KV cache transfer between pools is far smaller than ma
62
Hot
76
Quality
68
Impact
Analysis
TL;DR
- Adding GPUs requires diagnosing the actual problem: model too large to fit, insufficient throughput, or KV cache memory exhaustion—each demands a different hardware strategy
- Disaggregated prefill/decode (splitting prompt processing and token generation across separate GPU pools) dramatically improves goodput under tight latency targets, with reported gains of 2-7x requests per GPU and up to 498% capacity increase at 100ms inter-token targets
- KV cache transfer between pools is far smaller than many assume: ~36 KiB/token for sliding-window models at 16-bit (halved with 8-bit cache), crossing a 400 Gb/s NIC in ~6ms for an 8K-token prompt
- Four GPU parallelism strategies exist with distinct communication costs: data parallel (no sync for dense models), tensor parallel (all-reduce twice per layer, pools KV cache), pipeline parallel (small activation transfers but bubbles), and expert parallel (expensive all-to-all token dispatch)
- Chunked prefill is the cheaper first resort for single-card contention; phase splitting is the reliable answer when tight latency targets demand it
Why It Matters
This article provides the most practical framework I've seen for diagnosing GPU scaling decisions in LLM serving—distinguishing between capacity, throughput, and cache problems that are routinely conflated in production deployments. The disaggregated prefill/decode analysis with concrete transfer size calculations and benchmark comparisons gives practitioners the arithmetic needed to justify infrastructure investments and set realistic performance expectations.
Technical Details
- Four GPU addition reasons: (1) model weights don't fit one GPU → tensor/pipeline parallel, (2) need more throughput → data parallel replicas, (3) KV cache too small for long context → tensor parallel to pool cache or add a second card specifically for cache, (4) prefill/decode interference on shared hardware → disaggregate phases across separate GPU pools
- Parallelism strategies and tradeoffs: Data parallel replicates full models (no sync for dense, lockstep for MoE); tensor parallel splits matrices with all-reduce after attention and FFN, uniquely pooling KV cache across cards; pipeline parallel splits layers with activation passing and bubble management; expert parallel distributes MoE experts with phase-dependent all-to-all communication (prefill: bulk token dispatch, decode: single-token)
- Disaggregated prefill/decode architecture: Prefill pool handles prompt processing (arithmetic-bound, above roofline), decode pool handles generation (memory-bound, below roofline). KV cache transfers between pools via connectors like NIXL, configured with
--kv-transfer-configspecifying connector type and role - KV cache transfer sizing: For gpt-oss-120b (18 growing layers + 18 sliding-window), ~36 KiB/token at 16-bit; 8-bit cache halves this. A 2K-token prompt transfers ~38-76 MiB, 8K ~146-292 MiB, 1 GiB at ~29K-58K tokens depending on quantization. Sliding-window absence in older models (80 layers, head dim 128) yields 320 KiB/token—gibibyte at ~3,300 tokens
- Benchmark comparisons: DistServe reports 1.6-7.4x requests/GPU vs DeepSpeed-MII (chunked prefill baseline), 2.0-4.6x vs vLLM. Mooncake (Kimi platform) shows +498% capacity at 100ms ITL, +157% at 200ms, +59% at 300ms vs vLLM v0.5.1. NVIDIA Dynamo claims vary: "up to 30x requests" vs "30x tokens" are different metrics; Llama 70B shows only ~2x on same graphic. vLLM's own llm-d integration reports up to 70% higher tokens/sec on GPT-OSS with B200
Industry Insight
- The "up to 30x" disaggregated serving claims circulating in marketing materials conflate different metrics (requests vs tokens, projected vs measured, disaggregation vs parallelism re-tuning)—practitioners should demand the specific baseline, metric, and workload before adopting these figures for capacity planning
- The sliding-window attention generational divide is a critical infrastructure consideration: newer models keep KV transfer costs manageable (gibibyte at ~29K tokens), while the installed base of older dense models faces transfer bottlenecks at ~3,300 tokens, making disaggregation far more expensive to operate on legacy architectures
- The expert parallel load-balancing insight—where popular experts become bottlenecks and balancing is "bought with cache" memory—suggests that MoE deployments need co-design of routing, cache allocation, and parallelism strategy rather than treating these as independent configuration problems
Disclaimer: The above content is generated by AI and is for reference only.
LLM Inference GPU Quantization Deployment
Related Articles
The Ultimate Guide to LLM Inference Optimization - Part 2
Perplexity Ships Portable Computer on NVIDIA DGX Spark: Local Harness, OS-Enforced Sandbox, and Zero Per-Token Cost for Local Steps
[GitHub] keras-team/keras
The Second Half of the AI War: No Longer About Who Has the Strongest Model, But Who Can Use It
OpenAI's first custom chip "Jalapeño" reportedly beats Nvidia's Blackwell and Rubin in inference benchmarks