How Does One GPU Serve Hundreds of Users at the Same Time?
Model weights are loaded once into GPU memory and shared across all concurrent requests; only per-request state (tokens, KV cache) is duplicated, enabling hundreds of users to share a single GPU LLM inference has two distinct phases with opposite hardware characteristics: prefill is compute-bound (processing many tokens in parallel), while decode is memory-bandwidth-bound (reading all weights to produce one token) Continuous batching rebuilds the batch at every token step, allowing finished requ
Analysis
TL;DR
- Model weights are loaded once into GPU memory and shared across all concurrent requests; only per-request state (tokens, KV cache) is duplicated, enabling hundreds of users to share a single GPU
- LLM inference has two distinct phases with opposite hardware characteristics: prefill is compute-bound (processing many tokens in parallel), while decode is memory-bandwidth-bound (reading all weights to produce one token)
- Continuous batching rebuilds the batch at every token step, allowing finished requests to exit and new ones to enter immediately, eliminating the idle slot waste of static batching
- KV cache memory, not compute capacity, is the primary bottleneck limiting concurrency; maximum concurrent users scale inversely with context length
- Chunked prefill mitigates the prefill/decode scheduling conflict by splitting large prompts into smaller chunks interleaved with decode steps
Why It Matters
Understanding LLM inference mechanics is essential for anyone deploying or optimizing generative AI systems, as the architecture directly determines throughput, latency, and cost efficiency. The counterintuitive finding that memory bandwidth—not compute—bottlenecks decode, and that KV cache—not GPU compute—limits concurrency, challenges common assumptions and guides better resource allocation decisions.
Technical Details
- Weight sharing architecture: Model weights are read-only during inference and loaded once into VRAM; each request only requires its own KV cache (attention state), tokens, and position bookkeeping, making the marginal cost of an additional user extremely small
- Prefill vs. Decode asymmetry: Prefill processes the entire input prompt (e.g., 5,000 tokens) simultaneously as a dense matrix operation, saturating GPU compute units; decode generates one token at a time, requiring a full weight read (~140 GB for a 70B model at 16-bit) per token, making it memory-bandwidth-bound at roughly 21 tokens/second on a single request
- Continuous batching: The scheduler rebuilds the batch before every forward pass, admitting new requests and releasing finished ones at the token level rather than waiting for batch boundaries, which dramatically reduces wasted GPU cycles from variable output lengths
- Chunked prefill: Long prompts are split into chunks and interleaved with decode steps, smoothing inter-token latency for streaming users while keeping compute units busier than pure decode scheduling would allow
- KV cache as the concurrency limiter: VRAM budget = model weights + KV cache + runtime buffers; since weights are fixed, the KV cache pool size—and thus maximum concurrent requests—is determined by context length, with 32K-token contexts reducing concurrency by an order of magnitude compared to 1K-token contexts
Industry Insight
- When sizing inference infrastructure, prioritize KV cache memory management and context-length-aware capacity planning over raw compute benchmarks; a GPU's effective concurrency is defined by your average context length, not its FLOPS rating
- Continuous batching should be considered a baseline requirement, not an optimization, for any production LLM serving system—static batching introduces unacceptable latency variance and GPU idle time given the high variance in LLM output lengths
- The prefill/decode scheduling conflict suggests that high-throughput deployments should evaluate chunked prefill or even separate GPU pools for each phase to balance TTFT and TPOT according to their specific latency SLAs
Disclaimer: The above content is generated by AI and is for reference only.