What Do You Do With a Model That's Too Big for Your GPU?
Model weights require approximately 1 GB per billion parameters at 8-bit precision, doubling to 2 GB at 16-bit; a 70B model in FP16 alone needs ~140 GB of weight memory before accounting for KV cache and other overhead Quantization reduces model size by lowering precision (FP32 → FP16 → INT8 → INT4), but introduces potential accuracy loss and must be evaluated against the model, hardware, and quantization method used Tensor parallelism splits computation within each layer across GPUs requiring f
Analysis
TL;DR
- Model weights require approximately 1 GB per billion parameters at 8-bit precision, doubling to 2 GB at 16-bit; a 70B model in FP16 alone needs ~140 GB of weight memory before accounting for KV cache and other overhead
- Quantization reduces model size by lowering precision (FP32 → FP16 → INT8 → INT4), but introduces potential accuracy loss and must be evaluated against the model, hardware, and quantization method used
- Tensor parallelism splits computation within each layer across GPUs requiring fast interconnects (NVLink), while pipeline parallelism splits by depth across stages, tolerating slower links but introducing pipeline bubbles
- CPU offloading moves model data between RAM and VRAM as a capacity solution, but incurs significant bandwidth penalties making it unsuitable for high-performance production inference
- Communication bandwidth becomes a first-class constraint once models are sharded; more GPUs do not automatically yield more performance due to collective operation overhead (all-reduce, all-gather, reduce-scatter)
Why It Matters
This article provides a foundational framework for understanding distributed inference architecture, which is critical for anyone deploying large language models in production. The clear distinction between sharding (solving model capacity) and replication (solving throughput capacity) directly addresses a common misconception that adding more GPUs is a monolithic strategy. For AI practitioners, the bandwidth hierarchy and the trade-off analysis between tensor and pipeline parallelism are essential for making informed hardware and architecture decisions.
Technical Details
- Memory estimation rule: Weight memory ≈ parameters × bytes per parameter; 8-bit = ~1 GB/B parameters, 4-bit = ~0.5 GB/B, 16-bit = ~2 GB/B. Actual VRAM requirements exceed raw weight estimates due to KV cache, activations, CUDA/runtime overhead, and communication buffers.
- Quantization methods: GPTQ, AWQ, and bitsandbytes are cited as common approaches. The precision trade-off is explicit: FP32 (280 GB for 70B) → FP16/BF16 (140 GB) → INT8 (70 GB) → INT4 (35 GB), with accuracy loss as the cost.
- Tensor parallelism (TP): Splits large tensor operations within each Transformer layer across GPUs. Requires the TP degree to evenly divide attention heads (or KV heads with grouped-query attention). Uses collective operations (all-reduce, all-gather, reduce-scatter) on the critical path. Best suited for fast intra-server NVLink connections.
- Pipeline parallelism (PP): Assigns contiguous groups of layers to different GPUs, creating stages. Communication occurs only between adjacent stages, reducing traffic significantly compared to TP. Introduces pipeline bubbles (idle stages) that require careful scheduling, especially for inference workloads.
- Bandwidth hierarchy: Compute capacity > memory bandwidth > communication bandwidth, with each tier roughly an order of magnitude apart. This hierarchy explains why TP is used within servers (fast NVLink) and PP across servers (slower network links).
- Data parallelism vs. sharding: Data parallelism replicates the full model across GPUs for throughput (useless when the model doesn't fit on a single GPU). Sharding splits the model to solve capacity constraints. The two address fundamentally different problems.
Industry Insight
- Production inference systems should prioritize quantization as the first optimization step before investing in distributed parallelism, as it can eliminate the need for sharding entirely and avoid communication overhead.
- The combination of tensor parallelism within servers and pipeline parallelism across servers represents the dominant architecture for serving very large models; engineers should design their deployment topology around the NVLink-to-network bandwidth gap rather than assuming uniform GPU connectivity.
- KV cache memory scaling with context length and concurrency is an often-underestimated constraint; capacity planning must account for the gap between raw weight size and actual VRAM requirements, which widens significantly under production load conditions.
Disclaimer: The above content is generated by AI and is for reference only.