Quantization Is Four Decisions, Not One
Quantization notation (W<bits>A<bits>) is critical: W4A16 reduces memory bandwidth for decode-bound serving, while W8A8/W4A4 leverage dedicated low-precision hardware for compute-bound workloads NVFP4 uses 16-value blocks with fractional scales vs MXFP4's 32-value blocks with power-of-two scales, costing 5.9% more storage but enabling significantly better precision retention FP8 is effectively lossless across tested model sizes with only a 0.4-point MMLU-Pro drop (p=0.54, statistically indisting
Analysis
TL;DR
- Quantization notation (W
A ) is critical: W4A16 reduces memory bandwidth for decode-bound serving, while W8A8/W4A4 leverage dedicated low-precision hardware for compute-bound workloads - NVFP4 uses 16-value blocks with fractional scales vs MXFP4's 32-value blocks with power-of-two scales, costing 5.9% more storage but enabling significantly better precision retention
- FP8 is effectively lossless across tested model sizes with only a 0.4-point MMLU-Pro drop (p=0.54, statistically indistinguishable from BF16)
- Common claims about INT4 "gutting code generation" are not supported by the source paper (arXiv:2411.02355); INT4 actually gains 0.8 points on HumanEval vs BF16
- Quantization decisions in production are fundamentally capacity decisions first, with accuracy as the gatekeeper constraint
Why It Matters
This article provides a rigorous, source-grounded analysis of LLM quantization that corrects widespread misinformation circulating in the community about accuracy trade-offs. For AI practitioners deploying models at scale, understanding the distinction between memory-bound and compute-bound quantization strategies directly impacts throughput, batch size, and cost efficiency. The detailed breakdown of FP8, MXFP4, and NVFP4 formats gives engineers the technical specificity needed to make informed hardware and software stack decisions.
Technical Details
- Quantization notation: W
A separates weight quantization (frozen model parameters) from activation quantization (intermediate computation values). W4A16 cuts memory fetch by 75% for bandwidth-bound decode but retains 16-bit multiply operations; W8A8 and W4A4 activate dedicated GPU tensor core paths for proportional speedups - Format comparison: FP8 is natively supported on recent hardware with negligible accuracy loss. MXFP4 uses 32-value blocks with power-of-two scales (4.25 bits/value), while NVFP4 uses 16-value blocks with fractional 8-bit float scales (4.5 bits/value). NVFP4's E2M1 four-bit floats have magnitudes {0, 0.5, 1, 1.5, 2, 3, 4, 6}, and its fractional scale avoids the 40%+ stretching that MXFP4's power-of-two constraint introduces
- Accuracy evaluation methodology: The source paper (arXiv:2411.02355, "Give Me BF16 or Give Me Death?") reports Llama-3.1-70B-Instruct results showing INT4 gains 0.8 points on HumanEval and FP8 gains 0.3 points, with no collapse in any benchmark. Statistical significance testing on published percentages shows MMLU-Pro's 0.9-point INT4 drop yields p=0.16 and FP8's 0.4-point drop yields p=0.54—neither statistically significant
- Production deployment: In vLLM, quantization is enabled via
--quantization fp8and--kv-cache-dtype fp8. The KV cache sits on the same memory layer as weights, so 8-bit KV cache effectively halves cache footprint, doubling concurrent conversation capacity and enabling larger batch sizes that amortize weight fetches
Industry Insight
- The community's prevailing skepticism toward INT4 quantization appears to be based on misread or misattributed data; practitioners should verify accuracy claims against primary sources and run paired evaluations on their own internal test suites rather than relying on aggregate percentages
- NVFP4's 5.9% storage premium over MXFP4 is a worthwhile trade-off for workloads requiring maximum precision retention at 4-bit, particularly as hardware support matures—this format choice should be evaluated per-deployment rather than assumed
- Quantization strategy should be driven by the serving bottleneck: memory-bound decode workloads benefit most from weight-only quantization (W4A16), while compute-bound large-batch serving benefits from joint weight-activation quantization (W8A8 or lower), making the optimal choice dependent on actual traffic patterns rather than headline speed numbers
Disclaimer: The above content is generated by AI and is for reference only.