OCGQuant: Outlier-Companion Grouping for NVFP4 Quantization
OCGQuant introduces a novel post-training quantization (PTQ) method specifically designed for NVFP4 microscaling format, addressing activation outlier degradation in low-bit inference The key innovation is Outlier-Companion Grouping (OCG), which adaptively pairs outlier channels with low-magnitude companion channels to improve block composition and reduce Collateral Quantization Error Experiments on Llama3 and Qwen3 demonstrate OCGQuant achieves the lowest WikiText-2 perplexity and highest avera
Analysis
TL;DR
- OCGQuant introduces a novel post-training quantization (PTQ) method specifically designed for NVFP4 microscaling format, addressing activation outlier degradation in low-bit inference
- The key innovation is Outlier-Companion Grouping (OCG), which adaptively pairs outlier channels with low-magnitude companion channels to improve block composition and reduce Collateral Quantization Error
- Experiments on Llama3 and Qwen3 demonstrate OCGQuant achieves the lowest WikiText-2 perplexity and highest average downstream accuracy among evaluated PTQ methods
- The method maintains prefill speedup close to Round-to-Nearest (RTN) and matches its peak decoding memory, making it practically deployable
Why It Matters
NVFP4 quantization is becoming increasingly important for efficient low-bit inference, yet activation outliers remain a persistent challenge that degrades accuracy. OCGQuant provides a targeted solution that doesn't require additional computation overhead, making it directly applicable for practitioners deploying quantized models in production. This work bridges the gap between theoretical quantization accuracy and practical deployment constraints.
Technical Details
- Collateral Quantization Error: The authors formally define the reducible error incurred by remaining block values when the block scale is dominated by outlier activations, providing a new analytical framework for understanding NVFP4 quantization limitations
- Outlier-Companion Grouping (OCG): An adaptive channel-pairing strategy that groups high-magnitude outlier channels with low-magnitude companion channels within each quantization block, optimizing the distribution of values sharing the same scale
- NVFP4-specific design: Unlike existing PTQ methods using mixed precision, rotation, or residual compensation (which are either not tailored to NVFP4 or add computation), OCGQuant is specifically engineered for the NVFP4 microscaling format
- Evaluation: Tested on Llama3 and Qwen3 models with benchmarks including WikiText-2 perplexity and downstream accuracy, comparing against existing PTQ methods while measuring prefill speedup and peak decoding memory
Industry Insight
- The channel-grouping perspective offers a new paradigm for tackling activation outliers in low-bit quantization that could be extended to other microscaling formats beyond NVFP4
- The balance between accuracy improvement and computational efficiency (matching RTN's speed and memory characteristics) makes this approach highly attractive for production deployment of quantized LLMs
- As NVFP4 gains adoption in hardware accelerators, PTQ methods specifically designed for this format will become increasingly valuable, and OCGQuant sets a new performance baseline for the field
Disclaimer: The above content is generated by AI and is for reference only.