ARCHead: Activation-Metric Residual Correction for Large Language Model Output Heads
ARCHead is a packed LM-head compressor that eliminates the need for dense BF16/FP16 output heads in quantized LLMs It combines a quantized low-rank core, group-wise INT4 residuals, and a low-rank correction fitted using an activation-derived metric On Qwen3-8B-Base, ARCHead reduces LM-head storage by 3.7-3.9x (using only 25.6% of BF16 head storage) while achieving 1.007 relative perplexity Naive INT4 quantization of the same storage budget yields significantly worse perplexity (1.14-1.16), demon
Analysis
TL;DR
- ARCHead is a packed LM-head compressor that eliminates the need for dense BF16/FP16 output heads in quantized LLMs
- It combines a quantized low-rank core, group-wise INT4 residuals, and a low-rank correction fitted using an activation-derived metric
- On Qwen3-8B-Base, ARCHead reduces LM-head storage by 3.7-3.9x (using only 25.6% of BF16 head storage) while achieving 1.007 relative perplexity
- Naive INT4 quantization of the same storage budget yields significantly worse perplexity (1.14-1.16), demonstrating the value of the residual correction approach
- ARCHead integrates seamlessly with existing block quantizers like AWQ and bitsandbytes, adding only 0.006-0.007 cross-entropy and less than 2% throughput overhead
Why It Matters
Weight-only quantization has been highly effective for compressing transformer blocks, but the LM-head remains a persistent bottleneck because naive quantization severely degrades vocabulary-logit distributions. ARCHead addresses this gap by providing a practical compression strategy for the output projection layer, enabling end-to-end quantized deployments without sacrificing generation quality. This is directly relevant to anyone deploying quantized LLMs in production who wants to minimize memory footprint without retaining large BF16 heads.
Technical Details
- Architecture: ARCHead decomposes the LM-head into three components: (1) a quantized low-rank core that captures the dominant projection structure, (2) group-wise INT4 residuals that encode fine-grained correction terms, and (3) a low-rank correction matrix fitted using an activation-derived metric that aligns the compressed head with actual input distributions
- Training/Calibration: The low-rank correction is fitted using an activation-derived metric, meaning it adapts to the statistical properties of hidden states rather than relying solely on weight statistics, which is critical for preserving logit distribution fidelity
- Benchmarks: Evaluated on Qwen3-8B-Base, achieving 1.007 relative perplexity with only 25.6% of the original BF16 head storage; storage-matched naive INT4 quantization yields 1.14-1.16 relative perplexity, a substantial degradation
- Integration: Designed to complement existing block quantizers (AWQ, bitsandbytes) that typically leave the LM-head untouched; replacing their retained BF16 head with ARCHead adds only 0.006-0.007 cross-entropy and less than 2% throughput change
- Storage Reduction: Achieves 3.7-3.9x compression of the persistent LM-head storage with no dense BF16 head required
Industry Insight
- Full-stack quantization becomes viable: ARCHead closes the last major gap in weight-only quantization pipelines by compressing the LM-head, enabling truly end-to-end quantized deployments without memory compromises on the output projection
- Activation-aware correction is a generalizable technique: The use of activation-derived metrics for fitting low-rank corrections could be extended beyond LM-heads to other sensitive projection layers in transformer architectures, potentially improving quantization robustness across model components
- Near-zero overhead makes adoption frictionless: With less than 2% throughput impact and minimal perplexity degradation, ARCHead can be drop-in integrated into existing quantization workflows (AWQ, bitsandbytes), making it immediately actionable for practitioners optimizing inference cost
Disclaimer: The above content is generated by AI and is for reference only.