Output-Aware Rotation for INT2 KV-Cache Quantization
Introduces OptR, an output-aware rotation method that minimizes post-W_O attention-output error rather than optimizing pre-attention cache statistics or proxy errors Decomposes the post-output error into key- and value-induced terms and learns per-head orthogonal corrections through the full INT2 quantization and attention path Applies an attention-equivalent key reparameterization to reduce large channel-wise offsets without altering the softmax distribution Consistently improves both QuaRot an
Analysis
TL;DR
- Introduces OptR, an output-aware rotation method that minimizes post-W_O attention-output error rather than optimizing pre-attention cache statistics or proxy errors
- Decomposes the post-output error into key- and value-induced terms and learns per-head orthogonal corrections through the full INT2 quantization and attention path
- Applies an attention-equivalent key reparameterization to reduce large channel-wise offsets without altering the softmax distribution
- Consistently improves both QuaRot and OSCAR across three models and five reasoning/coding benchmarks while strengthening long-context retrieval
- Preserves the paged KV-cache format with negligible inference overhead
Why It Matters
Ultra-low-bit (INT2) KV-cache quantization is critical for enabling long-context LLM inference within practical memory and bandwidth constraints. By aligning the quantization objective with the actual end-to-end output error rather than intermediate proxy metrics, this work closes a fundamental gap between optimization targets and model performance, offering a more principled path toward deployable low-bit inference systems.
Technical Details
- Problem framing: Existing rotation-based INT2 methods optimize cache statistics or proxy errors computed before the complete attention readout, creating a mismatch with the actual error that propagates through attention and the output projection W_O.
- OptR core method: Minimizes post-W_O attention-output error by decomposing it into key-induced and value-induced error terms, then learns per-head orthogonal rotation corrections that account for the full INT2 quantization and attention computation path.
- Key reparameterization: Introduces an attention-equivalent key reparameterization technique that reduces large channel-wise offsets in the quantized KV cache while preserving the softmax attention distribution, ensuring correctness of attention outputs.
- Evaluation: Tested across three language models and five reasoning and coding benchmarks, showing consistent improvements over both QuaRot and OSCAR baselines, with particular gains in long-context retrieval tasks.
- System compatibility: Maintains the paged KV-cache format used by major inference frameworks, incurring negligible additional inference overhead compared to existing rotation-based quantization methods.
Industry Insight
- The misalignment between quantization optimization objectives and end-to-end output quality is a systemic issue likely affecting other low-bit quantization approaches; output-aware objectives should be considered as a general improvement principle.
- INT2 KV-cache quantization is approaching practical viability for long-context inference, and methods like OptR that preserve framework compatibility (paged KV-cache) will see faster adoption in production LLM serving systems.
- As context windows continue to expand, the memory and bandwidth savings from ultra-low-bit KV-cache quantization will become increasingly decisive, making output-aware rotation techniques a strategically important direction for both research and engineering teams working on efficient LLM deployment.
Disclaimer: The above content is generated by AI and is for reference only.