DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization
DAMP is the first post-training quantization method for recurrent states in Gated DeltaNet (GDN) and Kimi Delta Attention (KDA) based language models, addressing the memory and latency bottleneck of FP32 recurrent states. Uniform quantization (INT8/FP8/INT4/NVFP4) severely degrades accuracy on complex reasoning tasks; the authors find quantization error is concentrated in a small subset of channels with stable relative decay strength across prompts. DAMP combines quantization-error energy and de
Analysis
TL;DR
- DAMP is the first post-training quantization method for recurrent states in Gated DeltaNet (GDN) and Kimi Delta Attention (KDA) based language models, addressing the memory and latency bottleneck of FP32 recurrent states.
- Uniform quantization (INT8/FP8/INT4/NVFP4) severely degrades accuracy on complex reasoning tasks; the authors find quantization error is concentrated in a small subset of channels with stable relative decay strength across prompts.
- DAMP combines quantization-error energy and decay-based persistence to identify high-risk channels during offline calibration, storing them at higher precision while quantizing the rest to INT8, achieving 9.9 bits per state value.
- Evaluated on Qwen3.6-35B and Kimi-Linear-48B across six benchmarks, DAMP maintains near-FP32 accuracy while reducing recurrent-state storage by 69.1%, accelerating the recurrent-state update kernel by up to 2.01x, and lowering full-model TPOT by up to 10.9%.
Why It Matters
As long-context language models shift from softmax attention to recurrent-state architectures like GDN and KDA to manage KV-cache memory growth, quantizing these recurrent states becomes critical for deployment efficiency. DAMP provides a practical post-training quantization pathway that preserves reasoning accuracy while delivering significant memory and latency gains, directly enabling more cost-effective inference for large-scale recurrent-state models.
Technical Details
- Problem framing: Recurrent states in GDN/KDA layers are stored in FP32, consuming substantial GPU memory and making state updates memory-bandwidth bound during autoregressive decoding.
- Key empirical findings: Uniform quantization fails — INT8/FP8 degrade complex reasoning accuracy, and INT4/NVFP4 collapse it to near zero. Quantization error energy is highly concentrated in a small subset of channels, and the relative decay strength of state channels remains stable across prompts and tasks.
- Method: DAMP performs offline calibration to identify high-risk channels using two signals: (1) quantization-error energy per channel, and (2) decay-based persistence (channels with stronger decay are more sensitive to quantization). High-risk channels are stored at higher precision; the remainder are quantized to INT8, yielding an average of 9.9 bits per state value.
- Evaluation: Tested on Qwen3.6-35B and Kimi-Linear-48B across six benchmarks spanning mathematical reasoning, general reasoning, and code generation. Results show near-FP32 accuracy retention, 69.1% recurrent-state storage reduction, up to 2.01x kernel speedup, and up to 10.9% TPOT reduction.
Industry Insight
- The finding that decay strength is stable across prompts suggests that channel-level precision allocation can be determined offline without per-input calibration, making DAMP deployable in production without runtime overhead.
- As the industry moves toward recurrent-state architectures (GDN, KDA) for long-context efficiency, mixed-precision quantization of recurrent states will become a standard optimization; DAMP establishes the first baseline and highlights that naive uniform quantization is insufficient for reasoning-heavy workloads.
- The 9.9-bit mixed-precision scheme offers a compelling accuracy-efficiency tradeoff that could be adopted by model providers as a default inference optimization for GDN/KDA-based models, particularly in memory-constrained or latency-sensitive deployment scenarios.
Disclaimer: The above content is generated by AI and is for reference only.