What Matters for Aggressive Decoding-Time KV Eviction? Temporal Aggregation and Ranking Preservation
Aggressive decoding-time KV cache eviction research has over-indexed on token scoring functions while neglecting the temporal aggregation rule that combines scores across decode steps EMA-based temporal aggregation makes approximately order-preserving scorer modifications largely indistinguishable at the eviction-set level, explaining why many scoring variants perform similarly Value-norm and entropy-based scorers remain highly correlated with attention and preserve retention sets, while KeyDiff
Analysis
TL;DR
- Aggressive decoding-time KV cache eviction research has over-indexed on token scoring functions while neglecting the temporal aggregation rule that combines scores across decode steps
- EMA-based temporal aggregation makes approximately order-preserving scorer modifications largely indistinguishable at the eviction-set level, explaining why many scoring variants perform similarly
- Value-norm and entropy-based scorers remain highly correlated with attention and preserve retention sets, while KeyDiff, key norm, recency, and learned scorers alter rankings and degrade substantially
- The authors introduce InertiaKV (EMA-based eviction) and InertiaKV-Lazy (periodic-refresh variant achieving 1.34-1.46x decode throughput), plus Score-Free decoding that freezes ranking after one initial score with negligible quality change (+0.03)
- Temporal aggregation and ranking preservation are identified as distinct, consequential design factors in KV eviction, separate from but not replacing the importance of scoring quality
Why It Matters
This work reframes a key assumption in the KV cache compression literature by demonstrating that the temporal aggregation strategy is as critical as the scoring function itself—a factor largely overlooked by prior research. For practitioners building or deploying long-context LLM systems, these findings suggest that investing in smarter aggregation rules and periodic-refresh strategies can yield significant throughput gains without sacrificing quality, potentially simplifying production KV eviction pipelines.
Technical Details
- EMA Aggregation Analysis: The paper systematically evaluates how exponential-moving-average temporal aggregation interacts with different scoring functions, showing that EMA smooths out ranking differences among order-preserving scorers (value-norm, entropy variants), making them produce nearly identical eviction sets
- InertiaKV: An EMA-based decoding-time KV eviction method that leverages temporal aggregation stability; InertiaKV-Lazy extends this with periodic refreshing of scores rather than continuous updates, achieving 1.34-1.46x decode throughput improvement over full-refresh InertiaKV
- Score-Free Decoding: A novel operating point where the full context is scored exactly once at the first decode step, the ranking is frozen, and all subsequent scoring is eliminated—resulting in only +0.03 average quality change across benchmarks
- Evaluation: Tested across six open-weight backbones on LongBench, LongBench-v2, and RULER benchmarks, comparing value-norm, entropy, KeyDiff, key norm, recency, and learned scoring variants under aggressive compression regimes
- Key Finding: The stability of EMA aggregation is attributed to its coupling of layer weighting and temporal retention, which dampens the impact of scoring modifications that preserve relative token order
Industry Insight
- KV cache eviction systems should prioritize temporal aggregation design alongside scoring functions; the paper's InertiaKV-Lazy approach offers a practical path to 1.34-1.46x throughput gains with minimal quality trade-offs
- Score-Free decoding presents a compelling low-complexity operating point for production deployments where near-constant scoring overhead can be eliminated after initial context evaluation, particularly beneficial for long-context inference workloads
- The finding that many scoring variants collapse to similar performance under EMA aggregation suggests the field should stop chasing marginal scoring improvements and instead invest in aggregation-aware eviction strategies and periodic-refresh mechanisms
Disclaimer: The above content is generated by AI and is for reference only.