PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression
PuzzleKV introduces a page-wise low-rank decomposition approach for KV cache compression, partitioning per-head KV caches into fixed-length logical pages to exploit local low-rank structure The method is training- and calibration-free, treating each completed page as an independent compression unit and decomposing pages within each layer and KV head At ~60% of original KV cache storage, PuzzleKV achieves >96% of Full KV performance across evaluated models and benchmarks, outperforming Global SVD
Analysis
TL;DR
- PuzzleKV introduces a page-wise low-rank decomposition approach for KV cache compression, partitioning per-head KV caches into fixed-length logical pages to exploit local low-rank structure
- The method is training- and calibration-free, treating each completed page as an independent compression unit and decomposing pages within each layer and KV head
- At ~60% of original KV cache storage, PuzzleKV achieves >96% of Full KV performance across evaluated models and benchmarks, outperforming Global SVD on RULER
- When combined with quantization for aggressive compression, PuzzleKV retains >93% of Full KV performance using only 18.7% of original storage
- Previous low-rank methods relied on fixed projection spaces from model weights, calibration activations, or shared bases over broad regions, which may miss fine-grained important information
Why It Matters
KV cache memory remains a critical bottleneck for long-context LLM inference, and this work offers a practical, zero-overhead compression strategy that requires no additional training or calibration. For practitioners deploying long-context models in production, PuzzleKV provides a plug-and-play solution that can significantly reduce memory footprint while preserving near-full performance, making extended context windows more feasible on constrained hardware.
Technical Details
- Page-wise decomposition: Each per-head KV cache is partitioned into fixed-length logical pages, where substantial low-rank structure is observed within individual pages, enabling independent compression of each page
- Training- and calibration-free: Unlike prior methods that derive fixed projection spaces from model weights or calibration activations, PuzzleKV requires no additional training data or calibration step
- Incremental autoregressive compression: During decoding, newly eligible pages are incrementally compressed while attention is computed directly over both dense and factorized pages
- Benchmarks: Evaluated on RULER and LongBench across multiple models and context lengths, showing substantial gains over Global SVD on RULER and competitive performance on LongBench
- Quantization compatibility: PuzzleKV can be combined with quantization to achieve aggressive compression ratios, retaining >93% Full KV performance at only 18.7% storage
Industry Insight
- The page-wise approach suggests that local low-rank structure in KV caches is underexploited; future compression methods could similarly benefit from localized rather than global decomposition strategies
- The training- and calibration-free nature of PuzzleKV makes it immediately deployable in existing inference pipelines without retraining, lowering the barrier for adoption in production LLM systems
- The strong performance at 60% storage with near-full accuracy, and the ability to push to ~19% storage with quantization, indicates that KV cache compression is approaching a practical trade-off frontier where memory savings no longer require significant quality compromises
Disclaimer: The above content is generated by AI and is for reference only.