Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers
Looped Latent Attention (LLA) introduces a post-training cache codec that compresses K/V caches in weight-tied Transformers by exploiting the low-rank structure of recurrence steps. The method uses SVD initialization and KL/attention-output distillation to reconstruct loop-specific vectors on-demand, achieving up to 21.3x compression without performance loss. LLA significantly increases batch capacity (from 32 to 768 sequences at 4k context on H200) and improves reasoning accuracy on long math r
Analysis
TL;DR
- Looped Latent Attention (LLA) introduces a post-training cache codec that compresses K/V caches in weight-tied Transformers by exploiting the low-rank structure of recurrence steps.
- The method uses SVD initialization and KL/attention-output distillation to reconstruct loop-specific vectors on-demand, achieving up to 21.3x compression without performance loss.
- LLA significantly increases batch capacity (from 32 to 768 sequences at 4k context on H200) and improves reasoning accuracy on long math rollouts (MATH-500 score increase).
- The approach demonstrates that recurrent caches are low-rank but not safely collapsible to a single state, requiring structured compression rather than simple sharing.
Why It Matters
This research addresses a critical bottleneck in efficient Transformer deployment: memory overhead during autoregressive decoding. By enabling high compression ratios for weight-tied models, LLA allows practitioners to run larger batches or longer contexts on existing hardware, directly impacting cost-efficiency and scalability for inference-heavy applications.
Technical Details
- Mechanism: LLA stores compact K and V latents instead of full loop-indexed caches, reconstructing specific vectors only when attention requires them.
- Compression Modes: Default per-head codec compresses recurrence; LLA-2D folds heads into one latent for extreme compression regimes.
- Training: Codec initialized via SVD of teacher activations and refined using KL divergence and attention-output distillation.
- Performance: Outperforms head-axis MLA, cross-layer sharing, KV quantization, and final-loop reuse at matched cache budgets. Near-lossless compression up to 32x observed on Huginn-3.5B.
- Hardware Impact: On an H200 GPU, increased batch capacity from 32 to 768 sequences at 4k context length with 21.3x compression.
Industry Insight
- Efficiency Gains: Organizations deploying weight-tied or recurrent Transformer variants can significantly reduce VRAM requirements, enabling higher throughput without hardware upgrades.
- Long-Context Viability: The ability to maintain performance over long rollouts (e.g., math reasoning) suggests LLA is particularly valuable for complex, multi-step inference tasks where context window limits are often hit.
- Post-Training Applicability: Since LLA is a post-training codec, it offers a low-friction path for optimizing existing models without retraining from scratch, accelerating adoption in production environments.
Disclaimer: The above content is generated by AI and is for reference only.