Affix Cache for Diffusion Large Language Models
Diffusion Large Language Models (DLLMs) face a fundamental inference efficiency challenge: bidirectional attention couples shared context tokens with evolving generated tokens, making naive KV cache reuse stale while full recomputation is prohibitively expensive. ACache introduces an affix-oriented cache reuse mechanism that identifies a small, request-specific subset of critical "Anchor Tokens" by measuring their influence on masked generation tokens, selectively recomputing only these tokens w
Analysis
TL;DR
- Diffusion Large Language Models (DLLMs) face a fundamental inference efficiency challenge: bidirectional attention couples shared context tokens with evolving generated tokens, making naive KV cache reuse stale while full recomputation is prohibitively expensive.
- ACache introduces an affix-oriented cache reuse mechanism that identifies a small, request-specific subset of critical "Anchor Tokens" by measuring their influence on masked generation tokens, selectively recomputing only these tokens while reusing the rest of the affix cache.
- Built on Fast-dLLM, ACache recovers accuracy loss from direct affix-cache reuse across different settings by recomputing approximately 20% of affix tokens.
- A shared-prefix prototype on Nano-vLLM demonstrates that ACache reduces recompute latency by up to 55.7% and improves end-to-end throughput by up to 1.68×.
Why It Matters
This work addresses a critical bottleneck in deploying Diffusion LLMs at scale—efficient inference with shared context. As DLLMs gain traction for their non-autoregressive decoding and bidirectional modeling advantages, the inability to reuse cached computations for overlapping text spans has been a major barrier to production viability. ACache provides a practical, accuracy-preserving solution that could accelerate the adoption of DLLMs in real-world serving systems.
Technical Details
- Core Problem: In autoregressive LLMs, KV caches for shared prefixes can be directly reused. In DLLMs, bidirectional attention means the KV states of shared context tokens are entangled with the evolving generated tokens, so naive cache reuse produces stale values, while full recomputation is computationally expensive.
- Anchor Token Identification: ACache measures the influence of affix tokens on masked generation tokens to identify a request-specific subset of critical tokens ("Anchor Tokens") that must be recomputed, while the remaining affix tokens can safely reuse cached KV states.
- Implementation: Built on top of Fast-dLLM, ACache selectively recomputes KV states for only the Anchor Tokens (~20% of affix tokens) and reuses cached states for the rest, recovering accuracy loss from direct cache reuse.
- Performance Results: A shared-prefix prototype on Nano-vLLM shows up to 55.7% reduction in recompute latency and up to 1.68× improvement in end-to-end throughput.
- Scope: The mechanism generalizes beyond simple shared prefixes to arbitrary shared text spans within DLLM inference.
Industry Insight
- The affix cache reuse paradigm could become a foundational optimization for any non-autoregressive generative model that relies on bidirectional attention, extending beyond language models to multimodal diffusion architectures.
- The 20% recomputation threshold suggests a strong accuracy-latency tradeoff sweet spot; serving systems that can dynamically adjust this ratio based on workload characteristics could achieve significant cost savings without quality degradation.
- As DLLMs continue to compete with autoregressive models on quality, inference efficiency optimizations like ACache will be decisive in determining which architecture wins on total cost-per-token in production deployments.
Disclaimer: The above content is generated by AI and is for reference only.