GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving
GrowPage introduces an on-demand KV budgeting framework that treats KV cache capacity as a dynamic runtime resource rather than a fixed per-request allocation It uses lightweight dual-timescale query summaries to capture both recent and long-term attention behaviors, enabling accurate estimation of demand evolution during decoding The framework integrates with PagedAttention's page-level memory abstraction, preserving continuous batching and prefix caching while dynamically acquiring or compress
Analysis
TL;DR
- GrowPage introduces an on-demand KV budgeting framework that treats KV cache capacity as a dynamic runtime resource rather than a fixed per-request allocation
- It uses lightweight dual-timescale query summaries to capture both recent and long-term attention behaviors, enabling accurate estimation of demand evolution during decoding
- The framework integrates with PagedAttention's page-level memory abstraction, preserving continuous batching and prefix caching while dynamically acquiring or compressing KV pages
- Experiments across multiple reasoning benchmarks and models demonstrate a superior performance-throughput trade-off compared to existing KV compression approaches
Why It Matters
Long-output reasoning workloads place unprecedented memory pressure on LLM serving systems, making KV cache management a critical bottleneck. GrowPage addresses a fundamental limitation in current systems—static KV budgets—by introducing adaptive, demand-driven capacity allocation that can significantly improve serving efficiency without sacrificing reasoning quality.
Technical Details
- Dual-timescale query summaries: GrowPage maintains lightweight summaries operating at two temporal scales to capture both immediate and sustained attention patterns, enabling accurate prediction of evolving KV demand during generation
- On-demand page allocation: At each capacity boundary, the system makes a binary decision—either compress KV states within the current allocation or acquire an additional physical page when broader attention demand emerges
- PagedAttention integration: The framework builds directly on PagedAttention's page-level memory abstraction, ensuring compatibility with existing optimizations like continuous batching and prefix caching
- Benchmark evaluation: Tested across multiple reasoning benchmarks and model architectures, demonstrating consistent improvements in the performance-throughput Pareto frontier over baseline KV compression methods
Industry Insight
- Serving infrastructure for reasoning-heavy LLMs (e.g., o1-style models, CoT systems) will increasingly need dynamic memory management; GrowPage's on-demand approach could become a standard pattern for next-generation inference engines
- The dual-timescale summary design offers a reusable blueprint for other runtime resource allocation problems beyond KV caching, such as dynamic compute or attention head pruning
- As reasoning models produce longer outputs, the gap between fixed-budget and adaptive-budget systems will widen—early adopters of on-demand KV management will gain meaningful cost and latency advantages in production deployments
Disclaimer: The above content is generated by AI and is for reference only.