Stop Wasting GPU Memory: A Deep Dive Into vLLM's PagedAttention 停止浪费GPU内存：深入解析vLLM的PagedAttention

TL;DR

• PagedAttention reimagines GPU memory management for LLM serving by applying virtual memory paging concepts from classical operating systems to Key-Value (KV) Cache allocation
• It eliminates 60–80% memory waste found in traditional contiguous allocation by partitioning KV caches into fixed-size blocks (block size B=16) mapped non-contiguously via a Block Table
• Copy-on-Write (CoW) enables efficient parallel sampling by sharing physical blocks across branches until divergence, saving up to 30% of KV memory
• Beam search benefits from reference-counted block sharing and garbage collection, achieving 37–55% memory savings during pruning
• vLLM achieves a 2–4× throughput improvement on identical hardware by maximizing effective batch size through near-zero memory fragmentation

Why It Matters

This work directly addresses the primary bottleneck in LLM deployment—GPU memory waste—which is the dominant cost driver for production inference serving. By reducing KV cache fragmentation from ~70% to under 4%, PagedAttention makes high-throughput LLM serving economically viable on existing hardware, potentially saving organizations significant infrastructure costs. For AI practitioners, understanding this mechanism is essential for optimizing deployment pipelines and making informed decisions about serving framework selection.

Technical Details

• Core Architecture: PagedAttention partitions the KV cache into fixed-size KV Blocks (default block size B=16), where each block stores keys and values for a small token window. A Block Table maintains a mapping from contiguous Logical Blocks (model's perspective) to scattered Physical Blocks (GPU DRAM's perspective), enabling non-contiguous memory allocation.
• Single Sequence Generation: During the prefill phase, tokens are allocated into logical blocks that map to arbitrary physical addresses. During autoregressive decoding, new tokens fill remaining slots in the current physical block; once full, a new physical block is dynamically allocated. Memory waste is strictly bounded to the unfilled slots of the last block only.
• Copy-on-Write for Parallel Sampling: When multiple output branches share a common prompt prefix, their logical blocks reference the same physical blocks with a reference count. Upon divergence, vLLM copies only the diverging block to a new physical location and adjusts reference counts, enabling efficient branch-specific writes without duplicating shared history.
• Beam Search Optimization: Competing beams sharing common prefixes map to shared physical blocks. Pruned beams trigger reference count decrements; blocks reaching zero are immediately returned to the free pool. Only a single block copy is required per branching event, minimizing memory overhead during dynamic path selection.
• Performance Results: vLLM demonstrates 2–4× throughput improvement over traditional frameworks by increasing effective batch size through near-complete elimination of internal and external fragmentation, with memory waste reduced from ~70% to under 4%.

Industry Insight

• Organizations deploying LLMs should prioritize serving frameworks that implement virtual memory-style allocation (like vLLM) over traditional contiguous allocators, as the throughput gains translate directly into reduced GPU infrastructure costs and higher request capacity per cluster.
• The Copy-on-Write and reference-counting mechanisms introduced here are broadly applicable beyond LLMs—any autoregressive or tree-search workload with shared prefixes (e.g., multi-agent systems, recursive reasoning pipelines) could benefit from similar block-sharing strategies.
• As context windows continue to expand (128K+ tokens), memory fragmentation will become increasingly severe; PagedAttention's on-demand allocation model is essentially future-proof against this trend, making it a critical foundation for next-generation serving infrastructure.

TL;DR

• vLLM通过PagedAttention技术将GPU内存管理引入操作系统分页机制，解决LLM服务中KV Cache内存浪费问题
• 传统连续内存分配导致60%-80%的KV Cache内存浪费，PagedAttention通过Block Table映射将浪费降至4%以下
• 支持Copy-on-Write机制实现并行采样时30%的内存共享，Beam Search场景下可实现37%-55%的内存节省
• 最终在相同硬件上实现2-4倍吞吐量提升，显著降低LLM服务成本

为什么值得看

本文深入解析了vLLM的核心创新PagedAttention技术，揭示了LLM服务中GPU内存瓶颈的本质问题。对于AI从业者而言，理解这一技术有助于优化模型部署成本，提升推理效率，是当前大模型服务化落地的关键技术突破。

技术解析

PagedAttention核心架构：将GPU DRAM划分为固定大小的KV Blocks（默认block size B=16），通过Block Table建立逻辑块到物理块的映射关系，实现非连续内存分配。每个物理块存储固定数量的token的Key和Value向量，物理块可分散存储在GPU内存任意位置。

单序列生成优化：在Prefill阶段按需分配逻辑块并映射到物理块，解码阶段动态扩展。内存浪费仅局限于最后一个物理块未填满的槽位，相比传统预分配最大序列长度的方式，内存利用率从约30%提升至96%以上。

Copy-on-Write并行采样：多个输出分支共享相同prompt的KV Cache时，通过引用计数机制共享物理块。当分支 divergence 时，仅在块级别触发复制操作，将原始块引用计数减1，新分支写入独立物理块，实现高效的内存共享。

Beam Search动态管理：将Beam Search建模为动态进程树，共享历史路径的beams映射到相同物理块。被剪枝的beam释放逻辑块并递减引用计数，当引用计数归零时立即回收物理块至空闲内存池，避免频繁内存拷贝。

行业启示

LLM服务成本优化方向：GPU内存带宽和容量已成为LLM服务的主要瓶颈，而非单纯算力。采用PagedAttention等内存优化技术可显著降低推理成本，企业应优先关注内存效率而非仅追求FLOPS提升。

操作系统思想在AI系统的迁移价值：将经典OS的分页内存管理引入深度学习推理引擎，证明了跨领域技术迁移的巨大潜力。未来AI基础设施创新可更多借鉴传统计算机科学中的成熟设计模式。

服务框架选型建议：vLLM的PagedAttention技术已证明可带来2-4倍吞吐量提升，企业在部署LLM服务时应优先考虑支持该技术的服务框架，特别是在高并发、多beam搜索等场景下可获得显著成本优势。