Tuning vLLM: What Every Setting Does to the Arithmetic 调优vLLM：每个设置对推理性能的影响

TL;DR

• vLLM's V1 scheduler unifies prompt and output token budgeting into a single token-count map, eliminating the old prefill/decode phase separation and enabling features like chunked prefill, prefix caching, and speculative decoding to compose without conflict
• Only six settings truly matter for production tuning, and most defaults are already well-calibrated for typical workloads
• --max-num-seqs controls concurrency ceiling (not batch size), with defaults of 1024 on H100/H200/B200 and 256 on A100/L40S or cards under 70 GiB
• --gpu-memory-utilization sets a memory fence (default 0.92); pushing to 0.95 reallocates 3% of GPU memory directly into KV cache with minimal risk
• --max-num-batched-tokens is the latency dial (default 8,192 through API server on H100), prioritizing decode tokens before prefill, and --kv-cache-dtype fp8 halves cache cost while recovering 97-98% of baseline accuracy

Why It Matters

This article reframes vLLM tuning from a checklist of flags into a physics-based understanding of what each setting actually controls, which is critical for practitioners who have likely inherited default configurations without understanding their trade-offs. The unified scheduler architecture is the foundational change that makes modern vLLM nearly configuration-free, meaning most tuning advice online is outdated. For production LLM serving, recognizing which knobs move which levers prevents misdiagnosis of latency, throughput, and memory issues.

Technical Details

• Unified token budget scheduler: The V1 scheduler maintains a single {request_id: num_tokens} map where prompt and output tokens consume the same budget, replacing the old two-phase prefill/decode model. This enables chunked prefill (splitting long prompts across steps), prefix caching, and speculative decoding to stack without conflicting special cases.
• --max-num-seqs as concurrency ceiling: Defaults are tiered by GPU: 1024 on H100/H200/B200, 256 on A100/L40S/under 70 GiB. The actual batch size is determined by vllm:num_requests_running, not this ceiling. Lowering it trades arithmetic intensity for bounded batch size and faster per-token response.
• --gpu-memory-utilization as a memory fence: At 0.92, vLLM allocates no more than 92% of GPU memory after accounting for CUDA context and NCCL buffers. The KV cache receives whatever remains after weights, activation peaks, and CUDA graph buffers are reserved. Raising to 0.95 directly increases cache capacity.
• --max-num-batched-tokens as the latency control: Defaults split by deployment mode and GPU tier (e.g., 8,192 API-server on H100 vs 16,384 offline). Decode tokens are served first from the budget, so prefill gets only the remainder. Smaller values protect TTFT for existing requests at the cost of new request latency.
• --kv-cache-dtype fp8: Halves KV cache memory per token, potentially doubling concurrent conversations. vLLM's April 2026 study showed 97-98% AUC recovery at 128K on needle-in-a-haystack across models from Llama-3.1-8B to Kimi-K2.5, with a 1-2 point reasoning trade-off. Backend compatibility must be verified at startup.

Industry Insight

• Most published vLLM tuning guides are obsolete because the V1 scheduler redesign made the majority of flags redundant; practitioners should audit their configurations against the six key settings rather than applying generic optimization checklists.
• The --performance-mode flag (throughput vs balanced vs interactivity) silently doubles certain defaults, so teams must verify actual running values rather than assuming documented baselines apply to their deployment.
• FP8 KV cache is production-ready for throughput-bound dense and MoE models, but teams should validate backend selection and re-run quality evaluations, as the default backend may change when FP8 is enabled.

TL;DR

• vLLM V1调度器采用统一token预算机制，将prefill和decode视为同一预算下的不同需求，使chunked prefill、prefix caching和speculative decoding等技术能够协同工作
• 六个关键配置参数（--max-num-seqs、--gpu-memory-utilization、--max-num-batched-tokens、--enable-prefix-caching、--kv-cache-dtype）分别控制请求并发上限、KV cache容量、每步计算量、前缀缓存和缓存精度
• FP8 KV cache可将缓存容量翻倍，在保持97-98%基准性能的同时显著降低内存占用
• vLLM默认配置已针对通用场景优化，但实际部署需根据硬件规格和流量特征进行针对性调整

为什么值得看

这篇文章为生产环境中的vLLM部署提供了实用的调优指南，帮助AI从业者理解各配置参数的实际影响而非盲目调整。对于需要优化推理性能和资源利用率的团队来说，这些技术细节具有直接的应用价值。

技术解析

vLLM V1调度器采用统一的token预算机制，将prompt和output tokens纳入同一预算框架，使chunked prefill、prefix caching和speculative decoding等技术能够无缝协同。这一设计消除了传统调度器中各组件需要特殊处理的复杂性。

关键配置参数包括：--max-num-seqs控制并发请求上限（H100/H200/B200默认1024，A100/L40S默认256）；--gpu-memory-utilization设置GPU内存使用上限（默认0.92），实际缓存容量取决于权重、激活峰值和CUDA graph buffers的占用；--max-num-batched-tokens控制每步处理量（H100离线16384/API 8192，A100离线8192/API 2048），优先服务已生成的请求；--enable-prefix-caching默认开启，通过精确匹配从首token开始重用缓存；--kv-cache-dtype支持FP8精度，可将缓存成本减半。

行业启示

vLLM的默认配置已针对通用场景优化，生产部署时应根据实际流量特征和硬件规格进行针对性调整，而非盲目套用默认值。FP8 KV cache等量化技术可在保持性能的同时显著降低内存占用，为大规模部署提供可行方案。