AI Trending
All 全部 Deep Analysis 深度解析 Foresight 前瞻 AI News AI资讯 Open Source 开源项目 AI Products AI产品 Research Papers 论文研究 AI Security AI安全 AI Practices AI实践 AI Skills AI技能 AI Overseas AI出海
AI Practices AI实践 8h ago • Updated 5h ago 更新于 5小时前 49

Disaggregated prefill and decode for LLM inference on SageMaker HyperPod 在 SageMaker HyperPod 上对 LLM 推理进行解耦预填充与解码

Disaggregated Prefill and Decode (DPD) separates compute-bound prefilling and memory-bound decoding onto distinct GPU pools to eliminate interference between long-context processing and token generation. Implementation leverages Amazon SageMaker HyperPod, vLLM, and LMCache over Elastic Fabric Adapter (EFA) with RDMA to achieve negligible KV cache transfer latency (single-digit milliseconds). An intelligent router dynamically routes requests based on prompt length, sending short prompts directly 提出分离式预填充与解码（DPD）架构，将计算密集型预填充与内存密集型解码阶段隔离至独立GPU池，消除长上下文请求对并发生成的阻塞。 基于Amazon SageMaker HyperPod实现，利用vLLM、LMCache及EFA RDMA网络构建四层KV缓存传输栈，实现毫秒级跨节点数据移动。 引入智能路由机制，根据提示词长度阈值自动选择分离路径或端到端解码，支持前缀感知等策略以最大化缓存局部性并优化TTFT。 显著改善高并发流式工作负载下的尾部延迟控制，使首字延迟（TTFT）和令牌间延迟（ITL）可独立调优，提升长上下文RAG场景性能。

65
Hot 热度
75
Quality 质量
70
Impact 影响力

Analysis 深度分析

TL;DR

  • Disaggregated Prefill and Decode (DPD) separates compute-bound prefilling and memory-bound decoding onto distinct GPU pools to eliminate interference between long-context processing and token generation.
  • Implementation leverages Amazon SageMaker HyperPod, vLLM, and LMCache over Elastic Fabric Adapter (EFA) with RDMA to achieve negligible KV cache transfer latency (single-digit milliseconds).
  • An intelligent router dynamically routes requests based on prompt length, sending short prompts directly to decoders while directing long-context prompts through prefillers to optimize Time To First Token (TTFT) and Inter-Token Latency (ITL).
  • The architecture supports advanced caching strategies, including L1 CPU caching for recurring prefixes, significantly reducing redundant computation and improving throughput for high-concurrency streaming workloads.

Why It Matters

This approach addresses a critical scalability bottleneck in large language model inference, where colocated prefill and decode phases cause latency spikes for concurrent users when handling long prompts. By decoupling these phases, organizations can independently optimize resources for different workload characteristics, ensuring consistent performance for enterprise applications like RAG systems and agentic pipelines that require low-latency, high-throughput streaming responses.

Technical Details

  • Architecture Components: The solution comprises an intelligent router, prefiller pods, and decoder pods, orchestrated by the HyperPod Inference Operator. The router uses configurable token thresholds to determine routing paths, supporting strategies like prefixaware, kvaware, session, and roundrobin to maximize cache locality.
  • KV Cache Transfer Stack: Data movement utilizes a four-layer stack: LMCache PD (backend orchestration), NIXL (unified memory abstraction), libfabric (kernel-bypass provider), and AWS EFA (network interface). This enables GPU-Direct RDMA, keeping the host CPU out of the data path and minimizing transfer overhead.
  • Optimization Mechanisms: Prefillers employ layer-by-layer KV cache pushing with compute-transfer overlap to saturate GPUs. An L1 CPU cache within LMCache stores recurring prefixes (e.g., system prompts), allowing immediate retrieval without GPU recomputation. Decoders reserve specific GPU memory buffers (PD_BUFFER_SIZE) and utilize full CUDA graphs to start generation instantly upon transfer completion.
  • Performance Metrics: On ml.p5.48xlarge instances with 3,200 Gbps EFA, transferring an 8,000-token KV cache for models like Llama 3.3 70B takes single-digit milliseconds, making the network cost negligible compared to prefill compute time.

Industry Insight

  • Workload-Specific Deployment: Organizations should adopt DPD primarily for high-concurrency, long-context streaming workloads (e.g., RAG, chat assistants). For batch processing or short-prompt scenarios, colocated deployments remain more cost-effective due to the fixed overhead of cross-GPU KV transfers.
  • Infrastructure Requirements: Successful implementation requires infrastructure capable of high-bandwidth, low-latency interconnects (like EFA/RDMA). Teams must ensure their cloud environments support RDMA-capable networking to realize the performance benefits of disaggregation.
  • Operational Complexity vs. Gain: While DPD introduces architectural complexity via additional components (router, separate pods), it simplifies operational tuning by allowing independent scaling of prefill and decode resources. This decoupling provides more reliable tail latency control compared to traditional chunked prefill methods.

TL;DR

  • 提出分离式预填充与解码（DPD）架构，将计算密集型预填充与内存密集型解码阶段隔离至独立GPU池，消除长上下文请求对并发生成的阻塞。
  • 基于Amazon SageMaker HyperPod实现，利用vLLM、LMCache及EFA RDMA网络构建四层KV缓存传输栈，实现毫秒级跨节点数据移动。
  • 引入智能路由机制，根据提示词长度阈值自动选择分离路径或端到端解码，支持前缀感知等策略以最大化缓存局部性并优化TTFT。
  • 显著改善高并发流式工作负载下的尾部延迟控制，使首字延迟（TTFT）和令牌间延迟（ITL）可独立调优，提升长上下文RAG场景性能。

为什么值得看

该方案解决了大规模LLM推理中因预填充和共享GPU资源导致的“长提示阻塞”痛点，为高并发、长上下文场景提供了工业级的性能优化范式。对于需要部署大规模推理集群的AI从业者而言，理解DPD架构及其底层网络优化机制是提升服务吞吐量和降低延迟的关键。

技术解析

  • 架构分离与专用引擎：预填充（Prefill）被定义为计算密集型任务，并行处理整个输入提示以生成初始KV缓存；解码（Decode）被定义为内存密集型任务，逐令牌生成且需大量内存带宽访问权重和不断增长的KV缓存。两者运行在通过EFA连接的独立GPU池上，允许分配不同的并行策略。
  • 智能路由与自适应调度：部署了基于token阈值的智能路由器，长上下文提示走分离路径（预填充器->解码器），短提示直接由解码器处理以避免不必要的跨GPU传输开销。支持prefixaware、kvaware、session和roundrobin等路由策略以优化缓存命中率。
  • 高性能KV传输栈：采用四层传输架构（LMCache PD -> NIXL -> libfabric -> EFA）。NIXL提供统一的内存抽象，libfabric实现内核旁路和GPU-Direct RDMA，确保主机CPU不参与数据传输。在ml.p5.48xlarge实例上，8000令牌的Llama 3.3 70B模型KV缓存传输仅需数毫秒。
  • 预填充器优化与缓存机制：预填充器使用LMCache作为KV连接器，通过层叠发送重叠计算与传输以保持GPU饱和。引入L1 CPU缓存用于存储重复前缀（如系统提示、多轮历史），避免GPU重新计算，显著降低TTFT。解码器预留PD缓冲区并运行完整CUDA图，确保生成过程不受新请求干扰。

行业启示

  • 推理基础设施向专业化演进：随着LLM上下文长度增加，传统的共置部署模式已无法满足低延迟高并发需求，分离式架构（Disaggregated Inference）将成为大规模生产环境的标准配置，推动硬件资源按计算/内存特性精细化分配。
  • 网络带宽成为推理瓶颈的关键变量：DPD的成功依赖于极低延迟的网络传输（如EFA RDMA），这表明在构建分布式推理集群时，网络拓扑和RDMA性能优化与算法效率同等重要，需纳入整体架构设计考量。
  • 智能流量路由是提升资源利用率的核心：通过动态路由区分长短上下文请求，既能保障长文本服务的稳定性，又能简化短文本服务的部署复杂度，这种自适应调度策略有助于企业在混合负载场景下实现成本与性能的平衡。

Disclaimer: The above content is generated by AI and is for reference only. 免责声明：以上内容由 AI 生成，仅供参考。

LLM 大模型 Inference 推理 GPU GPU Deployment 部署
Read Original → 阅读原文 →

Related Articles 相关文章

AI Model Co-Design: Hardware-Friendly LLM Design AI模型协同设计：硬件友好的大语言模型设计
Deploying quantized models on Amazon SageMaker AI with Unsloth 使用 Unsloth 在 Amazon SageMaker AI 上部署量化模型
The Second Half of the AI War: No Longer About Who Has the Strongest Model, But Who Can Use It AI 战争下半场：不再比谁的模型强，比谁用得上
[GitHub] keras-team/keras Keras 3 多后端深度学习框架
Reducing High-Bandwidth Memory Bottlenecks in JAX-Based LLM Training with Host Offloading 通过主机卸载减少基于JAX的大语言模型训练中的高带宽内存瓶颈