Topology-Aware Data Movement for Disaggregated GPU Inference
Disaggregated LLM inference requires transferring large KV caches (2.6 GB per 70B request) between prefill and decode GPU pools, creating a critical networking bottleneck at production scale. Existing systems (DistServe, Splitwise, Mooncake) use uniform RDMA, ignoring that inter-GPU bandwidth varies by 72x depending on physical topology: 900 GB/s (NVLink), 50 GB/s (InfiniBand), 12.5 GB/s (TCP). The proposed topology-aware transfer orchestrator discovers interconnect hierarchy at startup and sele
Analysis
TL;DR
- Disaggregated LLM inference requires transferring large KV caches (2.6 GB per 70B request) between prefill and decode GPU pools, creating a critical networking bottleneck at production scale.
- Existing systems (DistServe, Splitwise, Mooncake) use uniform RDMA, ignoring that inter-GPU bandwidth varies by 72x depending on physical topology: 900 GB/s (NVLink), 50 GB/s (InfiniBand), 12.5 GB/s (TCP).
- The proposed topology-aware transfer orchestrator discovers interconnect hierarchy at startup and selects optimal transport per transfer, achieving 3–18x latency reduction over uniform RDMA.
- Three co-designed mechanisms enable this: pipelined layer-by-layer transfer (hiding 60–85% of latency), NVLink domain-aware placement for MoE models, and CXL 3.0 memory expanders as a shared overflow tier.
- Full evaluation is deferred to hardware not yet available in GPU clouds; results are based on analytical bandwidth models and projected analysis across three architectures.
Why It Matters
As LLM serving scales to production workloads, disaggregated inference—separating prefill and decode across GPU pools—becomes increasingly attractive for resource efficiency, but KV cache transfer has emerged as a critical unsolved bottleneck. This work directly addresses a gap in the infrastructure stack that every major LLM serving framework currently overlooks, making it essential reading for anyone building or optimizing large-scale inference systems.
Technical Details
- Bandwidth heterogeneity: The paper documents a 72x bandwidth variation across physical relationships: NVLink intra-domain (900 GB/s), InfiniBand cross-node (50 GB/s), and TCP cross-datacenter (12.5 GB/s). Prior systems treat all transfers uniformly via RDMA, wasting the topology advantage.
- Topology-aware transfer orchestrator: Discovers the full interconnect hierarchy at startup and dynamically selects the optimal transport path for each KV cache transfer, rather than relying on a single default network path.
- Pipelined layer-by-layer transfer: KV cache is streamed in layers, overlapping transmission with ongoing prefill computation. This hides 60–85% of transfer latency behind useful work, dramatically reducing effective stall time.
- NVLink domain-aware MoE placement: For Mixture-of-Experts models, the system co-optimizes expert dispatch decisions with KV cache locality, ensuring that expert computations and cache transfers stay within high-bandwidth NVLink domains wherever possible.
- CXL 3.0 memory expanders: Proposed as a shared overflow tier providing 6x capacity over GPU memory at 86x lower latency than NVMe storage, bridging the gap between GPU memory and disk for KV cache spill scenarios.
- Evaluation approach: Due to the lack of multi-node heterogeneous clusters and CXL 3.0 hardware in academic/cloud settings, the paper presents analytical bandwidth models, component implementations, and projected performance across three architectures.
Industry Insight
- The 72x bandwidth variation across topologies means that infrastructure decisions about GPU placement and networking are not just cost considerations—they are fundamental performance determinants for disaggregated inference. Cloud providers and datacenter operators should prioritize NVLink-domain-aware scheduling as a first-order optimization.
- CXL 3.0 memory expanders represent a paradigm shift for KV cache management: treating memory as a tiered resource rather than a hard boundary could unlock new serving architectures. Teams should monitor CXL hardware availability and begin prototyping overflow strategies now.
- The pipelined layer-by-layer transfer technique is broadly applicable beyond this specific system—any disaggregated or pipeline-parallel inference framework can benefit from overlapping data movement with computation. This should be adopted as a standard optimization in next-generation serving runtimes.
Disclaimer: The above content is generated by AI and is for reference only.