NVIDIA Vera Rubin and Blackwell Set a New Standard for Agentic AI Performance per Watt
SemiAnalysis AgentX is an open-source benchmark in the InferenceX suite that evaluates agentic AI inference by replaying production-style coding agent sessions, capturing long-context prefill, KV-cache reuse, tool-call gaps, and dynamic concurrency NVIDIA Vera Rubin NVL72 achieves up to 30x higher AI-factory throughput per megawatt compared to GB300 NVL72 on AgentX workloads GB300 NVL72 delivers up to 80x throughput-per-megawatt gains over H200 NVL8 for large MoE models like Kimi K3 2.8T on agen
Analysis
TL;DR
- SemiAnalysis AgentX is an open-source benchmark in the InferenceX suite that evaluates agentic AI inference by replaying production-style coding agent sessions, capturing long-context prefill, KV-cache reuse, tool-call gaps, and dynamic concurrency
- NVIDIA Vera Rubin NVL72 achieves up to 30x higher AI-factory throughput per megawatt compared to GB300 NVL72 on AgentX workloads
- GB300 NVL72 delivers up to 80x throughput-per-megawatt gains over H200 NVL8 for large MoE models like Kimi K3 2.8T on agentic inference
- Key efficiency drivers include MoE serving runtimes (SGLang, TensorRT-LLM, vLLM), DeepGEMM-based kernels, mixed-precision formats (MXFP4, MXFP8), the NVIDIA Dynamo session-aware serving stack, and NVLink scale-up fabric connecting 72 GPUs
- Real-world data shows agentic requests consume 15x more tokens than ordinary chat, with average prompt tokens per request growing fourfold across 100 trillion tokens of usage
Why It Matters
As AI agents become the dominant inference workload, traditional benchmarks using fixed sequence lengths no longer reflect real serving performance—making new evaluation frameworks like AgentX essential for hardware purchasing and optimization decisions. The dramatic throughput-per-watt gains across NVIDIA generations signal that energy efficiency, not just raw compute, is becoming the critical differentiator for AI infrastructure at scale.
Technical Details
- AgentX Benchmark Methodology: Uses prerecorded Claude Code sessions replayed turn-by-turn via the AIPerf client, preserving context accumulation, input/output sequence lengths, reasoning time, and tool-call latency to reproduce realistic KV cache pressure
- Key Metrics: Tokens per megawatt reported against four user-experience values—E2E normalized interactivity, standard interactivity, E2E latency, and time to first token (TTFT)—enabling trade-off analysis between throughput and responsiveness
- System-Level Optimizations: Leverages MoE serving runtimes (SGLang, TensorRT-LLM, vLLM), DeepGEMM-based kernels, mixed-precision formats (MXFP4, MXFP8), the NVIDIA Dynamo session-aware serving stack, and high-bandwidth NVLink scale-up fabric for 72-GPU coordinated rack-scale inference
- Generational Performance: GB300 NVL72 leads H200 by up to 40x tokens per megawatt on static 8K/1K DeepSeek-R1-0528 workloads, with gains extending to 80x for large MoE models on dynamic agentic workloads
- Shift in Benchmarking: Fixed-sequence-length scenarios have been demoted to "maintenance mode" in InferenceX as they no longer represent production agent traffic, which features variable request lengths, interleaved tool use, and growing context
Industry Insight
- AI infrastructure buyers should prioritize throughput-per-megawatt metrics on agentic workloads rather than static benchmark scores, as the gap between generations widens significantly under real agent traffic patterns
- The fourfold growth in prompt tokens and 15x increase in per-request token consumption for agentic workloads signals that KV-cache management and context reuse will be increasingly critical system design concerns
- Hardware vendors that integrate session-aware serving stacks and efficient MoE execution at the rack scale will gain a decisive advantage as agentic AI becomes the leading inference workload across the industry
Disclaimer: The above content is generated by AI and is for reference only.