JAXBench: Benchmarking Autonomous TPU Kernel Optimization
JAXBench is introduced as the first TPU-native benchmark suite for autonomous AI-generated kernel optimization, addressing a gap left by existing GPU-focused benchmarks. The suite comprises 50 JAX workloads, including 17 operators extracted from major production models (e.g., Llama-3.1, AlphaFold2) and 33 translated from KernelBench, with expert upper-bound baselines provided by Tokamax. Research demonstrates that providing target-specific context (curated TPU documentation) significantly improv
Analysis
TL;DR
- JAXBench is introduced as the first TPU-native benchmark suite for autonomous AI-generated kernel optimization, addressing a gap left by existing GPU-focused benchmarks.
- The suite comprises 50 JAX workloads, including 17 operators extracted from major production models (e.g., Llama-3.1, AlphaFold2) and 33 translated from KernelBench, with expert upper-bound baselines provided by Tokamax.
- Research demonstrates that providing target-specific context (curated TPU documentation) significantly improves AI correctness (from 5.8% to 37.3%) over model scale alone when generating Pallas kernels.
- The Autocomp beam-search pipeline achieved a 1.36x geometric mean speedup over XLA on the full suite and recovered most of the expert baseline performance (1.60x over XLA) on hand-tuned kernels.
Why It Matters
This release establishes a standardized evaluation framework for TPU kernel optimization, enabling the community to measure progress in autonomous code generation for hardware-specific accelerators. By providing curated datasets and baselines, it lowers the barrier for researchers to develop and test AI-driven compiler optimizations, fostering open-source contributions to high-performance computing infrastructure.
Technical Details
- Dataset Construction: JAXBench includes 50 JAX workloads; 17 are production operators from MaxText-based architectures (Llama-3.1, DeepSeek-V3, Mixtral, Mamba-2, AlphaFold2), while 33 are adapted from KernelBench with problem sizes optimized for TPU v6e MXU utilization.
- Baselines: Eight production operators include hand-optimized Pallas kernels from the Tokamax library, tuned for block sizes to serve as an expert upper-bound baseline (achieving up to 2.08x speedup over XLA).
- AI Evaluation: Four feedback-driven methods were evaluated using Gemini 3 Flash. Conditioning on curated TPU documentation increased per-sample correctness to 37.3% and solved 48/50 benchmarks.
- Performance Results: Autocomp’s beam-search pipeline reached a 1.36x geomean speedup over XLA on the full suite. On the 8 hand-tuned kernels, it achieved 1.60x speedup, though it trailed on specialized paged and ragged attention operators.
Industry Insight
- Context Engineering is Critical: For sparse DSLs like Pallas, providing curated, domain-specific documentation to LLMs yields higher returns than simply scaling model size. Practitioners should prioritize context curation in their AI coding workflows.
- TPU Optimization Gap: As TPU usage grows in large-scale inference and training, the lack of automated optimization tools is a bottleneck. JAXBench provides the necessary infrastructure to close this gap, similar to how KernelBench accelerated GPU optimization.
- Open Source Collaboration: The release of the benchmark, harness, and baselines invites community-driven improvements. Organizations relying on TPU efficiency should monitor and contribute to this ecosystem to benefit from faster, AI-generated kernel optimizations.
Disclaimer: The above content is generated by AI and is for reference only.