[AINews] Hot Chips: OpenAI's Jalapeño, Cerebras CS-5, Groq 3 LPX, Apple M6
OpenAI unveiled Jalapeño, a custom inference chip that outperforms NVIDIA GB200/GB300 systems with 1.5–1.9× better work per watt and 1.7–3.6× lower end-to-end latency at a 700W TDP (running at ≤550W in tests) Jalapeño achieves these gains without relying on aggressive prefill/decode disaggregation or speculative decoding, suggesting a more balanced inference architecture that reduces the traditional throughput/latency tradeoff OpenAI used GPT-Astra + Codex to write and optimize low-level kernels
Analysis
TL;DR
- OpenAI unveiled Jalapeño, a custom inference chip that outperforms NVIDIA GB200/GB300 systems with 1.5–1.9× better work per watt and 1.7–3.6× lower end-to-end latency at a 700W TDP (running at ≤550W in tests)
- Jalapeño achieves these gains without relying on aggressive prefill/decode disaggregation or speculative decoding, suggesting a more balanced inference architecture that reduces the traditional throughput/latency tradeoff
- OpenAI used GPT-Astra + Codex to write and optimize low-level kernels, achieving 1.5–1.8× faster attention and MoE block implementations than human-expert-written code, signaling a new model-assisted compiler optimization paradigm
- Microsoft's AutoSaddler framework treats agent harnesses as code, patching prompts, tool configs, and control logic offline to deliver gains of +9.0 on GAIA2, +9.6 on SWE-Bench Pro, and +10.0 on Terminal-Bench 2.0
- Long-horizon software engineering remains critically unsolved: SWE Refactor Bench showed only a 5.4% survival rate across 520 runs of whole-repository migration tasks (C→Rust, Maven→Gradle, POSIX→WebAssembly)
Why It Matters
OpenAI's move to custom silicon marks a strategic inflection point where frontier labs are no longer strictly downstream of NVIDIA for inference economics, potentially reshaping the AI hardware supply chain. Simultaneously, the emerging evidence that harness quality can outweigh model choice in agent benchmarks forces practitioners to invest equally in evaluation infrastructure and system design rather than chasing marginal model improvements.
Technical Details
- Jalapeño chip specs: Custom inference ASIC rated at 700W TDP (operating at ≤550W), targeting Blackwell/Rubin-class systems; deployment in OpenAI's infrastructure begins by year-end with Gen 2 in development and Gen 3 underway
- Benchmark performance: 1.5–1.9× more work per watt at peak throughput, 1.7–3.6× lower end-to-end latency, and 2.1–4.1× higher performance on highly interactive workloads compared to NVIDIA GB200/GB300 systems
- Model-assisted kernel optimization: GPT-Astra + Codex were used to generate and optimize low-level kernels for three open-weight models, producing attention and MoE block implementations 1.5–1.8× faster than existing human-expert code within approximately two months
- AutoSaddler harness optimization: Microsoft-led approach that treats agent harnesses as patchable code, using failure traces to offline-optimize prompts, tool configurations, and control logic
- SWE Refactor Bench methodology: Measures whole-repository migration tasks across real projects (SQLite, zlib, libsodium) with a three-stage evaluation pipeline, revealing a 5.4% survival rate across 520 runs
Industry Insight
Frontier labs will increasingly pursue vertical integration into custom silicon to control inference economics, but foundry and packaging capacity (particularly TSMC CoWoS) remain hard bottlenecks that could delay widespread adoption beyond OpenAI's infrastructure. The AutoSaddler and harness variance findings suggest evaluation engineering should be treated as a first-class research discipline—poor harness design can invalidate benchmark comparisons and mislead model selection decisions. Practitioners should prioritize investing in robust agent scaffolding and standardized "Harness Card" disclosures before allocating additional resources to model upgrades.
Disclaimer: The above content is generated by AI and is for reference only.