OpenAI says its Jalapeño chip outperforms Nvidia in inference

OpenAI released benchmarks on Tuesday showing its custom Jalapeño inference chip outperforming Nvidia ↘ 0.59% and others in both throughput and energy efficiency, marking the AI company's most detailed public challenge yet to the dominant chipmaker's hardware supremacy.

The benchmarks, shared ahead of OpenAI's presentation at the Hot Chips semiconductor conference at Stanford, show Jalapeño delivering 1.5 to 1.9 times more AI work per watt and 1.7 to 3.6 times lower end-to-end latency than Nvidia's GB200 and GB300 systems when running models including DeepSeek R1, the 1-trillion-parameter version of Kimi K2.5, and OpenAI's own GPT-OSS 120B.axios+1

A Rack-Scale Inference Machine

The chip, co-developed with Broadcom ↘ 0.48% and first teased in June, was designed from scratch exclusively for LLM inference. At the system level, each Jalapeño rack contains 128 accelerators delivering 1.7 exaFLOPS of 4-bit compute, 27.5 TB of HBM4 memory, and nearly 2 petabytes per second of memory bandwidth.theregister

OpenAI VP of hardware Richard Ho told reporters the chip offers the "best of both worlds" with lower latency and higher throughput, as AI systems typically "have to make a trade-off between the two."theverge

According to SemiAnalysis, which ran its InferenceX benchmark suite on the chip in OpenAI's labs, Jalapeño achieved over 700 tokens per second per user at concurrency 1 on DeepSeek R1 — all without speculative decoding or prefill-decode disaggregation, techniques competitors rely on to boost their numbers.newsletter.semianalysis

Speed to Silicon — and Its Limits

OpenAI went from initial team hiring to manufacturing tape-out in roughly 16 months, with the chip taped out in November 2025. The company said AI assistance contributed to an 8 percent reduction in SIMD area and a 10 percent reduction in matrix-engine area during design.newsletter.semianalysis

Still, Jalapeño handles only inference, not training. "We're going to need a lot of compute," Ho said. "And Jalapeño is part of that. Cerberus is part of that. Nvidia is part of that. AMD Advanced Micro Devices, Inc. ↗ 1.74% is part of that."axios

Deployment and Competition

OpenAI plans limited deployment by end of 2026, with volume production ramping through 2027. The company is already developing a second-generation chip, with plans taking shape on a third.theverge+1

SemiAnalysis noted important caveats: the benchmarks used a relatively simple 8k-input, 1k-output workload rather than the more demanding agentic workloads that stress caching and routing, and comparisons against Nvidia's newer Vera Rubin architecture — which uses the same HBM4 memory — would be more apples-to-apples.newsletter.semianalysis

The chip joins a crowded field. Google Alphabet Inc. ↘ 1.08% has its TPU line, Amazon Amazon.com, Inc. ↘ 0.95% and Microsoft ↗ 0.66% have their own designs, and Anthropic confirmed earlier this month it is building an in-house chip effort. OpenAI has no plans to sell Jalapeño externally. "We have so much need for it that we can't imagine when we would be able to," Ho said.axios