OpenAI has unveiled its first custom-built inference chip, codenamed Jalapeño, which outperforms NVIDIA's leading systems in speed and energy efficiency across multiple large language models, including Moonshot AI's Kimi K2.5. The chip represents a major step in OpenAI's strategy to control its entire AI infrastructure, from model development through hardware and software optimization. Rather than relying solely on general-purpose graphics processors (GPUs), OpenAI designed Jalapeño specifically to handle the inference phase, where trained models respond to user requests in real time.

What Makes Jalapeño Different From Standard AI Hardware?

Most AI hardware systems force engineers to choose between two competing goals: processing more requests simultaneously (throughput) or responding to individual users faster (latency). Jalapeño attempts to achieve both simultaneously. The chip was co-designed with hardware company Cerebras and optimizes how data moves between memory, processing cores, and chips to minimize delays while maintaining high efficiency.

In benchmark tests using the public InferenceX standard, Jalapeño demonstrated significant advantages. On Moonshot AI's Kimi K2.5, a 1-trillion-parameter model, Jalapeño achieved approximately 1.5 times more AI work per watt of power compared to NVIDIA's GB300 system. More impressively, end-to-end latency, the time from when a user submits a request to receiving a response, dropped to roughly one-third of the comparison system's performance. At lower latency settings where interactive applications matter most, the advantage expanded dramatically, reaching up to 56.1 times better efficiency per watt.

The chip was tested on three publicly available models to demonstrate its versatility. Beyond OpenAI's own GPT-OSS 120B model, the tests included DeepSeek R1 (a 670-billion-parameter reasoning model) and Moonshot AI's Kimi K2.5. This multi-model testing suggests Jalapeño is not narrowly optimized for OpenAI's proprietary systems but functions as a more general-purpose inference accelerator.

How Does Jalapeño Solve the Latency-Throughput Trade-Off?

The key innovation lies in how Jalapeño handles different stages of language model inference. When a model processes text, it performs two distinct operations: the prefill phase, which requires heavy computational power to process an entire prompt at once, and the decode phase, which generates responses one token (roughly one word) at a time and is limited by memory bandwidth rather than raw computing power. Traditional systems optimize for one phase at the expense of the other.

Jalapeño's architecture explicitly manages where data lives in the system. Key-value cache data, which stores information the model needs to generate coherent responses, can be kept locally on the chip rather than shuttled back and forth across the network. This reduces the time spent moving data and allows the chip to adapt dynamically as the balance between prefill and decode workloads shifts during inference.

• Peak Throughput Performance: On Kimi K2.5, Jalapeño achieves 1.5 times more AI work per watt at peak throughput compared to current best-in-class systems.
• Latency Reduction: End-to-end response time on Kimi K2.5 drops to approximately one-third of comparison systems, with latency reduced by about 3.4 times.
• Interactive Workload Advantage: For highly interactive applications where users expect fast responses, Jalapeño delivers 2.1 to 4.1 times better performance per watt across tested models.
• Power Efficiency: The chip has a rated power consumption of 700 watts but operates continuously at 550 watts or below during typical workloads.

Why Should Companies Care About Custom AI Chips?

OpenAI's move reflects a broader industry trend. Google developed TPUs (Tensor Processing Units), Amazon built Trainium and Inferentia chips, Microsoft advanced its Maia processor, and Meta deployed custom AI accelerators. The underlying logic is straightforward: general-purpose GPUs offer flexibility but lack optimization for specific AI workloads. For companies running millions of inference requests daily, custom hardware tailored to their models and software systems can significantly reduce operational costs and improve user experience.

The timing is particularly relevant for AI agents, which are systems that break complex tasks into multiple steps. In agent workflows, even small latency improvements in individual requests accumulate across the entire task sequence. A response that takes 100 milliseconds instead of 300 milliseconds might seem minor in isolation, but across a 10-step task, that difference compounds into seconds of saved time. Jalapeño's latency advantages make it especially suited for this emerging use case.

"This capability is made possible by our deep partnership with Cerebras and their unique hardware architecture. In the future, this cooperation will further push the boundaries of the 'ultra-fast' experience," stated an OpenAI representative.

OpenAI, in partnership with Cerebras

What Do These Results Mean for the AI Industry?

Jalapeño's performance on Moonshot AI's Kimi K2.5 and DeepSeek's R1 model signals that OpenAI is building infrastructure designed to serve not just its own models but a broader ecosystem. This approach could reshape how AI companies think about hardware investment. Rather than competing solely on model capabilities, companies now compete on the efficiency and speed of deployment. OpenAI's willingness to publish results on competitors' models suggests confidence in Jalapeño's general-purpose capabilities.

The chip also underscores a shift in AI economics. Training large models remains expensive, but inference, the process of running trained models at scale, is becoming the dominant cost for deployed AI systems. A chip that cuts inference costs by 30 to 50 percent while improving response times could translate into billions of dollars in savings for companies operating large-scale AI services. For end users, faster and more efficient inference means more responsive AI assistants and lower-latency interactive experiences.