Nvidia and Cerebras are showcasing inference chip performance numbers that look impressive on paper but don't reflect how these systems actually operate in production environments. Both companies recently announced eye-catching benchmarks at the Hot Chips conference in California, with Nvidia's Groq-3-based LPX racks hitting 3,400 tokens per second and Cerebras claiming nearly equivalent performance from its CS-4 accelerators. However, these numbers represent peak performance under unrealistic conditions that inference-as-a-service operators would never use in practice.

Why Are These Performance Numbers Misleading?

The core issue is that both companies are measuring performance at a batch size of one, meaning they're processing a single user request at a time. While this produces stunning speed metrics, it's economically impractical. Running inference this way would be like bragging about a car's top speed on a race track when you actually need it to haul cargo efficiently on a highway. The math reveals the real constraints: an Nvidia LPX rack with 256 processing units can realistically handle only about 12 concurrent requests at 100,000-token input length before running out of memory, not the unlimited throughput the marketing suggests.

Cerebras faces similar limitations. Each CS-4 rack contains three massive accelerators with 44 gigabytes of memory each, totaling 132 gigabytes of capacity. Despite this substantial memory pool, the system still maxes out at roughly 12 concurrent batch requests at the same input length. The bottleneck isn't computing power; it's memory capacity. Both systems simply cannot store the key-value caches, temporary data generated during processing, needed to serve more users simultaneously without adding more hardware.

What Actually Matters for Inference Economics?

The real performance metric that matters to operators is how efficiently a system scales across what researchers call the Pareto frontier, a curve that maps the tradeoff between speed per user and total number of concurrent users. High-throughput, low-interactivity applications sit on the left side of this curve, while ultra-low-latency, single-user scenarios sit on the right. The sweet spot for most businesses is in the middle, where you balance some latency for better throughput and cost-effectiveness.

This is why Nvidia, AMD, and AWS are all pursuing heterogeneous compute architectures, combining traditional GPUs with specialized inference accelerators like Groq's LPUs or Cerebras' waferscale chips. GPUs excel at high-throughput batch processing, while these SRAM-heavy accelerators shine as decode accelerators, handling the memory-intensive final stage of token generation. Nvidia even spent $20 billion to license Groq's technology and recruit its engineering team, signaling how seriously the company takes this hybrid approach.

How to Evaluate Real Inference Performance

• Benchmark Methodology: Look for benchmarks that test systems across multiple batch sizes and concurrent user scenarios, not just peak single-request speed. SemiAnalysis' InferenceX tool provides this fuller picture by measuring performance across the entire Pareto frontier, from bulk token processing to ultra-low-latency scenarios.
• Memory Constraints: Examine the actual memory capacity and how it limits concurrent requests when processing realistic input lengths. A system's peak speed is irrelevant if it can only serve a handful of users before running out of memory.
• System Integration: Evaluate how well specialized inference chips integrate with GPUs and other compute resources. Standalone performance means little; what matters is how the combined system performs at scale and cost.
• Power Efficiency: Compare performance per watt across different batch sizes and concurrency levels, not just raw throughput. A chip that consumes massive power for single-request scenarios may be economically unviable.

OpenAI's approach to custom inference hardware offers a contrasting perspective. Rather than chasing peak single-request speeds, the company developed its Jalapeño accelerator with a balanced architecture designed to handle three distinct phases of inference: prefill (compute-bound), draft (latency-bound), and verification (bandwidth-bound). This unified design avoids the problem of specialized accelerators sitting idle when workload demands shift, and it reduces power consumption by eliminating the need to move large key-value caches between different processor types.

Jalapeño features 216 gigabytes of HBM4 memory and delivers up to 3.4 million floating-point operations per second in mixed-precision format, along with 13.4 million operations per second in lower-precision FP4 format at 700 watts of power consumption. The system scales to 128 accelerators in a single rack and up to 2,048 ASICs in a 16-rack pod configuration, offering 27 exaFLOPS of aggregate performance and 32 petabytes per second of memory bandwidth.

What distinguishes Jalapeño is its memory-sliced, or NUMA-style, architecture. The chip contains 64 core slices, each paired with its own memory slice to guarantee predictable latency and bandwidth. This design eliminates conflicts that plague unified memory systems and keeps frequently accessed data close to the processors that need it. OpenAI also developed the chip using AI-assisted optimization, which delivered improvements of 56 percent for certain multiplier designs and 21 percent for floating-point operations compared to human baseline designs.

"Jalapeño's physical hardware around the Broadcom-made chip is set to be made by Celestica," according to reporting on OpenAI's Hot Chips presentation.

Tom's Hardware, reporting on OpenAI's Hot Chips 2026 presentation

The inference chip market is entering a critical phase where marketing claims and real-world performance are diverging sharply. While Nvidia and Cerebras are correct that their systems can achieve the speeds they're advertising, those speeds represent a narrow slice of possible operating conditions. The companies winning in inference will be those that optimize for the middle of the Pareto curve, where most actual revenue is generated, rather than chasing headlines with unrealistic peak performance numbers. Until benchmarks emerge that test these systems in realistic multi-user, multi-request scenarios, customers should view headline speed claims with appropriate skepticism.