A single rented graphics processing unit (GPU) can exceed the speed of specialized AI inference chips when properly optimized, suggesting that software configuration matters far more than hardware choice for many AI applications. A developer working with a NVIDIA B200 GPU achieved 1,366 tokens per second on the GPT-OSS-120B model by adjusting just four software settings, reaching speeds that surpassed Groq's specialized LPU (Language Processing Unit) chip and most commercial GPU providers.

Why Are Specialized AI Chips Losing Their Speed Advantage?

For years, companies like Groq, Cerebras, and SambaNova have built custom silicon specifically designed for AI inference, the process of running trained models to generate predictions or text. These chips promised dramatic speed improvements over general-purpose GPUs. Cerebras demonstrated this with its wafer-scale system, achieving 1,991 tokens per second on the same model, a performance that seemed to prove custom hardware's superiority.

But the real story is more nuanced. When the developer rented a B200 GPU on Modal, a cloud computing platform, the machine initially delivered only 411 tokens per second using default settings. This matched what most commercial GPU providers were achieving. After adjusting configuration parameters, the same hardware produced 1,366 tokens per second, representing a 3.32-fold improvement without changing any code or recompiling anything.

This gap reveals a critical insight: the frameworks and software libraries that manage GPU inference are not optimized for single-user, single-stream scenarios. They are built for throughput, meaning they prioritize serving many users simultaneously rather than maximizing speed for one person waiting on one response.

What Software Changes Unlocked the Performance Gains?

The developer made four specific adjustments to the SGLang inference framework, each contributing measurably to the speed improvement. The changes were straightforward configuration tweaks that required no kernel writing or compilation.

• Streaming Interval Adjustment: Modified the stream_interval setting from its default of 1, eliminating unnecessary detokenization and dispatch operations on every single token. This small change yielded a 4.5 percent improvement by reducing overhead on a 0.621-millisecond token budget.
• Speculative Decoding: Implemented n-gram speculative decoding, a technique where a draft model proposes multiple tokens and the main model verifies them in a single forward pass. This approach allowed the system to produce accepted tokens without re-reading the full 4.97 gigabytes of model weights for each token, mirroring the technique that gave Groq its speed advantage.
• Attention Mode Adjustment: Set the speculative_attention_mode flag from its default "prefill" setting to "decode," yielding a 9.7 percent performance boost. This flag controls how the attention mechanism operates during different phases of inference.
• N-gram Search Tuning: Adjusted the max_bfs_breadth parameter from its default value of 10 to 2, and reduced max_trie_depth from 18 to 8. These changes optimized the CPU-side n-gram proposer to reduce serialization bottlenecks with the GPU.

The developer noted that the shipped defaults were not mistakes. They were intentionally set for throughput serving, where many requests share GPU and CPU resources simultaneously. For a single user waiting on a single response, however, these defaults left approximately 40 percent of the GPU's potential performance on the table.

How Does This Compare to Specialized Inference Hardware?

The optimized B200 achieved speeds that positioned it competitively against the specialized inference chips. On the same GPT-OSS-120B model, the results showed:

• Groq LPU: The optimized B200 at 1,366 tokens per second exceeded Groq's published 476 tokens per second by 2.87 times.
• SambaNova RDU: The B200 outperformed SambaNova's 708 tokens per second by approximately 1.93 times.
• Cerebras Wafer: The B200 reached 69 percent of Cerebras's 1,991 tokens per second, the highest performance measured, though Cerebras maintains an architectural advantage by keeping model weights in SRAM instead of HBM (high-bandwidth memory).
• Commercial GPU Providers: The optimized B200 at 1,366 tokens per second vastly exceeded Google Vertex at 423, Databricks at 324, and Azure at 300 tokens per second.

Across all GPU providers measured, there was a 13-fold performance spread on identical hardware. Google Vertex achieved 423 tokens per second while another provider managed only 33 tokens per second, all running the same silicon. This variation demonstrates that software configuration and serving code account for the entire difference.

What Does This Mean for AI Infrastructure Decisions?

The findings challenge the conventional wisdom that specialized AI chips are necessary for fast inference. While Cerebras's wafer-scale approach does maintain a genuine architectural advantage through its use of SRAM, the gap between general-purpose GPUs and specialized inference chips is far smaller than published benchmarks suggest when software is properly optimized.

For organizations evaluating inference infrastructure, the implication is significant: before investing in specialized hardware, it may be worth exploring whether existing GPU infrastructure can be optimized through software tuning. The developer's work demonstrates that the abstraction layers in popular frameworks, while useful for general purposes, can obscure the actual capabilities of the underlying hardware.

The framework cannot tell you that it is the thing holding you back. It throws no error, everything looks fine, and the number it gives you is steady and repeatable, so it feels like physics when it is actually a configuration file. This means organizations relying on default settings may be operating at a fraction of their hardware's true capacity without realizing it.

How to Optimize GPU Inference Performance

• Measure Baseline Performance: Calculate the theoretical maximum throughput of your hardware by dividing total bandwidth by the bytes required per token. For the B200, this calculation showed a theoretical ceiling of 1,610 tokens per second, revealing that stock SGLang was achieving only 25.5 percent of theoretical maximum.
• Audit Framework Defaults: Review the default settings in your inference framework for parameters like stream_interval, scheduler_recv_interval, and attention modes. These are often optimized for multi-user throughput scenarios rather than single-stream latency.
• Implement Speculative Decoding: Deploy speculative decoding techniques, either with trained draft models or n-gram matching, to reduce the number of full weight reads required per token. This technique can provide substantial speedups with minimal code changes.
• Profile CPU-GPU Interaction: Identify serialization bottlenecks between CPU and GPU components, particularly in token proposal and verification steps. Tuning parameters like max_bfs_breadth can reduce overhead when CPU operations block GPU execution.
• Test Across Multiple Runs: Accept that some optimizations produce variable results depending on runtime conditions. The n-gram speculative decoding approach showed a 17 percent spread across identical configurations, so multiple measurements provide more reliable performance estimates than single runs.

The broader implication is that inference performance is not fixed by hardware alone. Organizations currently using GPU providers or considering specialized chips should first ensure their software stack is properly tuned. The difference between 411 and 1,366 tokens per second on identical hardware represents the kind of efficiency gains that can reduce costs, improve user experience, and eliminate the perceived need for specialized silicon in many scenarios.