NVIDIA's latest inference processor, the Vera Rubin NVL72, has fundamentally shifted how the AI industry measures performance. The chip delivers up to 3.7 times faster throughput than NVIDIA's previous GB300 NVL72 on demanding workloads like video understanding, while maintaining near-perfect efficiency when scaled across multiple racks. This matters because inference, the process of running an AI model to generate responses, has become the dominant cost driver for AI companies operating at scale.

What Makes Inference Economics So Critical Right Now?

For AI companies, the economics of inference have become as important as the quality of the models themselves. When a user asks ChatGPT a question or requests an image from an AI generator, that's inference. Unlike training, which happens once, inference happens millions of times per day. A company serving 100 million users needs infrastructure that can handle that volume efficiently, or costs spiral out of control. NVIDIA's latest results highlight three interconnected factors that determine whether an AI company thrives or struggles financially.

Higher system performance means more tokens, or chunks of text, generated per second. More tokens per second translates directly to more users served and more revenue. Efficient scaling means that when a company adds more hardware, throughput grows proportionally rather than plateauing. Continuous software optimization means extracting more value from existing infrastructure investments without buying new chips. All three levers matter, and NVIDIA's latest submission to MLPerf Inference v6.1, an industry-standard benchmark, demonstrates progress across all of them.

How Does Vera Rubin Achieve These Performance Gains?

The performance improvements come from a combination of hardware innovations and software refinements working in tandem. Vera Rubin features enhanced Tensor Cores, specialized circuits designed to accelerate matrix multiplication operations that power language models, and an upgraded Transformer Engine that speeds up both the prefill stage (processing the user's input) and the decode stage (generating the response). The chip also introduces NVFP4 precision, a lower-precision format that reduces the memory footprint of model weights and attention mechanisms without significantly degrading output quality.

On the software side, NVIDIA's engineering teams implemented disaggregated serving, a technique that separates prefill and decode operations and distributes them across multiple GPUs for maximum efficiency. For models like DeepSeek-R1 and Qwen3-VL, which use mixture-of-experts architectures, this approach allows large-scale expert parallelism, ensuring that different parts of the model run on different processors without bottlenecks. The interconnect, powered by sixth-generation NVLink technology, provides the high-bandwidth, low-latency communication backbone that makes these techniques effective at rack scale.

What Do These Benchmarks Actually Tell Us About Real-World Performance?

MLPerf Inference v6.1 tested Vera Rubin on two of the most demanding benchmarks in the suite: DeepSeek-R1, a reasoning model that performs complex multi-step thinking, and Qwen3-VL, a vision-language model that processes both text and images. On Qwen3-VL, Vera Rubin delivered up to 3.7 times higher throughput than GB300 NVL72 across offline, server, and interactive scenarios. On DeepSeek-R1, throughput was up to 2.5 times higher. These aren't marginal improvements; they represent a fundamental shift in how much work a single rack of hardware can accomplish.

The scaling efficiency results are equally striking. When NVIDIA scaled a DeepSeek-R1 submission from a single GB300 NVL72 rack with 72 GPUs to four racks with 288 GPUs, throughput grew nearly linearly, achieving 99 percent scaling efficiency in the offline scenario. This means that adding four times the hardware delivered nearly four times the throughput, a rare achievement in distributed computing. For comparison, many systems see efficiency drop to 70 or 80 percent at scale, meaning that adding more hardware yields diminishing returns.

How Are AI Workloads Changing What Benchmarks Measure?

The AI industry is beginning to recognize that traditional inference benchmarks miss a critical dimension of modern AI systems: reasoning and agentic behavior. AI agents, which reason, plan, and act across multiple steps to solve problems, are reshaping how inference performance should be measured. In preview testing on SemiAnalysis AgentX, a benchmark designed to capture agentic inference workloads, Vera Rubin NVL72 delivered 30 times better performance than GB300 NVL72. Additionally, MLPerf is introducing a new Endpoints benchmark specifically designed to measure agentic inference workloads beyond traditional throughput metrics.

This shift reflects a broader industry trend. As AI models become more capable at reasoning and planning, the bottleneck is no longer just how fast a model can generate tokens, but how efficiently it can orchestrate multiple steps of reasoning, planning, and action. The infrastructure that powers these systems must support not just raw throughput, but also the complex orchestration patterns that agentic AI requires.

Steps to Understanding Inference Performance in Your AI Infrastructure

• Measure Throughput, Not Just Latency: Throughput, measured in tokens per second, determines how many users you can serve simultaneously. A system that responds to one user in 100 milliseconds but can only handle 10 concurrent requests is less valuable than a system that responds in 150 milliseconds but handles 1,000 concurrent requests. Focus on tokens per second per dollar spent on hardware.
• Test Scaling Efficiency Across Multiple Racks: A system that performs well on a single GPU or single rack may not scale efficiently when you add more hardware. Test your infrastructure by doubling the number of GPUs and measuring whether throughput increases by roughly 2x. If it increases by only 1.5x or less, you have a scaling bottleneck that will limit your ability to grow.
• Monitor Software Optimization Velocity: Hardware performance is important, but software improvements can deliver 1.6x performance gains or more without any hardware changes. Track how frequently your inference framework receives updates and whether those updates improve performance on your specific models and workloads. A platform with rapid software iteration will outpace competitors over time.
• Plan for Agentic Workloads: If your AI applications involve reasoning, planning, or multi-step problem solving, traditional throughput benchmarks will underestimate the infrastructure you need. Test your systems on agentic workloads that reflect how users will actually interact with your AI agents, not just simple text generation tasks.

What Does This Mean for the Broader AI Infrastructure Market?

NVIDIA's latest results signal that the inference market is entering a new phase of competition. For the past year, much of the AI industry's focus has been on training larger models and pushing the boundaries of reasoning capabilities. But as those models move into production, the economics of inference have become the limiting factor for profitability. Companies that can serve users with lower latency and lower cost per token will have a significant competitive advantage.

The breadth of NVIDIA's partner ecosystem also matters. Nineteen partners submitted results on Vera Rubin systems, including cloud providers like Azure and Oracle Cloud Infrastructure, hardware manufacturers like Dell Technologies and HPE, and specialized AI infrastructure companies like CoreWeave and Nebius. This ecosystem participation suggests that Vera Rubin will be widely available across multiple cloud and on-premises deployments, not locked to a single vendor.

Software optimization continues to be a major driver of performance gains. Between MLPerf Inference v6.0 and v6.1, NVIDIA's software improvements delivered up to 1.6 times higher performance on Qwen3-VL, achieved through lower KV cache precision, additional kernel fusion, better kernels, and disaggregated serving with vLLM and NVIDIA Dynamo. Even after the official v6.1 submission deadline, NVIDIA continued optimizing, with post-submission results showing further gains on models like GPT-OSS-120B and DLRMv3. This suggests that the performance ceiling for current hardware is still rising, and companies that invest in software engineering will continue to extract more value from their infrastructure.

For organizations making AI infrastructure decisions, the lesson is clear: performance, scaling efficiency, and software velocity are the three pillars that determine long-term inference economics. A system that excels in one dimension but falters in another will create bottlenecks that compound over time. The companies that win in the inference era will be those that optimize across all three.