NVIDIA has unveiled a dramatic efficiency breakthrough in AI inference that could reshape how companies deploy autonomous AI agents at scale. The company's new Vera Rubin NVL72 GPU system delivers up to 30 times higher throughput per megawatt of power than the prior GB300 NVL72 on agentic workloads, while reducing the cost per million tokens by up to 35 times. This efficiency gain matters because AI agents consume roughly 15 times more tokens than simple chat requests, making power consumption and token economics critical to profitability.

Why Do AI Agents Consume So Many Tokens?

Understanding the efficiency challenge requires knowing how AI agents actually work. When an AI agent researches a company for an investment decision, it queries financial databases, searches news filings, spawns sub-agents to run comparisons and valuations, then synthesizes everything into a recommendation. Each step feeds its output as input to the next step, accumulating context that can reach hundreds of thousands of tokens in a single session. This is fundamentally different from a simple chatbot conversation, which typically uses only 1,000 to 8,000 tokens per request.

NVIDIA measured the Vera Rubin performance using real-world agentic coding sessions from the SemiAnalysis AgentX workload, which preserved actual context growth, tool calls, and sub-agent spawning. The results show that Vera Rubin NVL72 delivers up to 30 times higher throughput per megawatt than GB300 NVL72 on the DeepSeek V4 Pro model, with similar gains across other agentic models including Kimi K3, MiniMax M3, GLM5.3, and Qwen3.5.

How Does NVIDIA Achieve This Efficiency Gain?

• Disaggregated Serving: The system separates context processing (prefill) from response generation (decode) so each can scale independently, preventing bottlenecks in either phase.
• Distributed KV-Caching: Key-value cache memory extends across the GPU domain, with less-active context tiered to host storage, keeping previously processed context accessible without recomputation.
• Large-Scale Expert Parallelism: For mixture-of-experts models, expert sub-networks distribute across the scale-up GPU domain, enabling efficient processing of specialized model components.
• KV-Aware Routing: Incoming requests route to GPUs that already hold relevant cached context, reducing redundant computation across long agent sessions.
• Fused CUDA Kernels: Operations like MegaMoE combine computation and inter-GPU communication into single execution passes, keeping GPUs active rather than waiting for data transfers.

The Vera Rubin architecture also includes enhanced fifth-generation Tensor Cores and a third-generation Transformer Engine that accelerate both prefill and decode stages of inference. NVIDIA's NVFP4 quantization compresses model weights to 4-bit precision, reducing memory footprint and increasing throughput without sacrificing output quality. The NVLink interconnect technology, now in its sixth generation, delivers 10 times higher packet rates and 3 times lower latency than off-the-shelf Ethernet alternatives, enabling the high-bandwidth, low-latency communication essential for these optimization techniques.

What Does This Mean for AI Cloud Providers?

For power-constrained AI factories, the efficiency gains translate directly into revenue and profit margin improvements. NVIDIA's DSX MaxLPS technology manages power across the GPU, rack, and workload levels to provision up to 40 percent more GPUs within the same megawatt budget, pushing throughput per megawatt further at AI factory scale. At 35 times lower cost per million tokens than GB300 NVL72, Vera Rubin NVL72 enables companies to run agents continuously and at scale across diverse workloads.

The efficiency breakthrough arrives alongside the announcement that NVIDIA's Groq 3 LPX, an interactive AI inference accelerator extending the Vera Rubin platform, is now in full production. Groq 3 LPX delivers 3,400 output tokens per second on the Gemma 4 31B model with a 100,000-token context, described as the fastest result ever recorded for this model. The accelerator provides up to 4 times faster responsiveness for agentic and latency-sensitive workloads versus the nearest alternative platform, enabling agentic tasks such as coding to complete in minutes rather than hours.

"Inference is the growth engine of AI. NVIDIA Grace Blackwell and NVL72 revolutionized large language model inference with an unprecedented leap in performance and efficiency. Vera Rubin extends that vision with workload-optimized AI factory configurations designed for the era of agentic AI, advancing the performance frontier with LPX for ultrafast token generation," said Jensen Huang, founder and CEO of NVIDIA.

Jensen Huang, Founder and CEO of NVIDIA

Nebius, a leading AI cloud provider, will be the first to adopt Groq 3 LPX through its Nebius Token Factory production inference platform, with AI cloud provider Groq planning to be among the earliest additional adopters. The Vera Rubin platform itself is a seven-chip architecture that includes the Vera CPU, Groq 3 LPU, NVLink 6 Switch, BlueField-4 DPU, Spectrum-6 Ethernet, and ConnectX-9 SuperNIC, all purpose-built for AI factories deploying agents at scale.

Why Does Token Generation Speed Matter for Agents?

Agentic systems can generate massive volumes of tokens across hundreds or thousands of inference steps, making faster token generation critical for agents to reason, act, and complete complex tasks in real time. Groq 3 LPX is purpose-built to extend Vera Rubin's interactivity, which is the rate at which tokens are generated for an individual user and determines how quickly an agent can complete each step of its work. Faster generation gives agents more time to inspect files, write and test code, call tools, verify results, and iterate while maintaining a responsive user experience.

"Generation is the phase of inference that determines how responsive an AI system actually is, and that's exactly what NVIDIA Groq 3 LPX is built to accelerate. As the first AI cloud bringing it to production via Nebius Token Factory, we're making sure every step of an agent's loop feels instant," stated Danila Shtan, chief technology officer of Nebius.

Danila Shtan, Chief Technology Officer of Nebius

The Vera Rubin NVL72 and Groq 3 LPX announcements represent NVIDIA's accelerated pace of innovation in addressing the specific computational challenges of agentic AI. With continuous software optimizations, performance across both Vera Rubin NVL72 and GB300 NVL72 will continue to improve, according to NVIDIA. For enterprises and developers building AI agents that need to operate at scale with responsive, real-time performance, these efficiency and speed improvements could fundamentally change the economics of deploying autonomous AI systems in production.