NVIDIA has announced that its Groq 3 LPX inference accelerator is now in full production, delivering record-breaking token generation speeds for AI agents that need to process massive amounts of context while responding instantly. The system achieved 3,431 output tokens per second on a benchmark test using the Gemma 4 31B model with a 100,000-token context window, marking the fastest performance ever recorded for that model.

This breakthrough addresses a fundamental challenge in agentic AI: as agents work through multi-turn tasks, their context grows continuously. An agent writing code, debugging, testing, and iterating might accumulate hundreds of thousands of tokens across a single session. Without both speed and the ability to maintain long context, agents become sluggish and lose track of what they have learned, severely limiting their usefulness.

What Makes Serving Long Context at High Speed So Difficult?

Generating tokens quickly sounds straightforward, but combining ultrafast generation with massive context windows creates a complex systems engineering problem. Most inference systems use a technique called tensor parallelism, which splits computation across multiple chips to speed things up. However, at the tiny batch sizes required for responsive, single-user inference, the overhead of coordinating between chips can actually slow things down more than it speeds them up.

The bottleneck is something called "first bit latency." When data needs to move between chips, there are two time costs: the time to initiate the transfer and synchronize the chips, and the time to actually move the data. For small batches, the initiation overhead dominates, making coordination between chips a major drag on performance.

How Does Groq 3 LPX Solve This Problem?

Groq 3 LPX uses a deterministic execution model, meaning its compiler can see the entire system architecture in advance and schedule exactly how data will move between chips before the workload even begins. Rather than chips negotiating in real time about who sends what data when, the compiler produces a complete transfer schedule, similar to how a city planner might optimize traffic flow so pedestrians, cyclists, and cars move smoothly without needing to stop for each other.

This approach eliminates real-time arbitration overhead and allows data to move on the exact clock cycle when it is ready, rather than waiting for confirmation or contending with other transfers. The system includes 256 local processing units (LPUs), 128 gigabytes of total memory, and 96 chip-to-chip links per processor running at 112 gigabits per second each.

Why This Matters for AI Agents Right Now

Agentic systems represent a major shift in how AI is being deployed. Unlike traditional chatbots that simply respond to a single prompt, agents must complete multi-step tasks: they write code, call tools, inspect results, debug, and iterate. Each step generates tokens that become part of the context for the next step. Faster token generation means agents can complete each reasoning loop more quickly, maintaining responsiveness even as context grows.

The performance gains are substantial. Groq 3 LPX delivers approximately 4 times faster responsiveness for agents and latency-sensitive workloads compared to the nearest alternative platform, according to NVIDIA. On practical tasks like coding, this translates to work that might take hours completing in minutes.

How to Evaluate Test-Time Compute Improvements for Your Workload

• Context Window Requirements: Measure how much context your agents accumulate during typical sessions. If your agents work with 50,000 tokens or more, long-context performance becomes critical to responsiveness.
• Latency Sensitivity: Determine whether your use case requires near-instant responses for each agent step. Interactive coding, real-time tool use, and multi-turn reasoning benefit most from ultrafast token generation.
• Batch Size Patterns: Assess whether your inference workload typically handles single users or small groups. Systems optimized for single-user interactivity may not be ideal for high-throughput batch processing.
• Cost Per Request: Compare the total cost of serving agents on different platforms, accounting for both hardware costs and the value of faster completion times for time-sensitive tasks.

Nebius, a leading AI cloud provider, plans to bring Groq 3 LPX to its Nebius Token Factory production inference platform, making the technology available to developers through standard APIs without requiring migration to a new software stack. Groq, the purpose-built AI inference cloud company, is also planning to be among the earliest adopters of the platform.

"Inference is the growth engine of AI. NVIDIA Grace Blackwell and NVL72 revolutionized large language model inference with an unprecedented leap in performance and efficiency. Vera Rubin extends that vision with workload-optimized AI factory configurations designed for the era of agentic AI, advancing the performance frontier with LPX for ultrafast token generation," said Jensen Huang, founder and CEO of NVIDIA.

Jensen Huang, Founder and CEO at NVIDIA

The Groq 3 LPX accelerator integrates with NVIDIA's broader Vera Rubin platform, which includes multiple specialized chips and racks designed for different AI workload requirements. This modular approach allows organizations to build AI factories tailored to their specific needs, whether they are training frontier models or serving open-source models at scale.

The shift toward optimizing inference speed reflects a broader industry recognition that the bottleneck in AI deployment has moved from training to serving. As agents become more capable and more widely deployed, the ability to generate tokens quickly while maintaining long context will increasingly determine which platforms can deliver the responsive, intelligent systems that enterprises and developers need.