The AI industry is entering a new phase where how efficiently you run models matters as much as how powerful they are. AMD and VAST Data announced an expanded collaboration to build AI infrastructure optimized for inference workloads, reasoning systems, and agentic AI applications. The shift reflects a fundamental change: monthly token consumption has increased 158 times in just two years as AI moves from experimental projects into production services.

Why Is Inference Becoming the Bottleneck?

For years, AI companies focused on training larger and larger models. But as AI moves into real-world applications, inference,the process of running a trained model to generate answers,is becoming the dominant workload. A single user request can now trigger multiple reasoning steps, sub-agent calls, data retrieval operations, and tool interactions, all of which demand more compute, memory, and data movement.

The challenge is that many AI data centers were built for training-centric workflows. They struggle to support the expanding context windows required by retrieval-augmented generation (RAG) pipelines, which pull information from external databases, and long-thinking inference where models reason through multiple steps before answering. This mismatch is driving up infrastructure costs, leaving GPUs underutilized, and creating operational complexity.

What Is Test-Time Compute and Why Does It Matter?

Test-time compute refers to allocating computational power during inference, not just during training. Rather than fixing how a model thinks, test-time compute allows models to spend more processing cycles reasoning through complex problems before delivering an answer. This approach is particularly valuable for agentic AI, where models must plan, retrieve information, and make decisions dynamically.

Moonshot AI's Kimi K3 model demonstrates this principle in action. Instead of simply scaling up model parameters, Moonshot optimized its reasoning traces,the intermediate thinking steps a model takes before responding,allowing the model to allocate computational power dynamically during inference. This design choice, combined with a highly optimized Mixture of Experts architecture that routes tokens through specialized subnetworks, enables state-of-the-art performance while keeping inference costs lower than traditional dense models.

How AMD Helios Redesigns the AI Rack

AMD's response is AMD Helios, a complete rack-scale system designed around how AI workloads actually flow through infrastructure. The system combines 6th Generation AMD EPYC "Venice" server CPUs, 5th Generation AMD Instinct MI455X GPUs, and AMD Pensando networking into a single co-designed platform. The result is 72 GPUs connected in one scale-up domain with 260 terabytes per second of bandwidth for frontier AI inference and training.

The performance gains are substantial. On DeepSeek-V4-Flash, one of the leading open-weight models available, AMD Instinct MI455X GPUs deliver up to 34 times higher token throughput at high interactivity and up to 18 times lower token cost compared with the previous generation MI355X GPUs. For context, this means the same inference task costs significantly less to run while delivering answers faster.

How VAST Data's AI Operating System Solves the Data Problem

AMD's hardware is only half the equation. VAST Data's AI Operating System treats inference as fundamentally a data problem. The platform consolidates storage, database, streaming, and AI services into a single software layer, eliminating the cold-start bottlenecks that plague traditional setups. When a model needs to load into GPU memory, VAST delivers near-instantaneous loading that maximizes GPU utilization.

Early testing demonstrated the impact. Using an AMD Instinct MI355X GPU, VAST achieved a 9 times speedup in time-to-first-token,the delay before a model starts responding,and 9.7 times more token throughput when using VAST for KV cache offloading with high concurrency agentic AI workloads. KV cache refers to the stored key-value pairs that models use to remember context during long conversations; offloading it to fast storage rather than keeping it all in GPU memory allows systems to handle more concurrent users.

Key Infrastructure Innovations in the Collaboration

• KV Cache Offloading: VAST's system moves cached data from GPU memory to NVMe SSD storage, enabling large-scale inference and agentic AI workloads to serve more concurrent users without exhausting GPU memory.
• Automated Data Lifecycle Management: VAST's native data policies automatically expire and delete sensitive KV cache data, helping enterprises meet compliance requirements without manual operational overhead.
• High-Performance Networking: The AMD Pensando Pollara 400 AI network interface card provides the data path connecting AMD Instinct GPUs to VAST storage, using both traditional and remote direct memory access protocols to move data efficiently.
• Reference Architectures: VAST, AMD, and networking partner DriveNets developed documented infrastructure blueprints for model training, inference, reinforcement learning, and KV cache workloads to help AI cloud providers and enterprises deploy systems faster.

"AI is entering an operational phase where infrastructure efficiency matters as much as model performance," said John Mao, Vice President of Global Technology Alliances at VAST Data. "The industry is discovering that inference is fundamentally a data problem. Success depends on how effectively organizations can bring data, compute, memory and intelligence together as a single system."

John Mao, Vice President of Global Technology Alliances at VAST Data

How This Shift Changes AI Economics

The practical impact is lower cost per token and faster response times. AMD Helios is designed to deliver up to 15 percent more AI compute, 50 percent more high-bandwidth memory capacity, and 50 percent more scale-out bandwidth compared with NVIDIA's Vera Rubin NVL72 rack. These gains compound across thousands of inference requests, translating to significant savings for AI cloud providers and enterprises running large-scale services.

The shift also reflects a broader industry realization: as inference becomes the dominant workload, the bottleneck is no longer raw compute but the ability to move data efficiently, manage memory intelligently, and support multiple concurrent users. Companies that optimize for these factors will have a cost advantage in production AI services.

What This Means for AI Development Going Forward

The AMD-VAST collaboration signals that the next generation of AI infrastructure will be purpose-built for inference and reasoning at test time, not retrofitted from training-centric designs. This matters because it enables new capabilities: longer reasoning chains, more sophisticated agentic AI, and retrieval-augmented generation at scale become economically viable when infrastructure is optimized for them.

The timing is significant. As monthly inference token consumption continues to grow exponentially, companies that fail to optimize their infrastructure for inference will face rising costs and operational complexity. Those that redesign around test-time compute, efficient data movement, and intelligent memory management will be better positioned to compete in the production AI era.