NVIDIA is reducing memory capacity on its Vera Rubin NVL72 rack-scale AI system to address skyrocketing memory costs that could consume nearly a third of the system's total bill of materials. The move reveals a critical tension in the AI hardware boom: as demand for computing power explodes, the specialized memory chips that power these systems have become so expensive that even the world's dominant AI chip maker must make painful compromises.

What's Driving NVIDIA's Memory Cost Crisis?

The Vera Rubin NVL72 racks represent some of the most powerful AI systems ever built, capable of processing 800,000 tokens per second on mixture-of-experts workloads, roughly 10 times faster than systems powered by NVIDIA's Blackwell GPUs. But this raw power comes at a steep price. According to financial analysis firm Bernstein, a single Vera Rubin rack costs approximately $9.1 million, with memory expenses representing the largest single cost driver.

The culprit is HBM4 memory, a specialized high-bandwidth memory chip essential for AI workloads. Prices for HBM4 are expected to reach $53 per gigabyte in 2027, according to Bernstein's estimates. Without intervention, memory costs could consume 29 percent of the total bill of materials for the Vera Rubin system, far exceeding the preferred threshold of 20 percent.

How Is NVIDIA Trimming Costs Without Gutting Performance?

Rather than accepting these inflated costs, NVIDIA is making strategic reductions to memory configurations across its Vera Rubin platform. According to analysis from GF Securities, the company is implementing several targeted cuts:

• SOCAMM Module Reduction: NVIDIA is cutting the Small Outline Compression Attached Memory Module capacity in half, shipping 96GB modules instead of the original 192GB configuration for the NVL72 racks.
• CPU Memory Scaling: The Vera CPUs will reduce memory from 54TB to 55TB down to 28TB, while GPU memory remains stable at 20.7TB of HBM4 per rack.
• LPDDR5X Optimization: By reducing LPDDR5X capacity to one-fourth of original specifications, NVIDIA could cut costs from $1.2 million down to as low as $293,000 per system.

These adjustments could reduce memory costs to between $293,000 and $586,000, bringing the overall memory cost percentage down from the problematic 29 percent to a more manageable level. The financial impact is substantial: cutting memory capacity could lower total Vera Rubin NVL72 rack costs by nearly 50 percent, according to GF Securities analysis.

NVIDIA's ability to weather this crisis stems partly from its foresight. The company signed long-term memory agreements ahead of the shortage that allowed it to avoid the worst supply constraints that have plagued other technology companies. However, even with these contracts in place, the sheer cost of HBM4 memory has forced the company to make difficult engineering decisions.

What Does This Mean for the Broader AI Infrastructure Market?

NVIDIA's memory challenges highlight a critical vulnerability in the AI infrastructure boom. While the company dominates GPU (graphics processing unit) markets through its CUDA software ecosystem, it remains dependent on memory suppliers for a crucial component. As demand for AI computing explodes, memory bottlenecks and price spikes threaten to make even the most advanced systems economically unviable.

The Vera Rubin memory cuts also underscore why specialized AI cloud providers like CoreWeave have emerged as serious competitors. CoreWeave has grown to $2.1 billion in quarterly revenue by focusing exclusively on deploying and optimizing large NVIDIA GPU clusters for major AI companies. While CoreWeave remains far smaller than AWS or Microsoft Azure, its ability to operate dense GPU systems efficiently and deliver them quickly has made it attractive to companies like Meta, OpenAI, and Anthropic.

However, CoreWeave faces its own challenges in this competitive landscape. The company recorded a $144 million operating loss and a $740 million net loss in its latest reporting period, compared to AWS's $14.2 billion in quarterly operating income. This financial gap means CoreWeave must carefully manage infrastructure investments and match most large capital expenditures with customer contracts, leaving less room for error than hyperscalers like Amazon and Microsoft.

The longer-term threat to NVIDIA's dominance may not come from cloud providers but from proprietary chips developed by hyperscalers themselves. AWS's Trainium and Maia chips, along with Microsoft's custom silicon, could eventually become competitive for recurring inference and training workloads, giving these companies control over both hardware economics and customer relationships. When that moment arrives, NVIDIA's ability to maintain premium pricing and market share will face its most serious test yet.

For now, NVIDIA's memory cost management represents a pragmatic response to near-term supply constraints. But the underlying issue remains: as AI systems grow more powerful and more expensive, the economics of AI infrastructure will increasingly depend on solving the memory problem, not just the computing problem.