The artificial intelligence market is projected to exceed $3.6 trillion by 2033, yet the path to building a competitive AI model has become so expensive that even well-funded startups struggle to compete. One researcher recently posed a provocative question: give him $120 million and he could change AI. But behind that bold claim lies a deeper crisis in the industry,one rooted not in training costs, but in the unexpected explosion of inference demands that are fundamentally reshaping how AI systems work.

The AI landscape has shifted dramatically in the past 18 months. OpenAI is valued at $852 billion, Anthropic at $965 billion, and NVIDIA, the primary supplier of AI chips, has reached a $4.72 trillion valuation. For context, Ireland's entire GDP is $779 billion. Yet despite these astronomical figures, the industry faces a critical infrastructure challenge that no amount of capital alone can solve.

What Is Test-Time Compute and Why Does It Matter?

Test-time compute, also called inference scaling or "thinking" at test time, refers to the computational work an AI model performs when answering a question, rather than during the training phase. Until recently, this was considered a minor cost factor. Today, it's become the dominant driver of hardware demand and infrastructure complexity.

According to data presented at NVIDIA's GTC Taipei conference in June 2026, the average number of output tokens per question has surged at a rate exceeding 5 times per year since the second half of 2024, reaching approximately 30,000 to 40,000 tokens per response. To put this in perspective, a typical paragraph of text contains roughly 100 to 150 tokens, meaning modern AI systems are now generating responses equivalent to 200 to 400 pages of text per query.

"The memory system of AIs is going to cause the storage system to be completely revolutionized," stated Jensen Huang, NVIDIA founder and CEO.

Jensen Huang, Founder and CEO, NVIDIA

This explosion in output tokens directly translates to massive new demands on memory and storage infrastructure. The challenge isn't just processing power; it's managing the intermediate data that accumulates during inference, a problem that traditional AI infrastructure was never designed to handle.

How Are Hardware Makers Adapting to Inference Demands?

The industry is responding with a multi-layered approach to manage the surge in inference workloads. NVIDIA has introduced several new technologies and architectural changes to address the bottlenecks:

• KV Cache Management: During inference, AI models generate key-value vectors that must be stored to avoid redundant computation. As conversation length and batch size grow, this cache memory consumption expands dramatically. NVIDIA released Dynamo in March 2025, a software tool that offloads infrequently accessed cache data to lower-bandwidth but higher-capacity storage tiers like CPU RAM and solid-state drives, reducing the burden on expensive GPU memory.
• Context Memory Storage Platform: In January 2026, NVIDIA introduced CMX, a specialized platform designed to store and manage massive amounts of cache generated by long-context workloads. Built around the BlueField-4 data processing unit, CMX manages approximately 9,600 terabytes of capacity per rack, adding a dedicated tier between local storage and shared storage systems.
• CPU Architecture Redesign: Traditional AI deployments used a CPU-to-GPU workload ratio of roughly 1:4 or 1:8. Agentic AI applications, which require models to plan, call tools, and make decisions autonomously, are shifting this ratio to approximately 1:1, creating significant new demand for CPU processing power and memory.

NVIDIA debuted its Vera CPU at GTC, purpose-built for agentic AI workloads and supporting up to 1.5 terabytes of memory capacity, three times the capacity of the previous-generation Grace CPU. However, NVIDIA has halved the memory capacity of its next-generation Vera Rubin Superchip modules, not due to reduced demand, but because insufficient LPDRAM capacity has been allocated to NVIDIA under suppliers' preliminary 2027 production plans.

Why Is Memory Supply Becoming the New Bottleneck?

The surge in inference workloads has created an unexpected crisis: memory shortage. AI servers are now competing with smartphones as the top consumer of LPDRAM (low-power dynamic random-access memory), and supply is tightening rapidly. Contract prices for memory components are surging hard as inventories hit rock bottom.

This memory crunch is forcing difficult trade-offs. NVIDIA is trimming memory capacity on next-generation platforms to keep shipments on track, a decision that reflects the severity of the supply constraint. Meanwhile, traditional x86 vendors Intel and AMD, along with Arm-based chip makers, are all launching new CPU products in 2026 specifically designed for agentic AI workloads, intensifying competition for limited memory supplies.

The broader CPU landscape is experiencing a full-spectrum refresh. Intel launched its Xeon 6+ (Clearwater Forest), AMD released its EPYC Venice, Arm introduced the Arm AGI CPU, and Ampere's AmpereOne MX is expected to enter mass production later in 2026. This convergence of new products all competing for the same constrained memory supply suggests that 2026 will be a pivotal year for infrastructure costs and availability.

What Does This Mean for the $120 Million Question?

The researcher's proposal to transform AI with $120 million reflects a fundamental misunderstanding of where the real constraints lie. Training a competitive large language model like the Spanish Falcon LLM required roughly $300 million in compute costs, yet the model sees minimal adoption. The problem isn't just the upfront training investment; it's the ongoing operational costs of inference at scale.

As test-time compute demands continue to grow at 5 times per year, the infrastructure required to serve AI models at production scale becomes exponentially more expensive. A startup with $120 million could train a model, but it would struggle to deploy it profitably once inference costs begin to dominate. The real bottleneck is no longer innovation or training capital; it's the physical infrastructure and memory supply chains that support inference workloads.

The AI industry is entering a new phase where test-time compute, not model training, defines competitive advantage. Companies that can efficiently manage KV cache, optimize memory utilization, and navigate the tightening supply of LPDRAM will thrive. Those that cannot will find their models stranded, unable to scale despite having been trained at enormous cost. The $120 million question, then, is not whether you can build an AI model, but whether you can afford to run it.