The artificial intelligence industry is undergoing a fundamental shift in computing priorities, and it's reshaping which chips matter most. Inference, the process of running trained AI models to generate responses, now accounts for two-thirds of total AI compute demand, up from roughly one-third in 2023, and is expected to reach 70 to 85 percent by 2028 to 2030. This structural change is redefining competition in the chip market, moving away from "who has the fastest graphics processing unit (GPU) for training" to "whose chip delivers the lowest total inference cost and highest throughput."

The global AI inference chip market is valued at $85.4 billion in 2024 and is projected to grow from $105.47 billion in 2025 to $570.77 billion by 2033, with a compound annual growth rate of 23.5 percent over the forecast period. This explosive growth is attracting attention from chip makers across the industry, but the winners may not be who everyone expected.

Why Is Inference Demand Suddenly Exploding?

Training and inference are fundamentally different computational tasks. Training involves massive parallel matrix operations, trillions of floating-point calculations executed simultaneously across thousands of GPU cores, the domain where GPUs have historically dominated. Inference, especially for agentic AI systems that make decisions and take actions, involves task orchestration, tool invocation, multi-step logical reasoning, and sequential decision-making. These workloads rely heavily on complex logic control and serial processing, areas where central processing units (CPUs) traditionally excel.

A joint study by Georgia Tech and Intel found that in agentic AI scenarios, 50 to 90 percent of latency comes from the CPU, not the compute accelerator, because large models must call plugins, perform web searches, and handle multi-step logic, all managed by the CPU. This finding has profound implications for how companies design their AI infrastructure.

"The CPU is becoming the bottleneck in AI workflows," stated Dion Harris, an Nvidia executive, in March 2026.

Dion Harris, Executive at Nvidia

That admission from Nvidia, a company built on the belief that GPUs are essential for AI, signals how dramatically the landscape is shifting. The company itself has responded by launching Grace and Vera CPU product lines in 2026, with Vera CPUs specifically designed for inference and agentic AI workloads.

How Is the CPU-to-GPU Balance Changing?

The ratio of CPUs to GPUs in data centers tells the story of this transition. In AI training, CPU-to-GPU ratios typically sit at an extreme 1 to 8, with GPUs bearing most of the computational load. But in the inference era, this ratio is rapidly narrowing to between 1 to 1 and 1 to 2. Intel CEO Pat Gelsinger noted in the Q1 2026 earnings call that training workloads usually require 7 to 8 GPUs per CPU, but inference workloads have tightened to 3 to 4 GPUs per CPU, with the prospect of moving toward a 1 to 1 balance.

Referencing Nvidia CEO Jensen Huang's estimates, each gigawatt-scale data center requires about 300,000 Rubin GPUs and, based on 136 cores per ARM CPU, about 221,000 CPUs per gigawatt. This sets the new CPU-to-GPU ratio at roughly 1 to 1.4, a dramatic shift compared to the GPU-dominated era.

Steps to Understanding the New Chip Hierarchy in AI Infrastructure

• Recognize GPU's Persistent Advantage: GPUs use high-bandwidth memory like GDDR6X or HBM, offering bandwidth exceeding 800 gigabytes per second, compared to CPUs' 50 to 100 gigabytes per second from system DDR memory. For high-throughput, high-concurrency inference scenarios, such as large-scale cloud AI services, GPUs remain optimal.
• Understand CPU's Logical Processing Role: CPUs excel at sequential decision-making, tool invocation, and multi-step reasoning that agentic AI systems require. In Llama 3.1 8B model inference, CPU solutions deliver 819 tokens per second per task, while an 8-GPU cluster achieves 46,841 tokens per second.
• Monitor ASIC Growth as a Third Path: Application-specific integrated circuits (ASICs) are emerging as the fastest-growing segment in the inference market, with ASIC server shipments expected to grow 44.6 percent in 2026, compared to GPU server shipment growth of 16.1 percent.

Which Companies Are Winning the CPU Resurgence?

AMD is a standout beneficiary of the CPU comeback. AI server demand has boosted EPYC CPU shipments, with the fifth-generation Turin capturing a significant share of the server CPU market. AMD's server CPU business is expected to grow at least 50 percent in 2026, with Bernstein analysts forecasting that flagship EPYC processor sales could jump 30 percent in 2026. As of early 2026, Intel holds about 60 percent of the data center CPU market, AMD about 24 percent, and Nvidia about 6 percent.

Intel is also actively adjusting its strategy. At Computex in June 2026, new CEO Pat Gelsinger announced the return of CPUs to prominence in the inference era, leveraging 18A process technology and rack-scale decoupled architectures. AI infrastructure is moving from "one-stop shopping" to "Lego-style assembly," where companies mix and match different chip types based on specific workload requirements. Intel's Xeon processors feature Advanced Matrix Extensions (AMX), which accelerate inference for large language models with small to medium parameter sizes, even without GPUs or other AI accelerators.

The data center processor market is experiencing rapid growth, fueled by surging demand for generative AI workloads. Market size is projected to expand from $215 billion in 2025 to $656 billion by 2031, with hyperscale data centers entering an "upgrade cycle" and server CPU shipments expected to grow 25 percent in 2026.

What About GPUs and ASICs?

Despite CPUs regaining ground, GPUs still hold an irreplaceable position in AI inference, thanks to their advantages in memory bandwidth and parallel throughput. Nvidia's dominance in this field remains unchallenged. According to SemiAnalysis, Nvidia held a 92 percent share of the AI training chip market and 78 percent of the inference chip market in Q1 2026. The AI accelerator market is expected to reach $160 billion in 2025 and over $200 billion in 2026, with inference spending accounting for two-thirds.

However, GPU market share in inference is facing multiple pressures from the CPU's comeback, specialized ASIC competition, and practical cost considerations. Beyond the GPU-CPU binary, ASICs are emerging as the fastest-growing variable in the inference market. TD Cowen forecasts that commercial accelerator market share will drop from about 91 percent in 2025 to 75 percent in 2030, while custom ASICs will rise from 9 percent to 25 percent. This shift reflects hyperscalers' growing preference for chips optimized specifically for their inference workloads.

The semiconductor industry rally on June 22, 2026, reflected this broader shift. The Philadelphia Semiconductor Index jumped 6.42 percent in a single day, with Intel soaring over 10 percent on news of a chip manufacturing partnership with Apple, TSMC's American Depositary Receipt climbing 6.94 percent to close at $462.12, and Nvidia rising nearly 3 percent. This market movement signals investor recognition that the inference era is reshaping the entire chip landscape, creating opportunities across multiple chip types and manufacturers rather than concentrating power in a single dominant player.