AI inference, the process of running trained models to generate responses, is becoming so massive and specialized that it's driving a fundamental shift in how computing hardware gets built. Andreessen Horowitz (a16z) partners Raghu Raghuram and Sarah Wang argue that the economics of AI inference have reached a tipping point: the workload is now so large and stable that it demands custom-built systems rather than flexible, general-purpose processors.

What's Driving the Shift Away From General-Purpose GPUs?

For the past few years, graphics processing units, or GPUs, have been the workhorse of AI. They were the right tool when nobody knew which AI architectures would win because they could run almost anything. But inference is fundamentally different from training. Training is a one-time cost, like a film studio paying to produce a movie. Inference, by contrast, is an ongoing operational expense that scales with every user interaction.

Google announced in May 2026 that it was processing 3.2 quadrillion tokens per month across its products, roughly 300 times what it had been processing two years earlier. OpenAI now has roughly a billion monthly active users, and companies like Anthropic are seeing enormous success with Claude Code. This explosive growth means inference is quickly becoming the largest workload computers have ever run.

The mismatch between what GPUs do well and what inference actually requires is becoming impossible to ignore. Generating a single token requires a chip to read hundreds of gigabytes of model parameters from memory and perform only a couple of simple operations on each byte. GPUs solve this through "batching," serving many users' requests simultaneously so the expensive memory read is spread across dozens or hundreds of tokens. But batching trades speed for throughput, and the fastest-growing inference workloads, like coding agents and long-context reasoning, can't afford that slowdown.

How Is the Computing Industry Responding to This Challenge?

The pattern playing out now mirrors a recurring cycle in computing history. When a workload becomes large enough and stable enough, the industry builds specialized hardware designed specifically for that job. Graphics processing migrated from the CPU to dedicated GPUs. Networking packets stopped flowing through CPUs and moved to custom switch chips from companies like Broadcom. Now the same transformation is happening with AI inference.

Hyperscalers have already figured this out. Google developed the TPU, or tensor processing unit, along with custom interconnects, pods, and cooling systems. Amazon, Meta, and Microsoft have followed with their own custom silicon programs. Even frontier AI labs like OpenAI are now building their own inference hardware.

The key insight driving this shift is what a16z calls "tokens per watt." Data centers are now constrained by power consumption rather than dollars. Every watt that a general-purpose chip spends on flexibility that inference never uses is a watt not producing tokens. In a world where energy is the limiting factor, specialized hardware that maximizes output per unit of power becomes the winning strategy.

Steps to Understanding the Inference Hardware Revolution

• Recognize the Scale: AI inference has grown 300 times in just two years at Google, making it the largest computing workload in history and still accelerating.
• Understand the Economics: Unlike training, which is a fixed cost, inference is an operational expense that scales with every user query, making efficiency critical.
• See the Hardware Mismatch: GPUs were designed for flexibility across many tasks, but inference is now specialized enough that custom chips can do the job faster and cheaper.
• Grasp the Power Constraint: Modern data centers are limited by electricity, not money, so maximizing tokens per watt has become the real currency of AI inference.

Who Is Building the Next Generation of Inference Hardware?

One company betting heavily on this shift is Etched, founded in 2022 by Harvard dropouts Gavin Uberti, Robert Wachen, and Chris Zhu. Their thesis was straightforward: inference would become the largest market in AI, and general-purpose GPUs wouldn't be the endgame. While hyperscalers like Google and Meta have built custom silicon for their own use, they can't sell it to others. You can rent a TPU from Google, but you can't buy one or rack it in your own data center. For AI labs without silicon teams, inference cloud providers, sovereign AI programs, and enterprises building their own fleets, there was a massive opportunity for chips designed specifically for inference.

Etched has grown to more than 400 engineers, many hired from Nvidia, Google's TPU group, Broadcom, Apple, SK Hynix, and TSMC. Notably, a striking share of these engineers work not on chip design but on production: supply chains, rack assembly, burn-in, and logistics. In a supply-constrained market, the scarce skill isn't designing a fast chip; it's shipping complete, working systems by the thousands.

"Production is the product," noted the Etched founders, emphasizing that in a constrained market, the ability to manufacture and deliver systems at scale matters more than raw chip performance.

Gavin Uberti, Robert Wachen, and Chris Zhu, Founders of Etched

Etched's approach centers on two core technical ideas. The first is low-voltage inference: running the math blocks at roughly half the voltage of a typical AI chip, which packs several times more usable compute into the same power envelope. The second is cluster-scale memory: an ultra-low-latency interconnect that pools memory across chips so that during the decoding phase, an entire rack behaves like one enormous machine.

This shift from general-purpose to specialized hardware represents a fundamental inflection point in AI infrastructure. As inference demand continues to grow exponentially, the companies that can design and manufacture complete systems optimized for this specific workload will capture the largest market in AI. The age of one-size-fits-all computing for AI is ending, and the age of purpose-built inference systems is beginning.