The biggest misconception in AI infrastructure is that hardware improvements drive most of the efficiency gains. According to Dylan Patel, founder of SemiAnalysis, a 90-person semiconductor research firm, the real 100x leaps in AI performance come from simultaneously optimizing three layers: the chip itself, the software running on it, and the model architecture designed for that hardware.

What's Driving the Real Efficiency Gains in AI?

Over the past three years, NVIDIA's transition from its Hopper chip to the newer Blackwell architecture delivered roughly a 30x improvement in inference performance for the same model. That's significant, but it tells only part of the story. When you add system software improvements (like better kernel libraries and attention mechanisms), you get another 5x to 10x boost. And when you factor in model architecture changes (like switching to sparse, mixture-of-experts designs), the gains multiply dramatically.

The breakthrough happens when all three layers are designed together. Patel explained this multiplicative effect: "The real breakthrough innovation is when you leapfrog a few layers, you co-optimize and co-design them, and now all of a sudden you've taken what could have been a 2x here, 2x here, 2x here and instead of being multiplicative to 8x, it's actually 100x".

Patel

Consider DeepSeek's V3 and V4 models as a concrete example. The company explicitly shaped the size of its expert layers to match the dimensions of NVIDIA's tensor-core units on Hopper, and later optimized them further for Blackwell. This co-design locked the model into NVIDIA's ecosystem. When researchers tried running the same DeepSeek model on Google's TPU chips, performance suffered significantly, not because TPUs are inferior, but because the model's architecture was never optimized for TPU's different hardware topology.

How Does This Change the Competitive Landscape Between Chip Makers?

The traditional framing of "NVIDIA versus TPU" misses the real story. Patel argued that both NVIDIA and Google will ship tens of millions of chips annually, each generating over $100 billion in annual revenue from their respective owners. The winner is not determined by raw chip performance, but by which company can co-design the best models and software stack for its hardware.

Google's TPU v6e and v7 chips run Gemini models exceptionally well because Google controls both the hardware and the model architecture. Similarly, OpenAI's models are heavily optimized for NVIDIA's Blackwell chips. But this creates a coordination problem: labs that do not control both hardware and model design face a penalty when trying to run models optimized for a different chip family.

Patel revealed that Google, despite owning its own TPU fleet, still pays xAI (Elon Musk's AI company) approximately $11 per hour per GPU to access NVIDIA hardware for non-Gemini projects like drug discovery and Waymo autonomous driving. This signals that even the largest chip makers recognize the value of hardware diversity and the cost of being locked into a single architecture.

Ways to Understand the Three-Layer Co-Design Framework

• Hardware Layer: NVIDIA's Hopper-to-Blackwell transition improved inference performance by roughly 30x through faster memory (HBM), higher power density, and optimized matrix-multiply units. The ceiling for further gains is hitting physical limits around power density (approximately 1 watt per square millimeter).
• System Software Layer: Libraries like PyTorch, custom kernels such as FlashAttention, and better collective communication algorithms add another 5x to 10x improvement per year. These gains come from better utilization of existing hardware rather than new silicon.
• Model Architecture Layer: The shift from dense models like GPT-3 to sparse mixture-of-experts designs with only 2 billion active parameters (like DeepSeek V4) delivers 40x to 60x cost reductions for equivalent quality over three years. This layer has produced the largest single contribution to efficiency gains.

The critical insight is that evaluating a chip in isolation ignores the downstream effects of model design. A chip that looks optimal on paper may perform poorly if the model architecture was designed for a different hardware family. Conversely, a model that seems inefficient might be perfectly tuned for its target chip.

Why Is Continuous Benchmarking Becoming Essential?

SemiAnalysis launched a new platform called InferenceX to address a structural problem in the industry: traditional benchmarks become obsolete almost immediately. Inference software libraries update twice per week, new models appear every few days, and the optimal configuration for balancing speed and throughput shifts constantly.

InferenceX automates daily measurements across 15 or more chip types, including NVIDIA's H100 and B200, AMD processors, Google TPU, Amazon Trainium, and specialized chips from Groq and Cerebras. The platform has received over $50 million in donated hardware from companies including CoreWeave, Crusoe, Microsoft, Amazon, Google, and OpenAI, with plans to exceed $100 million as more chips are added.

The platform's core output is the throughput-interactivity Pareto frontier. Rather than reporting a single benchmark number, InferenceX reveals the full tradeoff curve: you can optimize for fast responses to individual users (paying a 4x cost premium) or batch process documents at lower cost. Every downstream decision about API design, priority queues, and batch processing is a point on that curve.

"Most things in hardware infrastructure, model application layer, everything is downstream of that curve," Patel stated.

Dylan Patel, Founder of SemiAnalysis

What Does the Compute Crunch Mean for AI Companies?

Patel described the current environment as a structural compute shortage that will persist for years, not a temporary supply squeeze. In 2026, approximately 20 gigawatts of new data center capacity came online. In 2027, over 30 gigawatts are expected, even accounting for typical construction delays. Yet demand is growing faster because model capability is expanding the total addressable market of tasks for which AI is economically useful.

This creates a paradoxical situation for the best AI labs. Anthropic, for example, achieved net-income profitability in the second quarter of 2026 (likely April through June) while maintaining gross margins exceeding 80 percent on its Opus 4.8 API tokens. The company is renting GPUs from xAI at above-market rates, yet still earns positive margins because the demand for its models is so strong.

Even if Anthropic had to pay double the current GPU rental rate, the company would still maintain 50 percent gross margins, meaning it will continue to expand compute spending regardless of price. This dynamic applies across the industry: every GPU that Amazon, Google, or Microsoft adds generates higher revenue, so the compute crunch will persist as long as model capabilities continue expanding faster than infrastructure can scale.

The implication is clear: the next phase of AI competition will not be won by the company with the most chips or the fastest hardware. Instead, it will be won by the teams that can co-design across all three layers simultaneously, creating models and software stacks that extract maximum efficiency from their chosen hardware platform. For NVIDIA, this means the CUDA moat is "at least partially disentangled" not because AI can write kernels, but because downstream model shapes themselves lock customers into one hardware family.