While policymakers debate whether artificial intelligence should be deployed more cautiously, the infrastructure supporting AI is being built at breakneck speed. At the AI Infra Summit in Santa Clara this week, executives from Amazon Web Services, Oracle, Broadcom, Qualcomm, and other major tech companies outlined aggressive plans to optimize data centers for AI's exploding computational demands, signaling that the infrastructure race is far outpacing any regulatory or ethical concerns about AI's pace.

What's Driving the Infrastructure Buildout?

The shift in AI workloads is creating unprecedented infrastructure challenges. For most of 2025, training large language models (LLMs), which are AI systems trained on vast amounts of text data, required enormous amounts of graphics processing unit (GPU) power. But in 2026, the focus has shifted dramatically to inference, the process of running a trained model to generate outputs. This shift has created a new problem: token usage, which measures how much text an AI system processes, is skyrocketing, and the cost of tokens is spiraling out of control for many enterprises.

The infrastructure demands are staggering. A typical AI server uses roughly eight times more memory than a traditional server, and the memory market for AI is projected to explode from $35 billion in 2025 to between $175 billion and $190 billion by 2027, roughly a fivefold increase. This growth reflects a fundamental reality: inference is now the dominant workload.

"The majority of compute is going to be serving inference. Inference is going to be a massive workload," said Peter DeSantis, a senior vice president at Amazon.

Peter DeSantis, Senior Vice President at Amazon Web Services

How Are Companies Solving the Memory and Power Problem?

The infrastructure challenge boils down to a simple problem: data movement is slower than computation. Companies are pursuing different strategies to overcome what engineers call the "memory wall," where computing power has outpaced both memory capacity and bandwidth.

• Qualcomm's Approach: The chipmaker has shifted from traditional high-bandwidth memory (HBM) to a proprietary architecture called high-bandwidth compute (HBC), which the company estimates provides six times the bandwidth per watt versus HBM for large batch sizes and 200 times the capacity per watt versus static random-access memory (SRAM) solutions.
• d-Matrix's Solution: The computing startup has integrated higher-throughput 3D DRAM into its next-generation chip architecture called Raptor, which stacks multiple layers of memory cells vertically to allow for higher storage density and improved performance.
• AWS's Strategy: Amazon is leveraging its Graviton family of 64-bit Arm-based central processing units (CPUs) to provide energy efficiency, with the company launching the Graviton5 CPU in June to support real-time AI reasoning and multistep task orchestration.

"The bottleneck is data movement, not arithmetic. The way to solve that is to bring the compute even closer to the memory," explained Tony Pialis, executive vice president and general manager of datacenter and AI at Qualcomm.

Tony Pialis, Executive Vice President and General Manager of Datacenter and AI at Qualcomm

d-Matrix recently unveiled a collaboration with Nvidia to incorporate Raptor into Nvidia's rack reference architecture, NVLink Fusion, designed for AI labs and hyperscalers to deploy ultra-low-latency token services.

Why Is Networking Becoming as Critical as Computing Power?

As companies scale AI infrastructure across massive data centers, networking has emerged as a central bottleneck. Optimizing every layer of the infrastructure stack, from compute to memory to data movement, requires rethinking how servers communicate with each other.

Oracle has developed Acceleron, a high-performance network virtualization architecture and converged SmartNIC technology designed for Oracle Cloud. The company's alliances with Nvidia and Advanced Micro Devices resulted in a networking technology called Acceleron RoCE that boosts performance by bypassing routing of data through the central processing units of servers that host GPUs.

"We really have to optimize every part of the stack. Networking is becoming just as important as the compute itself," noted Karan Batta, senior vice president of Oracle Cloud Infrastructure.

Karan Batta, Senior Vice President of Oracle Cloud Infrastructure

Broadcom has taken a different approach, building its network architecture for AI around Ethernet, a technology developed in the 1970s. The company's Tomahawk 6 networking chip, introduced last year, was optimized to power Ethernet switches in data centers and uses a set of AI features known as Cognitive Routing 2.0 to detect network congestion and reroute data to other connections.

What Does This Mean for Enterprise AI Costs?

Despite the massive infrastructure investments, enterprises are grappling with soaring token costs. Some companies have seen token spending increase eightfold in a single year. However, executives at the conference suggested that the focus should shift from cost per token to value per task, treating high token bills as the cost of discovery and optimization.

"We've actually let the meter run. That is the bill for discovery. For us, it's less about the cost per token and more about the value per task," said Arun Nandi, chief data and AI officer for Carrier Global Corp.

Arun Nandi, Chief Data and AI Officer at Carrier Global Corp.

The infrastructure buildout reflects a broader reality: AI is no longer a research project or a future technology. It is being deployed in enterprises right now, and companies need to make it faster, more efficient, and more cost-effective. The race to optimize AI infrastructure is not slowing down; it is accelerating, driven by the relentless demands of inference at scale.