The race to build faster AI chips may have been solving the wrong problem. While most accelerator makers compete on raw computing throughput, SambaNova argues that memory bandwidth and capacity, not arithmetic power, now determine whether AI inference systems actually perform well in the real world. The company's new SN40L Reconfigurable Dataflow Unit (RDU) tackles this constraint head-on, combining streaming dataflow architecture with three tiers of memory to reduce data movement and infrastructure requirements.

What's the Real Bottleneck in AI Inference?

For years, the AI hardware industry focused on one metric: how many calculations a chip could perform per second. But as generative AI models have grown larger and more complex, a different problem has emerged. Modern accelerators often sit idle, waiting for data to arrive from memory fast enough to feed their computational units. This mismatch between compute capability and memory delivery is what researchers call the "memory wall".

SambaNova's analysis reveals how dramatically this shift changes the competitive landscape. When comparing performance on Llama 3.1 70B, a popular open-source AI model, SambaNova reports using 16 SN40L chips to achieve competitive throughput, while Cerebras requires 336 chips and Groq needs 576 processors for similar results. These numbers illustrate a fundamental architectural difference: SambaNova's design prioritizes efficient data flow, while competitors rely on sheer chip count to compensate for memory constraints.

How Does SambaNova's Three-Tier Memory Strategy Work?

The SN40L's innovation lies in treating memory as a core architectural problem rather than a supporting subsystem. Each processor socket combines three distinct memory layers, each optimized for different tasks:

• On-Chip SRAM: 520 megabytes of ultra-fast memory distributed across computational units, providing the highest bandwidth for immediate calculations
• Co-Packaged HBM: 64 gigabytes of high-bandwidth memory holding actively running models and frequently accessed data, balancing speed and capacity
• Directly-Attached DDR: Up to 1.5 terabytes of standard memory serving as a local repository for large model collections and checkpoints

This arrangement solves a practical problem that enterprise AI teams face daily. High-bandwidth memory is expensive and limited in capacity, forcing companies to choose between keeping multiple models resident or constantly swapping them in and out. By adding directly-attached DDR, SambaNova lets accelerators move models from storage into high-speed memory at speeds exceeding 1 terabyte per second, eliminating the latency penalty of accessing models through slower PCIe connections.

Why Does Compiler-Controlled Fusion Matter for Real-World Performance?

SambaNova shifts significant optimization responsibility into software. The company's compiler maps AI models into dataflow graphs and combines operations into large fused kernels, keeping intermediate data on-chip rather than shuttling it back to external memory. This approach directly improves "operational intensity," the amount of computation performed for each byte of data moved through the system.

A concrete example illustrates the impact. For a Monarch FFT workload, without compiler fusion, the system achieves 39.5 operations per byte. Combining multiple operations raises this to 102.6 operations per byte, and full spatial fusion reaches 410.4 operations per byte. Higher operational intensity means more useful work happens on data already inside the processor, reducing traffic to external memory and improving overall throughput.

SambaNova claims its compiler can fuse an entire Llama 3.1 8B decoder, a common AI model, into one continuous dataflow kernel that executes repeatedly without kernel-launch overhead. For independent software vendors building AI applications, this shifts the optimization burden from hand-written, hardware-specific code toward compiler-directed mapping, potentially reducing development complexity.

How Does the Composition-of-Experts Approach Change AI Deployment?

Beyond single large models, SambaNova's architecture enables a different deployment pattern: combining multiple specialized models rather than routing every task through one monolithic system. The company describes Samba-CoE, a system containing 150 Llama-7B experts totaling more than 1 trillion parameters. A router selects the appropriate expert based on the incoming request, and the system stores inactive experts in DDR, keeps the router in HBM, and moves the selected expert into high-speed memory for execution.

This capability addresses a growing enterprise need. Companies increasingly operate dozens or hundreds of specialized models for coding, engineering, finance, customer service, translation, and domain-specific applications, rather than forcing every task through a single large language model. The three-tier memory architecture provides the infrastructure to manage this complexity efficiently, with DDR serving as a large local repository and HBM operating as a high-speed working set.

What Metrics Matter Beyond Raw Speed?

SambaNova's competitive argument extends beyond tokens per second, the standard benchmark for AI inference speed. The company wants customers to measure useful throughput against silicon area, rack count, memory capacity, power consumption, and deployment complexity. This shift reflects a maturing market where infrastructure costs and operational complexity matter as much as peak performance.

For chief information officers deploying AI inference as persistent infrastructure rather than experimental workloads, this distinction becomes critical. A processor that wins a benchmark while requiring substantially more racks, networking equipment, memory, or power may produce a less attractive total cost of ownership. Silicon architects face the same calculation: additional compute delivers diminishing value when memory and communication cannot keep it occupied.

The SN40L uses TSMC's 5-nanometer process and a 2.5D chiplet package containing two Reconfigurable Dataflow Dies and HBM. Each socket supplies 638 billion floating-point operations per second in brain float 16-bit precision through 1,040 Pattern Compute Units, with another 1,040 Pattern Memory Units providing distributed memory and address-generation resources.

What Do These Changes Mean for the AI Hardware Industry?

SambaNova's approach signals an important shift in how the AI hardware industry measures success. For years, competition centered on individual chip specifications and peak throughput numbers. The SN40L demonstrates that arithmetic throughput remains important, but memory placement, compiler behavior, model residency, interconnect efficiency, and rack-level utilization increasingly determine whether systems deliver real-world performance.

The company's competitive claims still require independent evaluation across broader workloads, different concurrency levels, various latency targets, diverse software environments, and total system power consumption. Groq, Cerebras, Nvidia, and other accelerator vendors continue to evolve their architectures and software rapidly. However, SambaNova's design demonstrates that treating the memory wall as an architectural problem spanning both silicon and software represents a viable alternative to the traditional approach of maximizing compute density.

As AI inference moves from experimental deployments into persistent infrastructure, the metrics that matter to enterprise customers are shifting. The winner in this emerging competition may not be the chip with the highest peak performance, but rather the system that delivers the most useful throughput per rack, per watt, and per dollar invested in infrastructure.