The artificial intelligence chip industry is hitting a wall, and it's made of memory. As demand for high-bandwidth memory (HBM) explodes, the three companies that manufacture it are running out of capacity, prices have skyrocketed, and startups are now pursuing radical engineering solutions to break free from the shortage. At Hot Chips 2026 this week, the industry's premier semiconductor conference, two competing visions of the future emerged: incremental improvements to existing memory stacks, and a complete architectural reimagining that puts compute and memory in direct contact.

Why Is Memory Becoming the Bottleneck in AI Hardware?

The problem sounds simple but has profound implications. Modern AI accelerators like Nvidia's GPUs need to move enormous amounts of data between the processor and memory at extreme speeds. Traditional approaches stack memory chips on top of each other using HBM technology, which offers impressive bandwidth but comes with a hidden cost: each generation of HBM requires roughly three times more silicon wafer area than conventional DDR5 memory to deliver the same capacity. As AI workloads demand faster and faster memory, manufacturers are forced to allocate more and more fab capacity to HBM production, starving the consumer and enterprise markets for standard memory.

The numbers tell a stark story. Conventional DRAM contract prices rose between 90% and 95% in the first quarter of 2026 alone, then jumped another 58% to 63% in the second quarter. A mainstream 32-gigabyte DDR5 memory kit that cost $110 to $140 a year earlier now sells for around $392. Over 12 months, 128 gigabytes of DDR5 memory has climbed 500% in price. These aren't abstract statistics; they're translating directly into higher PC costs, with HP reporting that DRAM now accounts for 35% of its PC build cost, up from 15% to 18% just a quarter earlier.

"The memory wall is still present, and, in fact, maybe getting worse," said Raghu Sreeramaneni, HBM Design Architecture Fellow at Micron.

Raghu Sreeramaneni, HBM Design Architecture Fellow at Micron

The root cause is a mismatch between compute and memory performance. Compute power roughly triples every two years, but HBM bandwidth only doubles in that same period, creating an ever-widening gap. To close that gap, memory makers add more banks, wider data paths, and more through-silicon vias (TSVs), which are tiny connections that link memory layers together. Each improvement requires more die area, pushing the silicon penalty higher with every generation.

How Are Startups Trying to Solve the Memory Crisis?

One startup, d-Matrix, is taking a fundamentally different approach. Instead of stacking memory on top of compute, the company is doing the opposite: placing a compute chip directly on top of a custom-designed DRAM die, creating what it calls the first 3D DRAM accelerator for generative inference. The product, called Raptor, achieves 100 terabytes per second of memory bandwidth from just 32 gigabytes of capacity per card, a density that would be impossible with conventional HBM stacking.

The engineering is intricate. D-Matrix worked with TSMC to fabricate a 4-nanometer compute die and bonded it face-to-face to a custom DRAM die at a 36-micron pitch, an extraordinarily tight spacing. The vertical interface between compute and memory consumes only 0.37 picojoules per bit of energy, compared to roughly 2.4 picojoules per bit for moving data into an HBM4 base die. That 6.5-fold efficiency gain exists only because the DRAM die was designed specifically for this pairing, with non-standard bank geometry and refresh behavior that would never appear in a conventional memory product.

The tradeoff is significant. Raptor's 32 gigabytes per card stands against 192 to 288 gigabytes for HBM4-equipped accelerators, forcing deployments to scale horizontally across many cards rather than vertically within a single package. A rack of 72 Raptor cards would hold 2.3 terabytes of memory, enough to run Kimi K3, a 2.8-trillion-parameter model, at 4-bit precision with room for roughly 54 concurrent users at 1-million-token context. That's a fundamentally different deployment model than what the industry has grown accustomed to.

What Are the Key Technical Challenges d-Matrix Must Overcome?

• DRAM Manufacturing Partnership: D-Matrix has never publicly identified which company manufactures its custom DRAM die. Only three companies make leading-edge DRAM at volume, and all three are allocating capacity to HBM4 production that is effectively sold out through 2026. A startup with roughly $450 million raised is negotiating from a far weaker position than Nvidia, the memory makers' largest customer.
• Thermal Management: The DRAM die is designed to operate at a junction temperature of 105 degrees Celsius, where memory retention collapses from a standard 32 milliseconds to just 4 milliseconds. The memory must refresh eight times more often, consuming precious bandwidth. D-Matrix addressed this by shrinking each microbank to 1,366 rows and about 5.33 megabytes, so a full refresh sweep costs only 1.37% of overall bandwidth.
• Density Penalty: Raptor's 11.4 megabytes per square millimeter is roughly half of HBM4's 21.9 to 26.3 megabytes per square millimeter. D-Matrix acknowledged using a less advanced DRAM process to achieve the custom geometry, a tradeoff that limits capacity per card.
• Scaling Limitations: When models and context windows grow beyond what a single rack can hold, inference spills into inter-card synchronization overhead, negating the vertical bandwidth advantage that Raptor was designed to eliminate. The company's own research paper concedes this failure mode.

D-Matrix CEO Sid Sheth told CNBC in June that Raptor is slated to launch in 2027, but at Hot Chips the company gave no firm date, volume commitments, or pricing information. Every performance figure shown, including 988 tokens per second per user on Kimi K3 at 1-million-token context, is a projection built on early silicon. No third party has yet measured Raptor on live hardware, making direct comparison to competitors difficult.

How Does This Compare to Existing Memory Solutions?

Cerebras, another AI accelerator maker, claimed 969 tokens per second on Llama 3.1 405B in November 2024, with third-party verification from Artificial Analysis on live hardware. D-Matrix's 988 tokens per second on a model seven times larger would represent a significant step forward if the numbers hold up in production. However, the comparison is complicated by the fact that Cerebras' benchmark was verified on real silicon, while d-Matrix's figures are simulations anchored to early characterization.

Samsung is pursuing a similar concept called zHBM, which stacks HBM directly on the processor and claims a 70% power-efficiency gain over HBM4E setups. The critical difference is that Samsung operates its own DRAM fabs, giving it the manufacturing flexibility that d-Matrix lacks. Samsung has not announced a production timeline before HBM5, the next generation of standard memory technology.

What Does This Mean for the Broader AI Hardware Market?

The memory crisis is already reshaping the economics of AI infrastructure. Nvidia has raised the price of its DGX Spark desktop from $3,999 to $4,699 and warned large customers of AI server price increases above 15%, both tied to memory costs. SK hynix CEO Kwak Noh-jung told Reuters in July that customer demand will remain higher than supply capacity even beyond 2030, and he has called 2027 the worst year for memory supply in the industry's history.

Nvidia is reportedly testing Rubin Ultra configurations with as little as 192 gigabytes of HBM4E, down from standard configurations, because the company may not be able to source enough supply. This suggests that even the world's most powerful chip maker is feeling the squeeze. For startups like d-Matrix, the window to solve the memory problem is narrow; if they cannot secure manufacturing capacity for custom DRAM dies, their entire roadmap collapses.

The industry faces three potential paths forward: China's CXMT ramping competitive DDR5 in volume, hybrid bonding arriving before HBM5 (which SK hynix has already ruled out), or DDR5 profitability slipping back below HBM. None of these scenarios is likely before new fab capacity comes online and scales up, meaning the memory crisis will persist for at least another two to three years.

How to Evaluate AI Chip Announcements in a Memory-Constrained Market

• Manufacturing Transparency: When a startup announces a new AI accelerator, ask whether the company has disclosed its memory supplier and secured volume commitments. If the memory partner is unnamed or unconfirmed, the product's timeline is speculative.
• Third-Party Verification: Performance claims from early silicon are projections, not guarantees. Look for independent benchmarks on production hardware from firms like Artificial Analysis or academic institutions before comparing competing products.
• Deployment Model Implications: Understand whether the chip requires horizontal scaling across many cards or vertical stacking within a single package. This affects total cost of ownership, power consumption, and data center architecture in ways that raw bandwidth numbers don't capture.
• Thermal and Reliability Trade-offs: High-density memory stacks generate heat and face reliability challenges. Ask whether the company has disclosed junction temperatures, refresh penalties, and error correction overhead, which directly impact real-world performance and operating costs.

The memory bottleneck is reshaping AI hardware faster than most observers realize. Startups are betting billions on radical new architectures, but success depends on solving manufacturing problems that are largely outside their control. For enterprises and data center operators, the lesson is clear: memory availability and cost will constrain AI deployment far more than compute performance over the next two years.

" }