Nvidia's $20 billion acquisition of Groq in late 2025 positioned the language processing unit (LPU) maker as a key player in AI inference, but the move has exposed a fundamental tension in how the industry approaches serving AI models at scale. The Groq 3 LPU, unveiled at Nvidia's GTC 2026 conference, delivers blazing-fast token generation speeds by relying entirely on on-chip SRAM (static random-access memory) rather than traditional high-bandwidth memory (HBM). Yet this design choice reveals a critical trade-off that competitors are now exploiting: raw speed versus practical capacity and flexibility.

What Makes Groq's Approach Different from Traditional AI Chips?

The Groq 3 LPU is purpose-built for ultra-low latency inference, meaning it prioritizes responding to requests as quickly as possible. It achieves this by packing 512 megabytes of SRAM directly onto the chip, delivering 150 terabytes per second of internal bandwidth. For comparison, Nvidia's Rubin GPU paired with HBM4 memory delivers only 22 terabytes per second. This speed advantage is real and measurable. In third-party benchmarks, the Groq 3 LPU generated 3,431 tokens per second on a 100,000-token context model, roughly four times faster than competing inference endpoints.

However, this speed comes with a severe capacity constraint. At 512 megabytes per chip, a 256-LPU rack holds only 128 gigabytes of total memory. Running a 31-billion-parameter model at FP8 precision (a common inference format) requires 62 chips just to store the model weights. This means the Groq 3 LPU excels at one specific task: generating tokens for models that fit in SRAM, with minimal latency. It is not designed for the flexibility that data center operators increasingly demand.

How Are Competitors Positioning Themselves Against Groq?

The competitive response reveals where the real market opportunity lies. Rather than chasing Groq's speed-at-all-costs approach, emerging startups and established memory vendors are betting that inference workloads will diverge from training workloads, and that the winning architecture will balance speed, capacity, and power efficiency.

Positron AI, founded by veterans from Lambda and Groq itself, raised $875 million in Series C funding to build what it calls "memory-first inference systems." The startup's Asimov chip uses low-power DRAM (LPDDR5X), the same memory technology found in smartphones, rather than the expensive HBM used in data centers. Positron claims its chips achieve over 90 percent utilization of available memory bandwidth, compared to just under 30 percent for GPUs running the same models. The Asimov chips can be interconnected into clusters containing up to 16,384 chips, with per-chip memory capacity ranging from 277 gigabytes to 2.3 terabytes. This represents a fundamentally different philosophy: prioritize capacity and efficiency over raw speed.

"Speed matters in this market, both in how quickly we ship new generations of silicon and in how quickly they reach customers," said Mitesh Agrawal, CEO of Positron AI. "Deploying Atlas at scale taught us an enormous amount about what inference customers actually need, and we have carried those lessons directly into Asimov and Titan."

Mitesh Agrawal, CEO, Positron AI

PieceMakers, a Nanya-backed DRAM designer that began trading on Taiwan's Emerging Stock Board on September 16, is pursuing yet another path. Rather than relying on HBM or low-power DRAM alone, PieceMakers is developing custom DRAM stacks bonded directly onto processor chips using hybrid bonding technology. This approach sits between Groq's SRAM-only design and traditional HBM architectures. The company claims its wafer-on-wafer products can deliver over 2 terabytes per second of bandwidth per layer with latency under 20 nanoseconds, while consuming less power than HBM.

Why Is the Inference Market Fragmenting Into Multiple Architectures?

The fragmentation reflects a deeper reality: inference is not a single problem. Different use cases demand different solutions. Groq's SRAM-based approach is ideal for latency-sensitive applications where users expect near-instantaneous responses, such as real-time chatbots or interactive AI assistants. But many inference workloads, particularly in cloud data centers, prioritize throughput and cost efficiency over individual request latency. For these applications, the ability to run larger models with more flexible memory configurations matters more than shaving milliseconds off response time.

At Nvidia's GTC 2026 conference, the company itself signaled this diversity by pairing the Groq 3 LPU with its Vera Rubin GPU architecture. The Groq 3 LPU handles the "serving" side of token generation, processing requests from users, while Rubin handles the heavier computational work of prompt prefill. Together, they form what Nvidia calls an "AI factory," a tightly integrated system designed to maximize tokens per second per megawatt of power consumed.

Steps to Understanding the Inference Chip Landscape

• Speed vs. Capacity Trade-off: Groq's SRAM-only design prioritizes latency but sacrifices memory capacity, requiring multiple chips for large models. Competitors like Positron and PieceMakers are betting that capacity and efficiency matter more for most inference workloads.
• Memory Technology Choices: The industry is exploring three distinct memory approaches: on-chip SRAM (Groq), low-power DRAM (Positron), and hybrid-bonded DRAM stacks (PieceMakers), each optimized for different cost, power, and performance profiles.
• System-Level Integration: Success in inference is no longer about individual chip performance but about how chips, memory, cooling, and power delivery work together as an integrated system, as demonstrated by Nvidia's AI factory concept.
• Revenue Models Shifting: Emerging competitors like PieceMakers are moving away from selling chips directly and toward custom design services and licensing fees, reflecting the reality that inference customers need highly specialized solutions.

The financial backing behind these competing approaches underscores the market's conviction that Groq's speed advantage, while impressive, does not solve the entire inference problem. Positron's $875 million Series C valuation at $5 billion reflects investor confidence that memory-first inference systems will capture significant market share. PieceMakers' trading debut at a valuation around $1.4 billion, with revenue from AI custom design work rising from 4 percent in 2024 to 40 percent in the first half of 2026, demonstrates that design services and custom silicon are becoming the real profit centers in inference.

The Groq 3 LPU will undoubtedly excel in specific use cases where latency is paramount. But the broader inference market is revealing itself to be far more nuanced. As data center operators grapple with the challenge of deploying AI workloads at scale, they are discovering that the fastest chip is not always the most practical one. The next phase of AI infrastructure will likely feature a portfolio of inference solutions, each optimized for different workloads, cost structures, and power budgets. Groq's acquisition by Nvidia may have elevated the LPU to prominence, but it has also accelerated the emergence of alternatives that challenge the assumption that speed alone defines inference success.