The transformer architecture that powers every major large language model today is becoming a bottleneck, and a growing number of startups are racing to build the next generation of AI technology. Nine years after Google researchers introduced transformers in a landmark 2017 paper, the technology has become so fundamental that it forms the foundation of the entire AI industry. But as AI models grow larger and tackle more complex tasks, transformers are revealing critical limitations that workarounds and patches can no longer hide.

Why Are Transformers Struggling Now?

Transformers excel at one core task: understanding text by comparing every word in a passage with every other word through a mathematical process called dense attention. This mechanism captures meaning with remarkable accuracy. However, the computational cost grows dramatically as text gets longer. A document containing 10,000 words might require a transformer to perform 50 million multiplications, which explains why large language models consume enormous amounts of electricity.

The problem extends beyond energy consumption. Transformers process text word by word, which makes them struggle to maintain large context windows, or the amount of information they can hold in memory at once. Yet modern AI applications increasingly demand exactly this capability. Reasoning models need to write and read internal notes to themselves. AI agents need to process entire code bases or libraries of documents. As these demands grow, transformers have become a fundamental constraint on what AI can do.

OpenAI alone is set to spend $50 billion on computing this year, according to the company's president Greg Brockman, and the International Energy Agency predicts that total electricity consumption by data centers will double by 2030. These escalating costs have created an urgent incentive for innovation.

What Solutions Are Startups Proposing?

A wave of startups is now proposing fundamentally different approaches to building large language models. Some are rethinking how attention works, while others are replacing it entirely with new mechanisms. Here are the main strategies emerging:

• Sparse Attention Models: Subquadratic, a Miami-based startup, has developed a sparse attention mechanism that performs calculations on only some word pairings instead of all of them, dramatically reducing computational requirements. The company claims its model, SubQ, rivals top mainstream LLMs on tasks including search and coding, with thousands already signed up for its waitlist.
• Power Retention Systems: Manifest AI, based in San Francisco, is replacing attention entirely with a mechanism called power retention that stores only the most relevant information for a given task. Instead of forcing models to track everything in their context window, power retention provides a rolling summary that drops less relevant information as new data arrives. The company has demonstrated this approach by converting an existing open-source coding model called StarCoder into PowerCoder and releasing a model called Brumby that rivals some versions of Alibaba's Qwen.
• Hybrid Liquid Models: Liquid AI, an MIT spinout, pairs transformers with liquid neural networks to create what it calls liquid foundation models (LFMs). These models are dramatically smaller and more energy-efficient than standard LLMs, running on devices as modest as a $50 Raspberry Pi. The company has achieved nearly 34 million downloads and reports that its latest models match the performance of rival models four times their size.

"The entire AI industry is built on transformers. They are one of the most important innovations in the history of computer science, and they've changed the world," said Justin Dangel, cofounder and CEO of Subquadratic.

Justin Dangel, Cofounder and CEO of Subquadratic

These startups face significant skepticism from the industry. Some researchers question whether sparse attention mechanisms can truly match the performance of dense attention. Others remain unconvinced that alternative architectures can scale to compete with the largest models built by tech giants. Yet the economic pressure is undeniable, and the potential rewards are enormous.

How Are These New Approaches Different from Transformers?

The fundamental difference lies in how these alternatives handle information. Transformers force models to maintain awareness of everything in their context window, even if most of it is irrelevant. Sparse attention throws out individual words but retains a rough picture of everything seen. Power retention takes a more radical approach, maintaining only a summary of the most relevant information and actively discarding less important details as new information arrives.

Liquid neural networks, inspired by the structure of worm brains, introduce a capability that transformers lack: the ability to adapt behavior to new information as the model processes it. Once a transformer is trained, its behavior is fixed. Liquid models can learn and adjust on the fly, opening possibilities for models that improve through interaction rather than requiring complete retraining.

These architectural differences have practical implications. Models built on alternative architectures can run on smaller devices, consume less power, and potentially handle longer sequences of information more efficiently. For applications like autonomous agents that need to operate for extended periods or vehicles that need to run AI models on embedded chips, these advantages could be transformative.

What Does This Mean for the AI Industry?

The race to move beyond transformers represents a fundamental shift in how the AI industry approaches model development. For nearly a decade, the strategy has been straightforward: scale up transformers, add more data, and increase computing power. The emerging alternatives suggest that the next phase of AI progress may come from rethinking the underlying architecture rather than simply making existing approaches bigger.

This shift could democratize AI development. If smaller, more efficient models can match the performance of massive transformer-based systems, the barrier to entry for companies and researchers drops significantly. It could also address the sustainability concerns that have plagued the AI industry as energy consumption has become an increasingly visible problem.

However, transformers are not going anywhere in the near term. The technology is too deeply embedded in the current AI ecosystem, and the advantages of alternatives remain unproven at scale. What appears to be emerging is a period of experimentation where multiple architectural approaches coexist, with different models optimized for different use cases. Some applications may continue to benefit from transformer-based approaches, while others may shift to sparse attention, power retention, or liquid neural networks.