Four startups betting the next wave of AI runs on something better than transformers
The neural network design powering every major chatbot is hitting real limits. A small group of companies thinks it has found a way around them.
Key points
- Google researchers invented the transformer neural network in 2017, and it now underpins every major AI chatbot on the market.
- OpenAI is on track to spend $50 billion on computing in 2025, largely because transformers are expensive to run at scale.
- The International Energy Agency predicts data-centre electricity use will double by 2030.
- At least four startups are building alternative AI architectures they say are faster, cheaper, and better suited to complex tasks.
- One of those companies, Liquid AI, has recorded almost 34 million downloads for models that run on a $50 hobbyist computer.
Back in the summer of 2017, a team at Google published a paper called "Attention Is All You Need." It described a new type of neural network (think of a neural network as software loosely modelled on how brain cells connect) called a transformer. Transformers turned out to be extraordinarily good at reading and generating text, and they became the engine inside every AI chatbot you have heard of, from ChatGPT to Claude to Gemini.
Seven years on, transformers are straining under their own success.
What is the actual problem?
To understand the meaning of a sentence, a transformer compares every word against every other word. That works brilliantly for short text. For long documents it gets expensive fast: a 10,000-word document can require 50 million individual calculations. The more text you feed in, the worse it gets.
That computational hunger has a real cost. OpenAI plans to spend $50 billion on computing this year, according to company president Greg Brockman. The International Energy Agency says total electricity consumed by data centres will double by 2030.
Transformers also struggle with the very tasks AI companies are now selling. Reasoning models, which write notes to themselves while working through a problem, produce large amounts of extra text that the transformer must then process. AI agents, software that can carry out multi-step tasks on its own, need to juggle information from many sources at once. Both push transformers toward their limit.
What are the alternatives?
Four companies are attacking the problem from different angles.
Subquadratic, a Miami startup, says it has built a model called SubQ that uses "sparse attention." Instead of comparing every word with every other word, sparse attention picks only the pairings that matter, cutting the number of calculations dramatically. The company claims SubQ rivals mainstream large language models on search and coding tasks, though some researchers remain sceptical. Thousands of users are reportedly on its waitlist.
Manifest AI, based in San Francisco, goes further and replaces the attention mechanism altogether. Its approach, called "power retention," gives the model a rolling summary of what it has read so far, dropping information that is no longer relevant as new material arrives. Think of it like a reader who keeps good notes rather than memorising every word. The company has used the technique to build PowerCoder, based on an existing open-source coding model, and Brumby, which it says competes with some versions of Alibaba's popular open-source Qwen model.
Liquid AI, a spinout from MIT based in Cambridge, Massachusetts, takes a hybrid approach. Its models, called LFMs (liquid foundation models), mix transformers with a different technology called liquid neural networks, inspired by the simple nervous systems of worms. The result: models small enough to run on a Raspberry Pi, a $50 hobbyist computer, yet the company says they match the performance of rivals four times their size. Liquid AI has logged nearly 34 million downloads. Mercedes is among its customers.
| Company | Approach | Notable claim |
|---|---|---|
| Subquadratic | Sparse attention (SubQ) | Rivals top LLMs on search and coding |
| Manifest AI | Power retention (Brumby, PowerCoder) | Competes with Alibaba Qwen variants |
| Liquid AI | Hybrid LFMs | Runs on a $50 computer; 34 million downloads |
What does this mean for ordinary people?
Nothing changes in your chatbot tomorrow. Transformers are deeply embedded in products, and replacing them takes years. But the direction matters.
If these alternative approaches work at scale, AI services could become cheaper to run, which would likely mean lower costs passed to businesses and, eventually, users. Smaller, more efficient models could also run on devices you already own, keeping your data on your phone rather than a distant server.
MIT Technology Review, which first surfaced several of these companies, calls this next generation of models "LLMs+." Whether any of today's startups are the ones who build it is an open question. But the problem they are trying to solve is very real.
Common questions
Do I need to do anything differently as an AI user right now?
No. These changes are happening at the engineering level, inside companies building AI tools. For now, the products you use look the same from the outside.
Will these alternatives definitely replace transformers?
Not necessarily. Transformers have survived predictions of their replacement before. Each of these startups still has to prove its approach scales to the size of models that power major products, which is a significant technical and financial challenge.
