We have trained Brumby-14B, the best RNN in history based on power retention. It's 100% attention free (because you don't need a hybrid if your RNN layer actually works). A lot of new LLM applications are about to be unlocked with cheap long context finetunes of Brumby!

We built the strongest attention-free base LLM, Brumby-14B-Base, for only $4,000. This is just a base model, but for the first time in 2 years, I'm seeing light at the end of the tunnel for the field to switch to an alternative architecture to transformers.

https://t.co/BOMBk4AEYv If I were @jefrankle, I'd be getting a bit nervous right about now :-)

The key innovation enabling this result was our power attention layer, which we open-sourced a few weeks ago. Its similarity to attention enables cost-efficient reuse of weights, but the resulting arch has no quadratic-cost bottleneck. More info here:

By retraining our architecture around the weights of existing (attention-based) models, we get an enormous boost to learning speed. We can catch up to the performance of frontier models at this scale in just a few thousand steps of training.

The end of the transformer era marches slowly closer: we trained a completely attention-free foundation model at the 14B scale for only $4,000. The performance matches other models of similar scale, including transformers and hybrid models.

Excited to share my recent interview on Recurrence and Attention for Long-Context Transformers with @samcharrington for the @twimlai podcast. Check it out! https://t.co/ezdkB1N9RM via @twimlai

Really fun conversation on @latentspacepod!

I knew what Carles and Jacob were working on for about ~1 year now. It is still extremely exciting to see these two graphs side by side📉📈

Thank you @RashiShrivast18 for highlighting our work @manifest__ai!

This is probably the most important architectural change to neural nets since the Transformer. It's time to train long-context models for everything :) Congrats @jacobmbuckman and everyone @manifest__ai!

@jacobmbuckman MORE POWER. MORE RETENTION.

Discover more details, access the code, benchmarks, and read the full paper here:

Power Retention represents a step change for the field. We've done nothing but exploit scale for the past five years; it's time to rethink the fundamentals. No more hacks, patches, or RAG. The long context era is about to begin. Are you ready?

For a deeper dive, our paper "Scaling Context Requires Rethinking Attention" is available on ArXiv:

Trying out Power Retention is as easy as `pip install retention`. We’re also releasing: - PowerCoder 3B: a super-fast, long-context code autocompletion model showcasing our tech - Vidrial: our framework for clean, high-performance CUDA kernels

Manifest's Power Retention enables LLMs to reason efficiently across millions of tokens. Applications that were previously out of reach due to cost of context become feasible. Assistants without session resets, days-long reasoning traces, full software engineering workflows...

Transformers are incredibly powerful, but this power comes at a cost. Transformers memorize everything that they encounter, and at long context, this becomes a bottleneck. Rich understanding *or* long context: transformers can only have one.

Transformers are broken. Today, Manifest AI is releasing Power Retention, an open-source architecture to replace them. More below 🧵:

I asked @sama, @tylercowen, @finbarrtimbers, @jacobmbuckman and anonymous: "How might AGI impact entrepreneurs?" @sama At some point there'll be a one-person billion-dollar company. I could see that get started this year and reach that valuation a few years after that.