OPRM is accepted to #COLM2025!.See you in Montreal 🇨🇦. Big thanks to our great collaborators from TAU, MIT, and IBM!.#LLM @COLM_conf.

RT @ItamarZimerman: 📄🚨 New!.Tired of waiting minutes for LLMs to "think"?.Test-time scaling (O3, DeepSeek-R1) lets LLMs reason before answe….

RT @HanGuo97: We know Attention and its linear-time variants, such as linear attention and State Space Models. But what lies in between?. I….

RT @YVinker: Thanks @MIT_CSAIL for featuring our work!🖊️🎨.Huge thanks to the CSAIL news team for the fun article + video!!. We'll be presen….

RT @MIT_CSAIL: Sometimes the best way to express an idea is by sketching it out. A system from MIT CSAIL & Stanford captures this iterativ….

RT @gm8xx8: Overflow Prevention Enhances Long-Context Recurrent LLMs. OPRM chunk-based inference:.- Split the context into chunks.- Process….

Very nice deep dive explaining OPRM by @xiaolGo .

This work was a great collaboration with @ItamarZimerman, @jmie_mirza, James Glass, @leokarlin, and @RGiryes. Check out the paper and our github repo for more experiments, details and code!. Arxiv: Github:

Lastly, our findings raise questions about whether existing recurrent models genuinely exploit long-range dependencies across multiple chunks, since our single-chunk strategy delivers stronger performance - even in tasks that presumably require cross-segment relations.

In addition, OPRM naturally acts as a context extension algorithm, allowing the model to handle sequences far longer than those it was originally trained on. OPRM even outperforms existing dedicated context extension methods - sometimes by a wide margin.

On average, OPRM delivers a 35% improvement on LongBench across various SoTA recurrent architectures, and even sets new SoTA results on the challenging LongBench v2 - all while being faster than vanilla inference and requiring a surprisingly small memory footprint.

To mitigate this, we complement these architectures with a training-free inference algorithm. Our method, OPRM (Overflow Prevention for Recurrent Models), processes the context in chunks, selects the most relevant one, and decodes it - effectively solving the overflow issue:

Recent works address this problem by increasing the state size (e.g. Mamba2) or managing the stored information more effectively (e.g. DeltaNet). While these approaches are valuable, they still fall short, as they attempt to compress the entire context into a fixed-size state.

As shown below by the Associative Recall curve, when the recurrent memory capacity is exceeded, the model exhibits an overflow-like behavior that worsens as more information is added. Therefore, a key principle is to *avoid* overloading the state with information beyond its

We study long-context performance from a less-explored perspective: recurrent memory capacity, and find that even very large hidden states, such as Falcon-Mamba-7B’s, struggle to retain all contextual information.

New work! 🚨. Recurrent LLMs like Mamba and RWKV can efficiently process millions of tokens, yet still underperform on real-world long-context tasks. What's holding them back? 🤔.And how can a lightweight fix boost their performance by 35% on LongBench? 👇🏼🧵. Github:.

RT @IdanShenfeld: The next frontier for AI shouldn’t just be generally helpful. It should be helpful for you!. Our new paper shows how to….

RT @YVinker: SketchAgent has been accepted to #CVPR2025 !.This is an early step toward new tools for visual thinking and richer interaction….

DeciMamba, the first context extension method for Mamba, is accepted to #ICLR2025! 🎉. New revision with more long-context results:. Special thanks to @ItamarZimerman @ShadyAbh @nadavcohen @amirgloberson @liorwolf @RGiryes !.