The Transformer–SSM retrieval gap is driven by just a few heads!. SSMs lag on tasks like MMLU (multiple-choice) and GSM8K (math) due to in-context retrieval challenges. But here’s the twist: just a handful of heads handle retrieval in both architectures. What we found 👇 1/

RT @rbuit_: Despite theoretically handling long contexts, existing recurrent models still fall short: they may fail to generalize past the….

RT @HanGuo97: We know Attention and its linear-time variants, such as linear attention and State Space Models. But what lies in between?. I….

RT @orvieto_antonio: We have a new SSM theory paper, just accepted to COLT, revisiting recall properties of linear RNNs. It's surprising….

RT @abk_tau: New work! 🚨. Recurrent LLMs like Mamba and RWKV can efficiently process millions of tokens, yet still underperform on real-wor….

For more details, experiments, and cool insights, check out:.📄 Full paper: [.📓 Notebook: [. This is joint work with @ericxing and @_albertgu .6/.

And this is why Hybrid models work 🦸.They solve the G&A bottleneck in SSMs. > During training, Aggregate roles go to attention. > Pretrained SSM improves sharply when one Aggregate is replaced with attention. Where should you place attention?.👉 Where G&A heads emerge. 5/

Although both architectures develop G&A heads, SSMs’ smoother attention maps weaken the implementation of these heads compared to Transformers. That is, SSMs handle language and memory well, but retrieval bottlenecks in a few G&A heads 🤏.4/.

What makes that head so special?.It’s part of a two-head mechanism: Gather-and-Aggregate (G&A) Heads. >Gather: pulls relevant info.>Aggregate: combines it to answer. Transformer and SSM seem different—but we find that both use G&A!.And in both, a few heads do all the work 💪.3/

We removed one head from a pruned 4B Llama. 📉 MMLU dropped: 66% → 25%.The model still knew the answer, but couldn’t retrieve the correct letter (A–D) from context. The same phenomenon appears in SSMs 🐍.2/.

RT @electronickale: ✨ Love 4o-style image generation but prefer to use Midjourney? Tired of manual prompt crafting from inspo images?. PRIS….

RT @kevinyli_: At #ICLR2025 to present two recent works on reasoning distillation and efficient VLM inference with my wonderful collaborato….

RT @ashertrockman: Are you a frontier lab investing untold sums in training? Are you trying to stay competitive? Are you finding that your….

RT @LiaoIsaac91893: Scores 4.17% on ARC-AGI 2 on Kaggle!. 🔗 Code provided in the Kaggle notebook:

RT @KrithikTweets: 🧬 Meet Lyra, a new paradigm for accessible, powerful modeling of biological sequences. Lyra is a lightweight SSM achievi….

RT @_albertgu: Announcing Cartesia’s Series A towards our mission of building real-time intelligence. I’m cooking up some new models in the….

Amazing!. Small & efficient on-device Llamba reduces communication with larger cloud models—pushing the edge forward 🚀. Check it out ->

RT @PranjalAggarw16: What if you could control how long a reasoning model “thinks”?. Presenting L1-1.5B, an RL-trained reasoning model with….

A joint work with @TobiasKatsch @nimit_sohoni @jundesai @_albertgu from @cartesia_ai.

6️⃣ Try it out!.We’ve released an optimized implementation for edge deployment:.🔗 For more details, check out:.📄 The full paper: 📝 The blog post: