Attention has been the key component for most advances in LLMs, but it can’t scale to long context. Does this mean we need to find an alternative? . Presenting Titans: a new architecture with attention and a meta in-context memory that learns how to memorize at test time. Titans

RT @gmongaras: Threw a paper I've been working on onto ArXiv. Trying to get a little closer to understanding why softmax in attention works….

RT @behrouz_ali: What makes attention the critical component for most advances in LLMs and what holds back long-term memory modules (RNNs)?….

Everyone is talking about reviewers who don't engage or provide low-quality reviews. While harmful, I don't see that as the biggest threat to the peer review system. As both an author and reviewer, I'm seeing zero-sum debates where a reviewer puts their full effort into rejecting.

RT @mirrokni: Proud to announce an official Gold Medal at #IMO2025🥇. The IMO committee has certified the result from our general-purpose Ge….

RT @reza_byt: 📄 New Paper Alert! ✨. 🚀Mixture of Recursions (MoR): Smaller models • Higher accuracy • Greater throughput. Across 135 M–1.7 B….

RT @yingheng_wang: ❓ Are LLMs actually problem solvers or just good at regurgitating facts?. 🚨New Benchmark Alert! We built HeuriGym to ben….

RT @tdietterich: The scope of what counts as research has narrowed considerably.

RT @leloykun: Fast, Numerically Stable, and Auto-Differentiable Spectral Clipping via Newton-Schulz Iteration. Hi all, I'm bacc. I have a l….

Very interesting work!.

RT @pmddomingos: The ratio of science to engineering in AI is approaching zero.

RT @TheTuringPost: Last week, @Google dropped a paper on ATLAS, a new architecture that reimagines how models learn and use memory. Unfort….

@mirrokni @meisamrr Here is the link to the paper: This is a work with Zeman Li, Praneeth Kacham, @daliri__majid, @yuandeng_cs, Peilin Zhong, @meisamrr, and @mirrokni.

@mirrokni @meisamrr In our experiments, we focus on language modeling, common-sense reasoning, needle in a haystack, in-context recall, and multi-query associative recall tasks. Atlas is very effective in all scales (tested up to 1.3B), even outperforming Titans, and other linear RNNs in long

@mirrokni @meisamrr How to incorporate the memory?. We follow Titans and use the variants of Memory as Context (MAC), as Gate (MAG), and as Layer (MAL) but without persistent memory tokens in our experiments.

@mirrokni @meisamrr Can we use what we have learned from the design of long-term neural memory (Atlas) and further enhance attention and Transformers?. We aim to strictly generalize original softmax Transformers from two important aspects: (1) Deep Memory: Transformers use matrix-value memories,.

@mirrokni @meisamrr Even with a powerful surprise metric and enhanced memory capacity, the memory needs to properly be updated and optimized. In fact, a bad update rule can cause the memory to be stuck in local optima and so does not properly memorize the context. While almost all models are based

@mirrokni @meisamrr Now that we addressed the first drawback, how do we enhance the memory capacity (i.e., the number of data samples that the memory can store in its parameters)?. Attention acts as an unbounded associative memory that tries to learn the mapping between a set of queries and a set of

@mirrokni @meisamrr How does memory prune the context? We provide additional flexibility for the model by gamma parameters to prune the context, whenever it is needed. This is similar to forgetting but with more direct access to the local tokens. That is, the model can simply ignore past tokens by

@mirrokni @meisamrr From the memory perspective: Our brain prioritizes events that violate the expectations (being surprising). While an event itself consists of different elements, the judgment of prioritization depends on them all. In Titans, however, the model computes the surprise metric for

@mirrokni @meisamrr Coming back to the first question: What makes attention the critical component for most advances in LLMs and what holds back long-term memory modules (RNNs)?. We observe three disjoint aspects that limit the performance of long-term memory modules in long context tasks: (1)