1/9 There is a fundamental tradeoff between parallelizability and expressivity of Large Language Models. We propose a new linear RNN architecture, DeltaProduct, that can effectively navigate this tradeoff. Here's how!

RT @JakeMRobertson: We present a new approach to causal inference. Pre-trained on synthetic data, Do-PFN opens the door to a new domain: PF….

RT @riccardograzzi: 📖 (1/n) DeltaProduct's theory got an update!. 1) For any nₕ>1 (# of Householders), only 3 layers are needed to solve al….

⚡DeltaProduct update with new results:.- Characterization of DeltaProduct’s state-tracking ability.- Inspection of the hidden state’s effective rank sheds light on why DeltaProduct extrapolates better to longer sequences than DeltaNet. - Improved scaling analysis.And more!

RT @behrouz_ali: What makes attention the critical component for most advances in LLMs and what holds back long-term memory modules (RNNs)?….

RT @SonglinYang4: 📢 (1/16) Introducing PaTH 🛣️ — a RoPE-free contextualized position encoding scheme, built for stronger state tracking, be….

RT @BlinkDL_AI: RWKV7-G1 "GooseOne" 🪿 1.5B release: pure RNN (attention-free) reasoning model, comparable with Qwen3 1.7B and fully multili….

@leloykun @jyo_pari DeltaProduct is now available in the flash-linear-attention library: .

RT @xiaolonw: Test-Time Training (TTT) is now on Video! And not just a 5-second video. We can generate a full 1-min video!. TTT module is a….

RT @maxmbeck: Yesterday, we shared the details on our xLSTM 7B architecture. Now, let's go one level deeper🧑‍🔧. We introduce. ⚡️Tiled Flash….

RT @riccardograzzi: @julien_siems @leloykun @jyo_pari In our DeltaProduct work we also add a bit of theory to DeltaNet, showing that it can….

9/9 Also take a look at these excellent blog posts:.by @leloykun.by @jyo_pari . We also discussed state-tracking in Linear RNNs at the ASAP Seminar—watch our full talk:

8/9 This was a great project with @timurcarstensen , @ZelaArber , @FrankRHutter , @MPontil , and @riccardograzzi .Check out our Oral at the FM-Wild Workshop at @iclr_conf :.

7/9 In language modeling tasks, DeltaProduct surpasses DeltaNet across lm-eval-harness benchmarks, with notable gains in length extrapolation performance as we increase nₕ.

6/9 Also on modular arithmetic with brackets, a context-free grammar, performance improves as nₕ increases.

5/9 To improve state-tracking, increasing the number of Householders nₕ is more effective than increasing the number of layers l: l=1,nₕ=2 (top row) yields much better performance than l=2 nₕ=1 (bottom row) on S₃, S₄, A₅. nₕ=4 gets good performance on S₅. nₕ=1↔DeltaNet

4/9 Building on this insight, DeltaProduct performs nₕ gradient steps per token (with different per-step keys and values), yielding a state-transition matrix A(xᵢ) as a product of nₕ generalized Householder transforms—interpolating between a rank-1 update and a dense matrix.

3/9 Following @SonglinYang4 et al. (2024), DeltaNet can be seen as performing one gradient descent step per token on an associative recall loss, resulting in a rank-1 state-transition matrix.

2/9 Linear RNNs’ expressivity depends on the state-transition matrix structure. Diagonal linear RNNs (Mamba, GLA, mLSTM) only allow token mixing. DeltaNet and RWKV-7 use a rank-1 update enabling token+channel mixing. DeltaProduct enables adjustable higher-rank updates—but how?.

RT @maxmbeck: 📢🔔I am excited to share the details on our optimized xLSTM architecture for our xLSTM 7B model!🚨. We optimized the architectu….