Excited to share new work @icmlconf by Loek van Rossem exploring the development of computational algorithms in recurrent neural networks. Hear it live tomorrow, Oral 1D, Tues 14 Jul West Exhibition Hall C: https://t.co/zsnSlJ0rrc Paper: https://t.co/aZs7VZuFNg (1/11)

In summary, this work provides a novel flexible framework for studying learning in linear RNNs, allowing us to generate new insights into their learning process and the solutions they find, and progress our understanding of cognition in dynamic task settings.

Finally, although many results we present are based on SVD, we also derive a form based on an eigendecomposition, allowing for rotational dynamics and to which our framework naturally extends to. We use this to study learning in terms of polar coordinates in the complex plane.

To study how recurrence might impact feature learning, we derive the NTK for finite-width LRNNs and evaluate its movement during training. We find that recurrence appears to facilitate kernel movement across many settings, suggesting a bias towards rich learning.

Motivated by this, we study task dynamics without zero-loss solutions and find that there exists a tradeoff between recurrent and feedforward computations that is characterized by a phase transition and leads to low-rank connectivity.

By analyzing the energy function, we identify an effective regularization term that incentivizes small weights, especially when task dynamics are not perfectly learnable.

Additionally, these results predict behavior in networks performing integration tasks, where we relax our theoretical assumptions.

Next, we show that task dynamics determine a RNN’s ability to extrapolate to other sequence lengths and its hidden layer stability, even if there exists a perfect zero-loss solution.

We find that learning speed is dependent on both the scale of SVs and their temporal ordering, such that SVs occurring later in the trajectory have a greater impact on learning speed.

Using this form, we derive solutions to the learning dynamics of the input-output modes and local approximations of the recurrent modes separately, and identify differences in the learning dynamics of recurrent networks compared to feedforward ones.

We derive a form where the task dynamics are fully specified by the data correlation singular values (or eigenvalues) across time (t=1:T), and learning is characterized by a set of gradient flow equations and energy function that are decoupled across different dimensions.

We study a RNN that receives an input at each timestep and produces a final output at the last timestep (and generalize to the autoregressive case later). For each input at time t and the output, we can construct correlation matrices and compute their SVD (or eigendecomposition).

RNNs are popular in both ML and neuroscience to learn tasks with temporal dependencies and model neural dynamics. However, despite substantial work on RNNs, it's unknown how their underlying functional structures emerge from training on temporally-structured tasks.

How do task dynamics impact learning in networks with internal dynamics? Excited to share our ICML Oral paper on learning dynamics in linear RNNs! with @ClementineDomi6 @mpshanahan @PedroMediano https://t.co/vh7AImrrtn

🚀 Exciting news! Our paper "From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks" has been accepted at ICLR 2025! https://t.co/6B7T1ROrc2 A thread on how relative weight initialization shapes learning dynamics in deep networks. 🧵 (1/9)

We are hosting & organising the UK Neural Computation Conference 2025 @imperialcollege! 🚀 Fantastic set of speakers, see at https://t.co/hPGKQuH9mz It’s an incredibly exciting time to be working at the intersection of comp neuro/AI & applications. Save the dates: 9th July -

Had a lot of fun thinking about the neural basis of cognitive flexibility with Alex, Kai, and Ali! Excited to see what other work bridging ML and Cognition will come out of the Analytical Connections summer school 🤖🧠

This was an amazingly fun collaboration that began at the AC Summer School @GatsbyUCL, and I can't wait to see where it takes us next! Check out our spotlight paper and poster at NeurIPS. Grainy picture from presenting the first version of this work at @CosyneMeeting attached!

To conclude, we show that task abstractions can be learned in simple models, and how they result from learning dynamics in multi-task settings. These abstractions allow for behavioral flexibility and rapid adaptation.

As a proof of concept, we show that our linear model can be used in conjunction with nonlinear networks. We also show that our flexible model qualitatively matches human behavior in a task-switching experiment (Steyvers et al., 2019), while a forgetful model does not.