How does in-context learning emerge in attention models during gradient descent training? Sharing our new Spotlight paper @icmlconf: Training Dynamics of In-Context Learning in Linear Attention https://t.co/rr8T8ww5Kt Led by Yedi Zhang with @Aaditya6284 and Peter Latham

📢 Job alert We are looking for a Postdoctoral Fellow to work with @ArthurGretton on creating statistically efficient causal and interaction models with the aim of elucidating cellular interactions. ⏰Deadline 27-Aug-2025 ℹ️

🎓Thrilled to share I’ve officially defended my PhD!🥳 At @GatsbyUCL, my research explored how prior knowledge shapes neural representations. I’m deeply grateful to my mentors, @SaxeLab and Caswell Barry, my incredible collaborators, and everyone who supported me! Stay tuned!

If you’re working on symmetry and geometry in neural representations, submit your work to NeurReps and join the community in San Diego ! 🤩 Deadline August 22nd.

If you can see it, you can feel it! Thrilled to share our new @NatureComms paper on how mice generalize spatial rules between vision & touch, led by brilliant co-first authors @giulio_matt & @GtnMaelle. More details in this thread 🧵 (1/7) https://t.co/CvmDQqokiF

🥳 Congratulations to Rodrigo Carrasco-Davison on passing his PhD viva with minor corrections! 🎉 📜 Principles of Optimal Learning Control in Biological and Artificial Agents.

Come chat about this at the poster @icmlconf, 11:00-13:30 on Wednesday in the West Exhibition Hall #W-902!

👋 Attending #ICML2025 next week? Don't forget to check out work involving our researchers!

Excited to present this work in Vancouver at #ICML2025 today 😀 Come by to hear about why in-context learning emerges and disappears: Talk: 10:30-10:45am, West Ballroom C Poster: 11am-1:30pm, East Exhibition Hall A-B # E-3409

With this work we hope to shed light on some of the universal aspects of representation learning in neural networks, and one way networks can generalize infinitely from finite experience. (11/11)

Not all pairs of representations which can merge are actually expected to merge. Thus, the final learned automaton may look different for different training runs, even when in practice they implement an identical algorithm. (10/11)

The theory predicts mergers can only occur given enough training data and small enough initial weights, resulting in a phase transition between an overfitting regime and an algorithm-learning regime. (9/11)

Since these pairs share outputs, mergers do not affect the automaton's computation. With enough mergers, the automaton becomes finite, fixing its behavior for long sequences. If the training data uniquely specifies the task, this results in full generalization. (8/11)

Using intuitions based on continuity, we derive local interactions between pairs of representations. We find that pairs of sequences which always agree on target outputs after receiving any possible additional symbols will merge representations under certain conditions. (7/11)

We find two phases: -An initial phase where the RNN builds an infinite tree and fits it to the training data, reducing only the training loss. -A second merging phase, where representations merge until the automaton becomes finite, with a sudden drop in validation loss. (6/11)

To understand what is happening in the RNN, we extract automata from its hidden representations during training, which visualize the computational algorithm as it is being developed. (5/11)

When training only on sequences up to length 10, we find complete generalization for any possible sequence length. This cannot be explained by smooth interpolation of the training data, and suggests some kind of algorithm is being learned. (4/11)

To explore this, we consider the simplest possible setting in which a neural network implicitly learns an algorithm. We train a recurrent neural network on sequences of ones and zeros to predict if the number of ones is even or odd. (3/11)

How can the continuous gradient descent dynamics of deep neural networks result in the development of a discrete algorithm capable of symbolic computation? (2/11)

Excited to share new work @icmlconf by Loek van Rossem exploring the development of computational algorithms in recurrent neural networks. Hear it live tomorrow, Oral 1D, Tues 14 Jul West Exhibition Hall C: https://t.co/zsnSlJ0rrc Paper: https://t.co/aZs7VZuFNg (1/11)

How do task dynamics impact learning in networks with internal dynamics? Excited to share our ICML Oral paper on learning dynamics in linear RNNs! with @ClementineDomi6 @mpshanahan @PedroMediano https://t.co/vh7AImrrtn