Our paper, “Make Haste Slowly: A Theory of Emergent Structured Mixed Selectivity in Feature Learning ReLU Networks” will be presented at ICLR 2025 this week (! We derive closed-form dynamics for some, remarkably linear, feature learning ReLU networks (1/9).

RT @raillabwits: Great news! This work has also been accepted into RLC for the Journal to Conference Track. Congrats @geraudnt and teams fo….

RT @kiradusterwald: Looking forward to presenting Tree-WSV for fast unsupervised ground metric learning at @iclr_conf 2025 tomorrow at 3 pm….

RT @ClementineDomi6: Had so much fun presenting our two posters at ICLR with @devonjarvi5 and @nico_anguita ! Thanks to everyone who came b….

RT @caandl_lab: CAandL Lab is at ICLR this week with 3 poster presentations. Come by and chat if you are interested! Congrats to all of the….

A huge thanks to my supervisors @kleinric, @BenjaminRosman and @SaxeLab for their constant assistance and guidance. Looking forward to chatting more on Thursday at 3pm, Poster #349! 👋 (9/9).

Finally, we provide an initial hidden layer clustering algorithm which is able to identify a ReLN for a given ReLU network, with the goal of enabling future work with our paradigm. (8/9)

We consider the effect of adding depth to the ReLU network and find that the network still favours linear pathways with mixed-selective representations. However, there is variance in the dynamics and we show a corresponding GDLN can still model the distribution of dynamics. (7/9)

We obtain closed-form dynamics for the ReLU network in this setting and see near perfect agreement with the predicted dynamics. We prove that the identified ReLN is unique and show that the mixed-selectivity preference remains when the number of contexts is increased. (6/9)

We then consider a complex nonlinear, contextual task. We find that the ReLU network has an inductive bias towards mixed-selective latent representations where no hidden neuron is selective for an item or context. Instead it couples linear pathways to favour learning speed. (5/9)

We demonstrate the paradigm on an extended XoR task which includes a third feature, making the dataset linearly separable. We observe a transition from the ReLU network using the linear strategy to the nonlinear strategy based on the magnitude of the new feature. (4/9)

We define the Rectified Linear Network (ReLN) as the GDLN which has the same output as the ReLU network at all points in time and prove that a ReLN always exists for a given ReLU network. Thus, we obtain the dynamics of learning for the ReLU networks as well! (3/9)

Our main idea is to map ReLU networks onto Gated Deep Linear Networks (GDLNs) (Saxe et al., 2022). GDLNs are neural networks which nonlinearly combine linear neural networks using gating operations. Importantly, GDLNs permit a dynamics reduction for their learning dynamics. (2/9)

More information can be found at our website: If you are interested, keep an eye out for more to come and get in touch! 🙌.

I'm excited to announce that I have received the Thuthuka Research Grant from the @NRF_News! I will use this grant to start my own lab focused on comp neuro and ML theory at @WitsUniversity: the @caandl_lab! This is a joint effort with @VictoriaWi83762, @stefsmlab and @geraudnt.

RT @unireps: 🚀Exciting News! 🚀. Join us for the first UniReps x @UCL_Neuro_AI Meetup at @ucl @SWC_Neuro on Sept 8, 2025! 🎉. 📍 Register now!….

RT @ClementineDomi6: Our paper, “A Theory of Initialization’s Impact on Specialization,” has been accepted to ICLR 2025!..

RT @BlavatnikAwards: 2025 @Blavatnikawards UK 🇬🇧 Finalist Andrew Saxe from UCL was featured on the @BBC Science Focus Instant Genius Podcas….

RT @Sahba_Besh: The countdown begins for our South African Brainhack!! Watch this space for ways to connect in virtually too! @CIFAR_News @….

RT @raillabwits: 🚀 Exciting News! Our RAIL Lab team has been helping organize and will be speaking at the ICRA@40 Africa satellite conferen….