🌟Announcing NeurIPS spotlight paper on the transition from lazy to rich🔦 We reveal through exact gradient flow dynamics how unbalanced initializations promote rapid feature learning co-led @AllanRaventos and @ClementineDomi6 @FCHEN_AI @klindt_david @SaxeLab @SuryaGanguli

🚀 Exciting news! Our paper "From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks" has been accepted at ICLR 2025! https://t.co/6B7T1ROrc2 A thread on how relative weight initialization shapes learning dynamics in deep networks. 🧵 (1/9)

Join millions who have switched to Grok.

Wednesday, April 9th at 11AM: TTIC's Young Researcher Seminar Series presents Daniel Kunin (@KuninDaniel) of @StanfordEng with a talk titled "Learning Mechanics of Neural Networks: Conservation Laws, Implicit Biases, and Feature Learning." Please join us in Room 530, 5th floor.

1/ Our new paper: “Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning” on how to change training to better exploit test-time compute! co-led by @AllanRaventos, w/ Nan Cheng, @SuryaGanguli & @ShaulDr https://t.co/xM49OB6sk7

Excited to finally share this work w/ @SuryaGanguli. Tl;dr: we find the first closed-form analytical theory that replicates the outputs of the very simplest diffusion models, with median pixel wise r^2 values of 90%+. https://t.co/SYkAAh6k4C

Come check out our #NeurIPS2024 spotlight poster on feature learning tomorrow! 📍East Exhibit Hall A-C #2102 📅Thu 12 Dec 4:30 p.m. — 7:30 p.m. PST

Great job, it was an honor being part of this amazing project! Congrats to the team 💪

Also, big shoutout to @yasamanbb, @CPehlevan, and @HSompolinsky for coordinating last year's 'Deep Learning from Physics and Neuroscience' program @KITP_UCSB. Our amazing team met there, and this project is a direct result of the conversations we had!

Check out our work for more details: https://t.co/VK0Wa0tdCa ...and stay tuned for follow-ups!

We provide empirical evidence that an unbalanced rich regime drives feature learning in deep networks, promotes interpretability of early layers in CNNs, reduces sample complexity of learning hierarchical data, and decreases time to grokking in modular arithmetic

Applying our function space analysis to shallow ReLU networks, we find that rapid feature learning occurs from unbalanced initializations that promote faster learning in early layers, driving a large change in activation patterns, but a small change in parameter space

We find three regimes in function space: 1. lazy akin to linear regression, 2. rich akin to silent alignment (Atanasov et al. 2021), 3. delayed-rich initially lazy followed by rich We extend this analysis (with mirror flows and implicit biases) to wide & deep linear networks

We derive exact gradient flow solutions for a minimal two-layer linear model displaying lazy and rich learning, which reveals that the relative scale between layers influence feature learning through conserved quantities that constrain the geometry of learning trajectories

Reproducing Fig.1 in Chizat et al. 2019 we find that even at small overall scale, the relative scale between layers can transition the network between rich and lazy learning and the best generalization occurs at small scale and large relative scale!

Really cool theory project on feature learning. If you are at the HiLD workshop @icmlconf check it out!

Interested in exactly solvable models of learning dynamics and implicit bias? Come check out our "Get Rich Quick" poster at the HiLD Workshop @icmlconf at 10am! With @KuninDaniel, myself, @ClementineDomi6, @FCHEN_AI, @klindt_david, @SaxeLab, and @SuryaGanguli.

Reminder! Happening Tomorrow! @ELLISforEurope

To get the zoom link and get notified about other interesting talks check out

🚨@FCHEN_AI, @atsushi_y1230, and I will be presenting our NeurIPS2023 paper https://t.co/6NuqR0nRJA to ELLIS Mathematics of Deep Learning reading group tomorrow Feb 13 at 5pm CET. Join to learn more about stochastic collapse!

We are delighted to announce our next speakers for the @ELLISforEurope RG are fr @Stanford, the authors of the NeurIPS2023 paper (https://t.co/FNikSjwN97)-@FCHEN_AI @KuninDaniel @atsushi_y1230 & @SuryaGanguli on 13th Feb '24 @ 5pm CET on Zoom. Save the Date! Link to join the RG👇