Come see our analysis of a Gaussian multi-index model #AISTATS2025 on Sunday at Hall A—E 183. My favorite result is when the dot product between the ideal vectors exceeds a threshold, gradient flow fails to separate them under correlation loss! 😎

Very exciting research direction ! 🙌🏼

cross-posted @bsimsek.bsky.social (I'm slowly migrating there)

This was done in collaboration with Amire Bendjeddou & Daniel Hsu. 🙌 Paper link:

Another result is a tight characterization of the time complexity of the gradient flow early in dynamics, a generalization of the known result for single-index to multi-index models. This generalization applies to arbitrary geometries as well. 🙃

Below this threshold, a mild overparameterization (log k factor) for k index vectors is sufficient to match the neurons to the ideal vectors due to a coupon collector argument. 🤠

When the ideal vectors form an equiangular frame, all learned weights converge to their average (no matter how much overparameterization is used), which has turned from a saddle to a local minimum after a certain threshold of the dot product. 😲

Searching for an exact inverse map from learned weights to ideal (concept) vectors is an intricate geometry question, even for "simple" idealized models. 🧐

I now have a new account on Bluesky https://t.co/rroIdJtlF8, follow me !

Here is a link to my talk on distillation for neural networks https://t.co/DPdLT3HEth at Les Houches together with many other talks on algorithmic theories of learning 🙌 Thanks to organizers @_brloureiro and Vittorio

Nice talk by Jarod Alper at JMM’25 🙌🏼

I don’t mind if o1 does not think clearly like humans. It’s great for computing formulas like integrals, even better in combination with Wolfram alpha 🙌🏼

Ekin Akyürek (@akyurekekin) builds tools for understanding & controlling algorithms that underlie reasoning in language models. You’ve likely seen his work on in-context learning; I'm just as excited about past work on linguistic generalization & future work on test-time scaling.

6/n My application package is available at https://t.co/N5QfbVdsyL, feel free to reach out! (n=6)

5/n I'm enthusiastic about championing Gaussian multi-index models as mathematically analyzable and insightful models for MLPs. I continue developing new results for this model, it is fun!

4/n Here is a blog post: https://t.co/XfBW2Vc07e You may enjoy this exposition if you like Toy Models of Superposition.

3/n It is unclear whether the LLMs in the wild can be robustly interpreted by ad-hoc methods. My approach is to analyze toy models that give insight into the non-linear feature compression of the MLPs. Fascinating math challenges are to be expected in this journey!

2/n Studying the loss landscape is essential for understanding deep learning optimization and generalization. I developed a combinatorial complexity framework that quantifies the non-convexity of the deep learning loss landscapes.

📢 I'm on the faculty job market this year! My research explores the foundations of deep learning and analyzes learning and feature geometry for Gaussian inputs. I detail my major contributions👇Retweet if you find it interesting and help me spread the word! DM is open. 1/n

The Center for Computational Mathematics at Flatiron Institute is hiring research fellows (postdocs) to start next year -- applications are due December 15. This would be a great position for those working in e.g. the theory/science of deep learning. https://t.co/TdsHvFEs3k