Emergent in-context learning with Transformers is exciting! But what is necessary to make neural nets implement general-purpose in-context learning? 2^14 tasks, a large model + memory, and initial memorization to aid generalization. Full paper 🧵👇(1/9)

RT @LauraRuis: Revisiting @LouisKirschAI et al.’s general-purpose ICL by meta-learning paper and forgot how great it is. It's rare to be ta….

AI Scientists will drive the next scientific revolution 🚀.Great work towards automating AI research @_chris_lu_.@RobertTLange @cong_ml @j_foerst @hardmaru @jeffclune.

RT @FaccioAI: Had a great time presenting at #ICML2024 alongside @idivinci & @LouisKirschAI. But the true highlight was @SchmidhuberAI hims….

RT @JacksonMattT: Meta-learning can discover RL algorithms with novel modes of learning, but how can we make them adapt to any training hor….

RT @hardmaru: Amazing that @SchmidhuberAI gave this talk back in 2012, months before AlexNet paper was published. In 2012, many things he….

RT @agopal42: Excited to present “Contrastive Training of Complex-valued Autoencoders for Object Discovery“ at #NeurIPS2023. TL;DR -- We in….

RT @SRSchmidgall: There is still a lot we can learn from the brain in artificial intelligence. In our new review article, we delve into th….

RT @DFinsterwalder: But looking deeper into GPT and its capacity for in-context learning (ICL) is fascinating. Recent works on ICL (like th….

RT @jakeABeck: Excited to share our new survey paper of meta-RL!.📊🤖🎊. Many thanks to my co-authors for the hard wo….

RT @ThomasMiconi: To start the new year (🥳) I'd like to highlight 2 recent papers that ask essentially the same question, but from very di….

Video and poster at

This paper is the outcome of an incredibly fun internship with @Luke_Metz @jmes_harrison @jaschasd at @GoogleAI over the summer.

Finally, meta-optimization can be tricky. We observed that learning gets stuck on loss plateaus at the beginning of training. Just slightly biasing the training distribution to allow for memorization followed by generalization mitigates this issue. (9/9)

Does this only work for Transformers? No! We tried a range of architectures. Compared to scaling laws, the number of parameters does not predict the learning-to-learn ability too well. Instead, what matters is how much memory (or the activation bottleneck) the model has. (8/9)

When meta-training on around 2^14 tasks (on the transition boundary) with multiple seeds, at the end of meta-training, the network either implements system identification or general learning-to-learn but rarely solutions in between. (7/9)

As we increase the number of tasks we observe several phase transitions: (1) instances are memorized with no learning, (2) tasks from the training are identified, (3) a general learning algorithm is implemented. The final transition is surprisingly discrete. (6/9)

When meta-testing the learned models (inference), we observe typical learning behavior - the bigger the training dataset in-context, the better the predictions! This generalizes to datasets not seen during training (eg train on MNIST, test on FashionM, CIFAR10, etc). (5/9)

By conditioning on a task-specific training dataset, models with many parameters trained on lots of tasks exhibit generalization to tasks that are not in the training distribution. Is this zero-shot generalization or learning-to-learn? (4/9)

To make black-box models like Transformers generalize we need to train on many tasks. We take existing datasets and generate variants by random transformation - linear projections and label permutations. This allows us to ask fundamental questions about learning-to-learn. (3/9)

General-purpose in-context learners (GPICL) allow us to solve new ML problems without relying on backpropagation and SGD! A model takes in training data, and produces test-set predictions, without any explicit inference model, training loss, or optimization algorithm. (2/9)