Transformers employ different strategies through training to minimize loss, but how do these tradeoff and why?. Excited to share our newest work, where we show remarkably rich competitive and cooperative interactions (termed "coopetition") as a transformer learns. Read on 🔎⏬

RT @AnthropicAI: New Anthropic Research: Project Vend. We had Claude run a small shop in our office lunchroom. Here’s how it went. https:/….

RT @danielwurgaft: 🚨New paper! We know models learn distinct in-context learning strategies, but *why*? Why generalize instead of memorize….

Excited to share this work has been accepted as an Oral at #icml2025 -- looking forward to seeing everyone in Vancouver, and an extra thanks to my amazing collaborators for making this project so much fun to work on :).

RT @joannejang: some thoughts on human-ai relationships and how we're approaching them at openai. it's a long blog post --. tl;dr we build….

Check out the full paper for more details, a great discussion, and extensive appendix: Huge shoutout to Yedi Zhang for leading this work, and Peter Latham and @SaxeLab for their mentorship throughout!.

As a meta point, theory on transformer learning has to make some assumptions to make progress. One interpretation of this paper is a progressive relaxation of these assumptions, and observing how they affect conclusions. More work in this vein coming soon :).

The paper also includes an extensive appendix, with derivations and additional results. Appendix G has a nice connection to our work on strategy coopetition (, now in this more theory-amenable setup. Excited for these connections to be further explored!

In addition to the main results from @SaxeLab's thread, we found that low-rank heads still show progressive learning, where now it happens in "chunks" with size equaling the ranks of the heads. A surprisingly clean generalization of the rank-1 result!

Was super fun to be a part of this work! Felt very satisfying to bring the theory work on ICL with linear attention a bit closer to practice (with multi-headed low rank attention), and of course, add a focus on dynamics. Thread 🧵 with some extra highlights.

RT @bneyshabur: @ethansdyer and I have started a new team at @AnthropicAI — and we’re hiring!. Our team is organized around the north star….

RT @OpenAI: We’re launching a research preview of Codex: a cloud-based software engineering agent that can work on many tasks in parallel.….

RT @AndrewLampinen: How do language models generalize from information they learn in-context vs. via finetuning? We show that in-context le….

RT @scychan_brains: Some years ago, I got trapped in a Massive Trough of Imposter Syndrome. It took more than a year to dig myself out of….

RT @OpenAI: We’re releasing PaperBench, a benchmark evaluating the ability of AI agents to replicate state-of-the-art AI research, as part….

RT @PoShenLoh: Oh my goodness. GPT-o1 got a perfect score on my @CarnegieMellon undergraduate #math exam, taking less than a minute to solv….

RT @ted_moskovitz: New work led by @Aaditya6284! This was a really fun and interesting project, and I think there are a lot of cool insigh….

RT @_rockt: Fascinating new paper on learning dynamics of in-context learners: "Strategy Coopetition Explains the Emergence and Transience….

RT @sama: we trained a new model that is good at creative writing (not sure yet how/when it will get released). this is the first time i ha….

RT @__nmca__: sometimes things are simpler: just stub out pandas (3/n)

RT @scychan_brains: This paper is dedicated to our close collaborator @FelixHill84, who passed away recently. This is our last ever paper w….