While at Meta, I worked on this optimizer-wrapper (outer step lookahead momentum) we're calling Snoo ( https://t.co/SSZLcYNXzp). You can use it with AdamW or Muon and see really strong scaling. Here's a plot where we ran it against (tuned) AdamW up to 1e23 training flop scales.

Fast, Numerically Stable, and Auto-Differentiable Spectral Clipping via Newton-Schulz Iteration Hi all, I'm bacc. I have a lot to talk about, but let's start with this fun side-project. Here I'll talk about novel (?) ways to compute: 1. Spectral Clipping (discussed in Rohan's

Ever wonder why switching optimizers never actually helps? Introducing HeavyBall Benchmark

By the way, I've also re-evaluated Llama 4 Scout via the Together API. Happy to report that they've fixed whatever issues they'd had earlier, and the score jumped from 66.83% to 74.27%!

Today is the start of a new era of natively multimodal AI innovation. Today, we’re introducing the first Llama 4 models: Llama 4 Scout and Llama 4 Maverick — our most advanced models yet and the best in their class for multimodality. Llama 4 Scout • 17B-active-parameter model

As of today, Llama 4 Maverick offers a best-in-class performance to cost ratio with an experimental chat version scoring ELO of 1417 on LMArena. It's wild to think Llama was a research project a couple of years ago & amazing to see how much progress we've made in the last two

Introducing our first set of Llama 4 models! We’ve been hard at work doing a complete re-design of the Llama series. I’m so excited to share it with the world today and mark another major milestone for the Llama herd as we release the *first* open source models in the Llama 4

GSM8K has been a cornerstone benchmark for LLMs, but performance seemed stuck around 95%. Why? Turns out, the benchmark itself was noisy. We fixed that, and found that it significantly affects evals. Introducing GSM8K-Platinum! w/@EdwardVendrow @josh_vendrow @sarameghanbeery

Have you ever spent hours chatting with your favorite character? 😍 Well, it turns out you're not alone! Today, users spend an average of two hours talking to their characters, and with our new C1.2 update, characters are more helpful than ever before! 🤖💬 https://t.co/bwi8aAGG2I

Announcing our Series A and our new AI model, C1.2!

https://t.co/BaHlD9Vg2O is in the news (NYTimes and TechRadar). Read about it here: https://t.co/xHLe383wmX,

Introducing Character What if you could create your own AI, and it was always available to help you with anything? Imagine everything it could do for you… https://t.co/L1X32llikX

https://t.co/vk1dPe3UaW becomes multimodal! Stay tuned, we got quality, consistency and personalization improvements on the way.

Not quite deepfake-level yet (source: https://t.co/m0grkqa2iq)

To test out the chatbot capabilities of @character_ai characters, I did what any reasonable person would do: put Joe Biden and Donald Trump together in a D&D party (moderated by a DM). What happened ended up being surprisingly touching. Well, at least some parts. A thread 🧵:

Here's what I've been working on! https://t.co/brp3ICgtD4 Check out our beta - create your own characters, and talk to more! We're excited to see what you think!

Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer By transferring from 40M parameters, µTransfer outperforms the 6.7B GPT-3, with tuning cost only 7% of total pretraining cost. abs: https://t.co/kYiuGDiUpE repo: https://t.co/TG4eZHErto

There's no evidence that SGD plays a fundamental role in generalization. With totally deterministic full-batch gradient descent, Resnet18 still gets >95% accuracy on CIFAR10. With data augmentation, full-batch Resnet152 gets 96.76%. https://t.co/iwIqQd7U1O

1/ Our new work: "How many degrees of freedom do we need to train deep networks: a loss landscape perspective." https://t.co/O7UKkSOO63 We present a geometric theory that connects to lottery tickets and a new method: lottery subspaces. w/ @_BrettLarsen @caenopy @stanislavfort

GPT-3 175B is powerful but too expensive to serve many finetuned copies. We use low-rank adaptation (LoRA) to learn task modules that are 10,000x smaller and can be swapped while the main model is frozen. No extra inference latency or quality drop! Paper: https://t.co/Nz7aMrgRDj