I've written about 500+ machine learning papers in the past year. Here are some of my most popular threads: [1/n]

The era of "The Era of Pretraining is Over" is over

Not that I have a favorite recent project, but... 🧵 LLM judges are the popular way to evaluate generative models. But they have drawbacks. They're: * Generative, so slow and expensive. * Nondeterministic. * Uncalibrated. They don't know how uncertain they are. Meet PGRM!

Anyway, I'm not a lawyer and I make no claims about what should happen with AI and public data, but I hope this thread helped you understand a bit about what is happening. Further disclaimer: all of this is general information that hundreds, maybe thousands of people inside the

A consequence of this caring-about-rules spectrum is that there's a "compliance gap" in many evals. If you have an idea where a given company (or even academic lab) sits, you should be either more or less impressed by their numbers.

Legal limbo also means that both compliant and non-compliant actors look the same from the outside. The non-compliant say nothing about their data because they did violate rules, and the compliant say nothing because they only incur risk by saying anything.

This legal ambiguity gives rise to hilarious situations like technical leaders saying they "can't recall" what went into their models.

Second is that the law isn't settled here. This means that no one can safely say anything about their training data. They can't even say what they didn't train on because failing to say you didn't train on X could be construed as evidence that you did.

The compliant players have two problems. First is that you can't really get opt-in from the whole internet. Which means you can't get 100% contract coverage. Unless you're willing to not train on most of the internet, but that's throwing away a lot of useful data.

Paying more to cover their butts makes sense for them because: 1. It's worth it to reduce legal risk 2. Employees and leaders want to "do the right thing," or at least brand themselves as the good guys internally 3. It helps them acquire risk-averse corporate customers

The group that does care about compliance is more interesting. They strike deals with publishers and copyright holders whenever possible. They buy books. They carefully filter out content that might have been pirated. Etc.

So they train on anything they can get their hands on, including outputs from rival models (in violation of the terms of service).

The companies that don't care either aren't subject to these laws or are more worried about dying from irrelevance than lawsuits. Legal battles can drain you but not having a good enough product can kill you.

There are roughly two groups of actors: 1. Those that care about US + EU laws and regulations. 2. Those that don't. But both look the same from the outside.

While I briefly have no employer, let me tell you what's really happening with AI companies training on public data: [1/n]

Paper is here: https://t.co/o8Erc1YdY3 If you like this thread, consider sharing it or following my awesome coauthors @saanarkethayan (with a banger of a first-ever first-author paper) @gupta__abhay and @mansiege. Happy to answer questions in the comments. [11/11]

(except that it was way harder than that, we eliminated a bunch of hparams, and we got it to work at scale + for all the linears instead of just some of them). [10/11]

tl;dr: we saw muParametrization and Unit Scaling and were like:

Our changes also let you transfer hyperparameters to orders-of-magnitude larger models. I.e., you can tune your hparams on small models and trust that they’ll be good for large models too. This saves *tons* of compute vs sweeping hparams on large models. [8/11]

Making all these changes lets you just straight up cast your weights to fp8 during training, with no dynamic rescaling or anything. Plus you get better convergence, even in bf16. [7/11]

Second, instead of just adding the residual branch output to the residual stream, we take a linear combination of the two. If both input tensors have unit norm and you make the squared coefficients sum to 1, the output tensor tends to have unit norm as well. [6/11]