BERT is just a Single Text Diffusion Step! (1/n) When I first read about language diffusion models, I was surprised to find that their training objective was just a generalization of masked language modeling (MLM), something we’ve been doing since BERT from 2018. The first

In era of pretraining, what mattered was internet text. You'd primarily want a large, diverse, high quality collection of internet documents to learn from. In era of supervised finetuning, it was conversations. Contract workers are hired to create answers for questions, a bit

Mech interp and performance people need to join forces. Understanding model behavior at the lowest level can inform so much about good design at the kernel level.

Over 4 years into our journey bridging Convolutions and Transformers, we introduce Generalized Neighborhood Attention—Multi-dimensional Sparse Attention at the Speed of Light: https://t.co/9awxf2Ogt9 A collaboration with the best minds in AI and HPC. 🐝🟩🟧 @gtcomputing @nvidia

Just finished. Great book @brianchristian :)

We have released arguably the toughest academic benchmark to ever exist 🎉 It includes even Putnam-like math questions and grad-level math and physics questions from PhD qualifying exams lol Obviously GPT4 still can't perform well on our benchmark 😅