Our latest model DBRX is out today! It’s a fast 132B MoE and currently the best open weights model. DBRX really shines at programming, beating all OSS models in the HumanEval programming benchmark, even code fine tuned models like CodeLLaMA-70B

https://t.co/UGpxCbyE2Y

Presenting MultiverSeg — a scalable in-context system for interactively segmenting new datasets — at #ICCV2025 today! 📍poster 110 (10:45 AM–12:45 PM)

Efficient training of neural networks is difficult. Our second Connectionism post introduces Modular Manifolds, a theoretical step toward more stable and performant training by co-designing neural net optimizers with manifold constraints on weight matrices.

RLVR and test-time compute are a powerful combo for enterprises, so much so that @databricks now leads overall BIRD single-model leaderboard. This isn't about BIRD, though. It's an example of what our customers are accomplishing in their domains with our RL recipe in Agent Bricks

This is the most rewarding thing I did during my PhD, looking forward to teaching an updated version!

Deep learning training is a mathematical dumpster fire. But it turns out that if you *fix* the math, everything kinda just works…fp8 training, hyperparameter transfer, training stability, and more. [1/n]

*LoRA Learns Less and Forgets Less* is now out in its definitive edition in TMLR🚀 Checkout the latest numbers fresh from the @DbrxMosaicAI oven 👨‍🍳

Pretraining data ablations are expensive: how can we measure data quality fast and cheap? If you're at ICML, come find out at the ES-FoMo poster session today in Lehar 2 at 1 pm:

some notes from paper! - 405B trained on 15.6T tokens, 3.8e25 flops - use SFT, rejection sampling and DPO - annealing is used to judge quality of domain specific data (s/o dbrx paper)

Popular #LLM scaling laws only factor in training costs, and ignore the costs of deployment. In a paper presented at @icmlconf 2024, @databricks Mosaic AI researchers Nikhil Sardana, @JacobianNeuro, and @sashadoubov propose a modified scaling law that considers the cost of both

People think LoRA is a magic bullet for LLMs. Is it? Does it deliver the same quality as full finetuning but on consumer GPUs? Though LoRA has the advantage of a lower memory footprint, we find that it often substantially underperforms full finetuning. However, it forgets less

An interesting bit of nuance missing from throughput charts like this is that tokens != generated text. Because DBRX / LLama3 / GPT4s tokenizer has a larger vocabulary (100k+) they actually generate much faster then (20-30%) then tokens alone will measure compared to say Mixtral

@deepwhitman @AIatMeta @lmsysorg no. people misunderstand chinchilla. chinchilla doesn't tell you the point of convergence. it tells you the point of compute optimality. if all you care about is perplexity, for every FLOPs compute budget, how big model on how many tokens should you train? for reasons not fully

Nice! @KagiHQ now includes links to hackernews and reddit discussion threads directly in the search results. It's the small things

DBRX is the best open model on AI2 WildBench! 😀

🧱DBRX🧱 is so good that it forced 3-4 companies to release "competing" LLMs in the last two days (and we've barely heard about them). Some of my thoughts are summarized below... Prior research from Mosaic. DBRX is the next model in the series of Open LLMs released by Mosaic.

TFW the code eval metrics are so good your boss does his hair blue

Introducing DBRX: A New Standard for Open LLM 🔔 https://t.co/0HpI6Sdv6J 💻 DBRX is a 16x 12B MoE LLM trained on 📜 12T tokens 🧠DBRX sets a new standard for open LLMs, outperforming established models on various benchmarks. Is this thread mostly written by DBRX? Yes! 🧵

ML+art project with @dmshanmugam, @jjgort, and artist, @agnieszkakurant ! Our GAN-based approach generates signatures containing features learned from a collection of MIT and Cambridge residents’ signatures. #creativeAI #MachineLearning @mit_caml https://t.co/mZCzsXFGz6