RT @ShaneBergsma: Power Lines paper now out: TL;DR - we identify how AdamW's weight decay should scale with batch….

(7/7) If you are looking to conduct research into deep models, contact us to collaborate! We are also hiring research scientists ( and research engineers (!.

(6/7) Implementing CompleteP is very simple, only requiring two lines of code. We provide a minimal implementation here:

(5/7) We propose a novel criterion called complete feature learning, which states that as depth increases, a model should not collapse to its linearization. Only CompleteP (α = 1) ensures both stable training and complete feature learning.

(4/7) CompleteP enables a wider range of model width/depth ratios to remain compute-efficient, unlocking shapes better suited for different hardware settings and operational contexts.

(3/7) The deeper the model, the more FLOP savings CompleteP (α= 1) has over µP. In 1.5B models, CompleteP saves 11.8% of FLOPs for optimal N:L and 34.4% FLOPs in the deepest models.

(2/7) CompleteP enables optimum HPs to remain stable for any depth, unlike popular parameterizations like SP, μP, and α=0.5. This dramatically reduces HP tuning budgets for deep models.

(1/7) @CerebrasSystems Paper drop: TLDR: We introduce CompleteP, which offers depth-wise hyperparameter (HP) transfer (Left), FLOP savings when training deep models (Middle), and a larger range of compute-efficient width/depth ratios (Right). 🧵 👇

Published "Neuron-based explanations of neural networks sacrifice completeness and interpretability" in TMLR 2025!. TL;DR: The most important principal components provide more complete and interpretable explanations than the most important neurons.

RT @AiEleuther: 🎉We're excited to announce our joint work with @Cerebras on a new guide to Maximal Update Parameterization (μP) and μTransf….

RT @davisblalock: So, uh, it turns out that 30+ years of neural net sparsity research have been confounded by optimal hyperparameters varyi….

RT @CerebrasSystems: (1/n) Paper drop: TLDR: We introduce the sparse maximal update parameterization (SμPar), whic….

RT @vithursant19: Successfully ported @karpathy's nanoGPT to the new @Apple MLX framework, possibly enabling quick prototyping of training….

RT @CerebrasSystems: 📣 Paper drop: Position Interpolation Improves ALiBi Extrapolation . We found a simple method to 2x the context lengt….

RT @CerebrasSystems: We just dropped the BTLM-3B-8K paper on arXiv! It distills our recipe for training SOTA LLMs:.- Extensively deduplicat….

RT @CerebrasSystems: Cerebras BTLM-3B-8K model crosses 1M downloads🤯.It's the #1 ranked 3B language model on @huggingface!.A big thanks to….

RT @CerebrasSystems: 📣 New dataset drop!.Introducing SlimPajama-627B: the largest extensively deduplicated, multi-corpora, open-source data….

RT @CerebrasSystems: 🚨 New podcast: how we made Cerebras-GPT with @DeyNolan and @QuentinAnthon15. A deep look on what it's like to train….

You can fine-tune or pre-train on a Cerebras Wafer-Scale Cluster today using Cerebras AI Model Studio: Very proud of the @CerebrasSystems team for this work. Happy to answer any questions here!

By using weight-streaming, Cerebras systems can scale from 1B to 1T parameters without any alteration in the model or training workflow. Cerebras-GPT provides the first demonstration of this type of scaling.