Neural network trainings are nondeterministic. Repeated runs each produce a unique network, often with significantly _varying_ test-set performance. 🆕📜 I demonstrate that this variation has a simple statistical structure, and is harmless & inevitable https://t.co/1zzpNHi0Vy

TIL that Muon is in PyTorch stable now. Pretty cool.

How did an intern with no AI experience break a world record? Read our new blog post:

@yonashav Oops, I meant to say observable universe, not known universe.

Teslas have the lowest maintenance & repair costs of any brand

Update: @yonashav pointed out that if we allow FTL travel, then this bound fails because the mass (and therefore computational capacity) of the entire universe is potentially much larger than the known universe. So to get a bound, we do have to assume the impossibility of FTL.

Theorem: The maximum possible duration of the computational singularity is 470 years. Proof: The FLOPs capacity of all computers which existed in the year 1986 is estimated to be at most 4.5e14 (Hilbert et al. 2011). Based on public Nvidia revenue and GPU specs, this capacity

We challenged our intern @ramadan_al76760 (zero prior AI experience) to beat the CIFAR-10 training speed record using @hivergeai's algorithmic discovery engine. Result: Sub-2-second (!!) training for the first time ever.

Introducing DOJE: The first U.S. memecoin ETF giving you spot exposure to Dogecoin via a traditional ETF.

- TTA: Skip for easy examples - Thermal throttling: Sleep for 8s between runs (only affects average not record time) Note: The authors reported a time of 2.02 seconds. My reproduction (torch 2.7.0; hardware seen below) had a min time of 1.99s. Code: https://t.co/qPN6oebF5T 3/3

- Data aug: Add color jitter and vectorize random crop - Compilation: Compile xent fwd/bwd - Architecture: Replace GELU with SiLU, use SVD for first layer init, and use channels_last format with fp16 for all convs - Hparams: tweaks including bsz 2000 -> 1536 & epochs 8 -> 7.6 2/3

New CIFAR-10 training speed record: 94% in 1.99 seconds on one A100 Previous record: 2.59 seconds (Nov. 10th 2024) New record-holder: Algorithmic discovery engine developed by @hivergeai Changelog: - Muon: Vectorize NS iter and reduce frequency of 'normalize weights' step 1/3

Excited to release new repo: nanochat! (it's among the most unhinged I've written). Unlike my earlier similar repo nanoGPT which only covered pretraining, nanochat is a minimal, from scratch, full-stack training/inference pipeline of a simple ChatGPT clone in a single,

There's been significant recent progress in the NanoGPT speedrun. Highly recommend this post by @classiclarryd https://t.co/QNoI4wVAJg

Down to 146.8s on modded-nanogpt! https://t.co/OV0TaesL4I Surprising result: Different parameter groups have different sensitivity to batch size. Instead of picking a single batch size, grad accumulation can be managed on a param level to simulate different batch sizes.

Even with full-batch gradients, DL optimizers defy classical optimization theory, as they operate at the *edge of stability.* With @alex_damian_, we introduce "central flows": a theoretical tool to analyze these dynamics that makes accurate quantitative predictions on real NNs.

FUTURES TRADERS: Get 40% off all evals, no activation fees, end-of-day drawdown in our live-market PRO+ accounts…and still daily PRO payouts!

Like, I think there’s a strong case to ask for better citation. But it should be recognized that using residuals to get ImageNet SOTA has value in of itself. Otherwise it’s as if Fermat walked on stage with Andrew Wiles and said “guys what’s the big deal, I told you this already”

The problem with this line is that any history of neural network science that focuses only on the ideas, while ignoring the quality of evidence given for those ideas, is as incomplete as a history of math that focuses only on the theorems w/o the proofs https://t.co/G9p3V8m8an

…when it’s much more likely that only the explosion is characterized that way, not the future. Another easy inference: Neither global population-explosion nor population-collapse can be “tensions of the far-future as such.” Since they both stop themselves in the near-future.

For the purposes of Figuring Out Wtf Is Going On, it seems crucial to keep the explosion in mind. For example, someone looking only at the distinctive qualities of the present might say “the future is characterized by parents being perplexed by the lives of their children.” …

COACHES These are the guys you want to take with you to a fight 🪓 Class of 2026 Berserker Hammer Linebackers @PateBowers @ZachCullen8 @LukeFaulkner28 @CameronMelvin26 @Bo_davis18 @Josh_Mcc10 @Cason2411 @lemoine_luke27 @glilly_01 @704wurster @calebfowler14 @thomas_green3

The singular thing that we can say about the far-future is that it will almost-always *not* be undergoing an explosion. Since in a finite universe, exponentials run out. So in that sense the far-future will look more like the far-past than the present. https://t.co/pXMDnZWNZf

Random idea: It should be possible to make any dataset autoregressively unlearnable by prefixing each of its contexts with (encrypted ciphertext of that context, randomized decryption key, execution trace of decryption) => every next-token is either deterministic or noise-like.

Great to see this effort towards rigorous hyperparameter tuning. Two areas for improvement: 1. IIUC, the scaled up run here isn't actually tuned at all - its hparams are set via extrapolation 2. Sensitive hparams need a more granular sweep than power-of-2 https://t.co/O5vG58q3Wx