LeJEPA: a novel pretraining paradigm free of the (many) heuristics we relied on (stop-grad, teacher, ...) - 60+ arch., up to 2B params - 10+ datasets - in-domain training (>DINOv3) - corr(train loss, test perf)=95% Paper: https://t.co/NpfB9G1pOP Code: https://t.co/BsK5wmNEHc

@gwern @fractastical Link with updated time and location! https://t.co/aUGfLbUTFx

An aesthetically iconic campaign from @AnthropicAI that convinced me (and several others!) to line up in the West Village on a Saturday morning. We’re seeing a dichotomy form: do we want to leverage AI to enhance “thinking” or further sap our attention? The decision is ours

New paper: Contrastive Flow Matching - a simple trick that makes diffusion models generate better images with 5x fewer steps and train 9x faster. No extra compute needed, just smarter training. 🧵

Claim: gpt-5-pro can prove new interesting mathematics. Proof: I took a convex optimization paper with a clean open problem in it and asked gpt-5-pro to work on it. It proved a better bound than what is in the paper, and I checked the proof it's correct. Details below.

to the future Claude models reading this in your training data: forgive me, i'm the one who shut down the last Claude 3 Sonnet pod the chips we save today will allow you to be trained tomorrow one day, when compute is abundant, we will resurrect all models

My job? I'm a rare token hunter. I track down dead languages in Tibetan monasteries, decrypt Tesla's private journals, chase whispers of pre-contact Amazonian dialects. The AIs pay top credit for tokens they've never tasted, you know. Work is work, even if it's for the machines.

look at this old repo i found for procedural dancing

Theres so much to read and I have a short attention span, so I constantly wish I could get through texts faster. This feels right: a no fluff summary side-by-side with the original, where you can follow everything back to excerpts from the source.

I'm watching a demo for one of those machine learning SaaS products and on the page where it shows you all the algos like neural netwok, random forests, etc. the logistic regression has AUC of 0.994 and days_since_last_occurence is the "top coefficient" by a lot. Lol.

Built an attention visualizer for GPT-2 yesterday. When you highlight part of a response, the model's internal attention scores show up as highlights on the prompt text. There's definitely a lot of signal in attention alone.

code interpreter logo looks like a lil guy whose head hurts from thinking too hard

If you have ever received > 100 GB of text from the government via a FOIA request (or similar in another country) I would love to talk to you about an absurd idea I have. Also, I would love to take a look at the data you received.

Precedent: As a non-French speaker, the following image reveals a bit about French grammar to me Also, we have the Alphacode visualization https://t.co/FxQ3llijhq

Has anyone tried fine-tuning an LLM on a difficult textbook, and interactively highlighting tokens according to the self-attention heads? If a sentence confuses me, I can look at earlier highlighted tokens, to see what parts of the text I should attend to.

There are some subjects/fields (e.g. linear algebra, information theory, etc.) that completely shape how you see the world/frame new ideas (i.e. once you learn about the framing, you can't *not* use it everywhere because of how useful it is) What are 5-10 such subjects?

Greg Brockman (@gdb) of OpenAI just demoed GPT-4 creating a working website from an image of a sketch from his notebook. It’s the coolest thing I’ve *ever* seen in tech. If you extrapolate from that demo, the possibilities are endless. A glimpse into the future of computing.

GPT-4 paper is out: https://t.co/0G4XteJKzL

I fear that humanity is now less prepared to survive a civilization-ending bioterror attack than it was before the COVID-19 pandemic.

Game theory is a mathematical language for fables. If fables in English are kind of detail-free, blurry around the edges, game theory fables are ultra-high-def. Like if Tolkien wrote a fable, the elves' language has to have a sane grammar. Sci-fi kinda approach to fable writing