New paper! Rare SAE dataset approach: We train Sparse Autoencoders using only synthetic data generated by the model itself, revealing features that truly reflect what’s inside the model.

My group & collaborators have developed many popular benchmarks over the years, e.g., MMLU, MATH, APPS---really excited about our latest benchmark OMEGA Ω: 🔍Can LLMs really think outside the box in math? a new benchmark probing 3 axes of generalization: 1️⃣ Exploratory 2️⃣

"Rank the Rankers" from CMU! #RAG #MCP

Beginning of autonomous bug discovery & defense! 🔥 AI agents now match elite hackers — 15 zero-days found, $30K+ bugs patched. Huge milestone by @dawnsongtweets & team!

In Large Language Monkeys, we showed the scaling laws of inference-time compute with repeated sampling--the power law relationship between the number of repeated attempts and the fraction of problems solved! The following amazing work theoretically proves the necessary and

🚨New Breakthrough in Tip-of-the-Tongue (TOT) Retrieval Research! We address data limitations and offer a fresh evaluation method for the TOT complex queries. Curious how TREC TOT track test queries are created? Check out this thread🧵 and our paper📄:

The most bullish AI capability I'm looking for is not whether it's able to solve PhD grade problems. It's whether you'd hire it as a junior intern. Not "solve this theorem" but "get your slack set up, read these onboarding docs, do this task and let's check in next week".

Agency > Intelligence I had this intuitively wrong for decades, I think due to a pervasive cultural veneration of intelligence, various entertainment/media, obsession with IQ etc. Agency is significantly more powerful and significantly more scarce. Are you hiring for agency? Are

We are releasing CodeMonkeys, a system for solving SWE-bench problems with a focus on careful parallel and serial scaling of test-time compute! CodeMonkeys solves 57.4% of issues on SWE-bench Verified and and running our selection mechanism on an ensemble of existing top

Announcing LANG-JEPA — a new language model architecture I’m working on that optimizes in “concept” space instead of “token” space. Inspired by @ylecun's JEPA (I-JEPA for images, V-JEPA for video), LANG-JEPA asks: What if we train for conceptual understanding directly, rather

Thanks for covering our work on test time scaling! Turns out repeated sampling alone is surprisingly effective (~ log linear relationship between num samples and coverage across many reasoning tasks) and even better if combined with sequential “thinking”!

New Anthropic research: Alignment faking in large language models. In a series of experiments with Redwood Research, we found that Claude often pretends to have different views during training, while actually maintaining its original preferences.

Excited to share our NeurIPS 2024 Oral, Convolutional Differentiable Logic Gate Networks, leading to a range of inference efficiency records, including inference in only 4 nanoseconds 🏎️. We reduce model sizes by factors of 29x-61x over the SOTA. Paper: https://t.co/Aptk35mKir

I read Google's paper about their quantum computer so you don't have to. They claim to have ran a quantum computation in 5 minutes that would take a normal computer 10^25 years. But what was that computation? Does it live up to the hype? I will break it down.🧵

Training Large Language Models to Reason in a Continuous Latent Space Introduces a new paradigm for LLM reasoning called Chain of Continuous Thought (COCONUT) Extremely simple change: instead of mapping between hidden states and language tokens using the LLM head and embedding

AI as AI compiler? Very excited to release KernelBench, a new code generation benchmark for evaluating models' ability to generate correct and efficient CUDA kernels. KernelBench has 4 levels: Level 1 (100 tasks): Single-kernel operators (e.g. matmuls) Level 2 (100 tasks):

AI for AI chips for AI!

After the news was announced, John and Demis reunited with their teams in London. A snapshot of what they had to say ↓

Winning the @NobelPrize is the honour of a lifetime and the realisation of a lifelong dream - it still hasn’t really sunk in yet. With AlphaFold2 we cracked the 50-year grand challenge of protein structure prediction: predicting the 3D structure of a protein purely from its

Timely and fascinating research on LLM security in multi-agent systems! The discovery of 'Prompt Infection' highlights the vulnerabilities of larger models and the critical need for robust safeguards.