How do LLMs learn to reason from data? Are they ~retrieving the answers from parametric knowledge🦜? In our new preprint, we look at the pretraining data and find evidence against this:. Procedural knowledge in pretraining drives LLM reasoning ⚙️🔢. 🧵⬇️

RT @alexUnder_sky: @LauraRuis yeah, Akbir might be the goat of AI safety . And I like his Shaper (multi-agent opponent modelling as a form….

Highly recommend reading this, or at least the intro and conclusion. Some gems about the future of safety research.

The goal isn't perfect logic. It's reasoning that contributes to knowledge, builds understanding, and makes human life better. Given the fact that AI doesn't share many of human's inherent limitations, achieving this could be very powerful.

Instead of forcing AI to mimic formal logic, we should see reasoning as fundamentally social and goal-oriented. The question becomes: How do we align AI reasoning with human values when we can't logically verify every step?.

Reasoning then works best not in isolation, but embedded in communities that coordinate towards shared goals. It's a social technology, not just a purely cognitive one.

Through this lens, reasoning works just fine. It helps us convince others, defend our positions, and build shared understanding.

Mercier & Sperber's interactionist theory offers a reframe: reasoning didn't evolve for individual cognition. It evolved for social cooperation: to justify our beliefs, build trust, and coordinate with others.

Human reasoning is famously flawed: biased toward existing beliefs and vulnerable to misinformation. If reasoning evolved to enhance cognition, why is it so broken?.

Logical reasoning isn't how humans naturally think, it's our attempt to reverse-engineer intuition into rules. Perhaps not coincidentally, AI only became useful when we abandoned rule-based systems.

RT @EkdeepL: 🚨New paper! We know models learn distinct in-context learning strategies, but *why*? Why generalize instead of memorize to low….

RT @MinqiJiang: Recently, there has been a lot of talk of LLM agents automating ML research itself. If Llama 5 can create Llama 6, then sur….

RT @ammar__khairi: 🚀 Want better LLM performance without extra training or special reward models?.Happy to share my work with @Cohere_labs….

RT @_rockt: Fantastic work by @JonnyCoook and @silviasapora on "Programming by Backprop: LLMs Acquire Reusable Algorithmic Abstractions Dur….

RT @silviasapora: 🧵 Check out our latest preprint: "Programming by Backprop". What if LLMs could internalize algorithms just by reading cod….

Many more interesting findings in the preprint: Awesome work by first authors @JonnyCoook and @silviasapora , and collaborators @aahmadian_ , @akbirkhan , @_rockt , and @j_foerst.

We just discovered this and models show limited performance, but we have many follow-up questions. We find big improvements with data and model scaling, so how far can we take this? How many programs can we store in the LLM? Can models internalise things like recursion or search?.

We hypothesise that PBB may be an additional reason why code training helps downstream reasoning; models internalise reusable algorithmic abstractions.

Strikingly, using RL in the second stage allows models to evaluate programs seen in an earlier SFT stage (without I/O pairs). By contrast, SFT is much worse at this backward generalisation. Our ablations indicate it's the on-policy nature of RL enabling this.

We also find LLMs can sometimes compose programs only trained on in isolation. Training on foo() and bar() as standalone code snippets allows models to eval baz(x) = foo(bar(x)). GPT-4o can do this w/o CoT, evaluating compositions in-weights (a form of out-of-context reasoning).

We find models can learn an input-general understanding of algorithms from a *single* piece of code. This indicates training LLMs with next-token prediction on source code can overcome (part of) the embers of autoregression (@RTomMcCoy).