Checkout these awesome new real-robot online RL fine-tuning results that @andy_peng05 and @zhiyuan_zhou_ got with our WSRL method. WSRL appeared at ICLR earlier this year -- check this out for more details: 👇.

RT @setlur_amrith: Since R1 there has been a lot of chatter 💬 on post-training LLMs with RL. Is RL only sharpening the distribution over co….

If you are interested in learning more about some of the work discussed in the post, check out:. - e3: curricula for structured exploration & discovery (best <2B model on AIME/HMMT): - dense rewards for reasoning: - PAVs:.

Given the confusion around what RL does for reasoning in LLMs, @setlur_amrith & I wrote a new blog post on when RL simply sharpens the base model & when it discovers new reasoning strategies. Learn how to measure discovery + methods to enable it ⬇️.

And this work would not have been possible without the new CMU FLAME cluster ( with 256 H100 GPUs!.

RT @matthewyryang: 🚨 NEW PAPER: What if LLMs could tackle harder problems - not by explicitly training on longer traces, but by learning ho….

RT @setlur_amrith: Introducing e3 🔥 Best <2B model on math 💪.Are LLMs implementing algos ⚒️ OR is thinking an illusion 🎩.? Is RL only sharp….

Checkout's @setlur_amrith's post for more details, including a discussion of how in-context exploration is different from work that claims RL only "sharpens" around the base model's capabilities (largely due to the data/budgets being trained upon).

This work builds on some prior work & insights my students at CMU have done on test-time compute:. 1. RL vs SFT: 2. Blog: 3. Dense rewards:

This was a very fun collab, led by @setlur_amrith & @matthewyryang w/ @ianwu97 @sea_snell @JeremyGreerOumi @gingsmith and @max_simchowitz. I learned a lot!. Website: Paper: Code, training data, ckpts. are all released.

With these three ingredients, we amplify in-context exploration and get SOTA results under 2B! The e3-1.7B model does better than 7B (OpenThinker) & even some 32B (s1-32B) models, without any explicit prompting to use more compute!. More results in the paper, w/ didactic tasks

Doing so is critical, since training on very long budgets is super hard for RL from an optimization / variance perspective. And training on small budgets curbs useful "in-context exploration" and chaining. We also prescribe a thumb rule for choosing the "right" budgets given

Ingredient 3: A "coupled" curriculum .It is important to keep RL in the "chaining" regime & doing so requires training on the right data,w/ the right max. token budget. We train on easy problems at a lower budget (8k) and then train on hard problems at a longer budget of 16k.

Ingredient 2: Negative gradient.There's a debate b/w RL & SFT, but we show that slightly off-policy RL w/ neg. grad learns to chain these skills to use more test compute in a better way. Chaining ➡️ length 📈, verifications 📈➡️ perf 📈. Related to stitching in offline RL,

To get LLMs to extrapolate, we had 3 ingredients:. Ingredient 1: Asymmetries in base model.The base model needs to have asymmetric competence in different "skills" (generation/verification/etc.) s.t. chaining more skills ➡️ 📈perf. If so, RL can discover chaining strategies.

Now, let's dive into details!. Problem: we wanted to build a model that could extrapolate test-time compute, meaning that it could translate more compute into better in-context search to improve performance. This is the real test of how effectively models can learn to reason, I

Full results below. Key ingredients of e3:.🔑Use base model with asymmetric competence in generation/verification.🔑Use off-policy RL so that negative grad can learn to "chain" these skills, grow length, increase perf.🔑Use coupled curriculum over data + budget to control

Our view on test-time scaling has been to train models to discover algos that enable them to solve harder problems. @setlur_amrith & @matthewyryang's new work e3 shows how RL done with this view produces best <2B LLM on math that extrapolates beyond training budget. 🧵⬇️

Also check out our past work doing RL with agents:. DigiRL: Digi-Q: TTI extends this line of work along the axis of scaling interaction to enable rich behaviors, building on very similar machinery from these prior works.

This was a fun collab led by @JunhongShen1 @jackbai_jkb, w/ @LunjunZhang @YifeiZhou02 @setlur_amrith @atalwalkar and many others!. Paper: Website: Code: Please reach out if you have feedback!.