🚀 Big time! We can finally do LLM RL fine-tuning with rewards and leverage offline/off-policy data!. ❌ You want rewards, but GRPO only works online?.❌ You want offline, but DPO is limited to preferences?.✅ QRPO can do both!. 🧵Here's how we do it:

RT @MiTerekhov: Well, to avoid steganography, let's make sure our multi-agent LLM research workflows are composed of agents with different….

RT @XiuyingWei966: If you’re interested in long-context efficiency, don’t miss our recent paper RAT—a joint effort with @anunay_yadav, Razv….

RT @MiTerekhov: AI Control is a promising approach for mitigating misalignment risks, but will it be widely adopted? The answer depends on….

RT @caglarml: I am proud that our latest work on a novel RL method for foundation models/LLMs is finally out!. 1️⃣ Why does QRPO matter?.Al….

RT @matrs01: I couldn’t be prouder to share this. 🎉.Our work on Quantile Reward Policy Optimization (QRPO) for LLM RL‑finetuning bridged de….

I’m really proud of this work! It’s been an amazing collaboration with @matrs01 and @caglarml!. 📰 Paper: Hidden gems and open questions in the 30+ page appendix💎. 🧑‍💻 Code: Star to show interest⭐. 🌐 Blog:

QRPO is a framework. You can shape the optimal policy! 🎛️. We derive a framework around QRPO for using transformations on top of the quantile reward. Each transformation reshapes the reward distribution and affects the properties of the optimal policy, while having

Is QRPO still subject to the "chosen probabilities decreasing" problem?. Our understanding of the KL-regularized closed-form solution gives insights into the "DPO chosen probabilities decreasing" problem! 🤔. For QRPO, this is not a mystery anymore; we know exactly where the

💬 The reward model we use has been trained to be robust to length bias, and we see that this is preserved in QRPO and REBEL, which use rewards. But when compressed to preferences for DPO and SimPO, it leads to the typical length bias trend, despite the reduction in mean length.

🥇 QRPO achieves top performance in chat and coding compared to DPO, REBEL, and SimPO, each capturing a different way to learn from the reward signal (preference, reward difference, length normalization).

Obviously, nothing comes for free, but we give you a great deal! 🤝. * QRPO does not need many reference rewards to estimate quantiles: for high-quality offline datasets, 1-3 are enough!. * And you can scale this number for off-policy data generated from the reference model! 📈

We tackle the infamous “. partition function is known to be intractable. ” problem 🧐. This is the problem that limits DPO-like methods to pairwise data. We solve it thanks to 3 insights! 💡. 1️⃣ The “infinite sum over all possible LLM generations” argument is a myth. We

RT @XiuyingWei966: Curious about making Transformers faster on long sequences without compromising accuracy? ⚡️🧠 Meet RAT—an intermediate d….

RT @abeirami: As we go through a lot of excitement about RL recently with lots of cool work/results, here is a reminder that RL with a reve….

RT @manuelmlmadeira: Unfortunately I’m not attending @iclr_conf , but excited to share our workshop papers!. 📡 Graph Discrete Diffusion: A….

RT @AnjaSurina: Excited to share our latest work on EvoTune, a novel method integrating LLM-guided evolutionary search and reinforcement le….

@TalkRLPodcast

A dream come true! I presented "No Representation, No Trust" on my favorite RL podcast, @TalkRLPodcast !.Make sure to check it out to learn why training with PPO for too long makes your agent collapse!.

RT @jdeschena: Just taking advantage of the wave of excitement around diffusion LLMs to announce that our acceleration method for diffusion….