Online interaction is probably a defining property of RL. But with the rise of offline algo, it is not clear if the “online” bit of RL is necessary for RLHF. We hypothesis test the causes of the perf gap between online and offline alignment. Details in🧵

Taking the k3 estimate as an example (from John's popular blogpost . Contrary to popular practice, differentiating the estimate as a loss ends up enforcing the reverse-KL, but only incidentally. See more details:

Maybe to one's surprise, taking KL estimates as `kl_loss` to minimize does *not* enforce the KL. This implementation, however, is quite common in open source RL repos and recent research papers. In short: grad of an unbiased KL estimate is not an unbiased estimate of KL grad.

RT @MistralAI: Announcing Magistral, our first reasoning model designed to excel in domain-specific, transparent, and multilingual reasonin….

It was refreshing to see the impact that small algorithmic changes have on the system performance. While the “double-sided” PPO/GRPO clipping is dominant in the literature, we argue that a single-sided clipping akin to IMPALA fits the design of distributed training more.

Introducing LlamaRL, a distributed RL framework for training LLM at scale. LlamaRL is highly modular, Pytorch-native, customizes optimization of actors/learners to max out throughput, and adjusts for systemic off-policyness to stabilize training.

RT @ZacKenton1: Eventually, humans will need to supervise superhuman AI - but how? Can we study it now?. We don't have superhuman AI, but w….

Thanks @_akhaliq for promoting our work!. Unlike regular RL where golden r(s,a) are available and online is generally deemed better than offline, in RLHF this is less clear. Complementary to some concurrent work, we investigate causes to the perf gap between online vs. offline.

The findings ought to be taken with a grain of salt due to limitations in our experimental setups. But hopefully this investigation contributes to a better understanding of RLHF practices. Finally, very grateful to my collaborators @GoogleDeepMind on this fun project!.

Some takeaways:.- There is something more to online than wider coverage of response generation.- Offline training improves policy is a much more implicit way than online (discriminative vs. generative abilities).- The gap persists across wider variants of algos and network sizes.

Hypothesis 5: Scaling policy network size is all you need! So that offline benefits more from scaling. Unfortunately, scaling policy networks alone does not seem to suffice, as the online vs. offline gap can actually increase as we scale up.

Hypothesis 4: Maybe much of the gap is due to interactions between offline and discriminative loss, rather than offline itself. We evaluate the same setting with best-of-2, which uses a SFT-like rather than discriminative loss. Online vs. offline gap still persists in some cases

Hypothesis 3: offline training learns a RM. So maybe improving offline policy’s accuracy as a RM would improve performance. We found there is little positive correlation between generative and discriminative abilities of policies both across and within different algos.

Hypothesis 2: offline dataset is of low quality. Relating offline algo to SFT, maybe offline is more susceptible to dataset quality. We generated “high-quality” data from a highly competent policy and ran offline algo over. Surprisingly, offline did not learn much in this case.

Hypothesis 1: online has wider response coverage than offline. Online sees more diverse responses than offline, as the latter uses a static dataset. To verify this, we saved the online (prompt, responses) as a dataset and ran offline algo over it. This did not explain the gap.

To study causes of the gap, we need to establish one. We compare online vs. offline IPO, so that they only differ in sampling distribution. In what sense is online better than offline? Online achieves better perf vs. KL trade-off than offline. Both are subject to Goodhart’s law.

RT @misovalko: Fast-forward ⏩ alignment research from @GoogleDeepMind ! Our latest results enhance alignment outcomes in Large Language Mod….

Interested in how . **non-contrastive representation learning for RL**. is magically equivalent to. **gradient-based PCA/SVD on the transition matrix**. and hence won't collapse and capture spectral info about the transition? . Come talk to us at #ICML2023 Hall 1 #308 at 1:30pm.

RT @wwdabney: Even if all you want is a value function, using quantile TD (QTD) can give a better estimate than standard TD. Today at #ICM….

Interested in how non-contrastive representation learning works in RL? We show.(1) Why representations do not collapses.(2) How it relates to gradient PCA / SVD of transition matrix.Understanding Self-Predictive Learning for RL #ICML2023 @GoogleDeepMind