🚨 Excited to share our new work: "Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning"! 📈. We propose gradient interventions that enable stable, scalable learning, achieving significant performance gains across agents and environments!. Details below 👇

RT @tanghyyy: (1/8)🔥Excited to share that our paper “Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn” has….

RT @MartinKlissarov: As AI agents face increasingly long and complex tasks, decomposing them into subtasks becomes increasingly appealing.….

RT @akhil_bagaria: New paper: skill discovery is a hallmark of intelligence--identify interesting questions about the world, and learn how….

RT @pcastr: proud to share a survey of state representation learning in RL that my student ayoub echchahed and i prepared, that was just pu….

RT @GlenBerseth: Being unable to scale #DeepRL to solve diverse, complex tasks with large distribution changes has been holding back the #R….

RT @pcastr: really excited about this new work we just put out, led by my students @creus_roger & @johanobandoc , where we examine the chal….

This work was a fantastic collaboration with @johanobandoc,@luli_airl, @pierrelux, @GlenBerseth, @AaronCourville, @pcastr 🙌. 📄 Paper: 💻 Code:

We also combine our proposed interventions with recent scaling strategies like Simba , and still see additional improvements in performance and stability 📊

🔁 Back to our artificial non-stationary supervised setting:. With both interventions, networks now show near-perfect plasticity. They can continually fit changing data across network scales. No collapse 💪

Combining both ideas, we see massive gains on the ALE (Atari)! 🎮🔥. We benchmark against PQN and PPO, and the improvements are remarkable, both in terms of performance and scalability.

💡 Our second intervention: better optimizers. Second-order estimators capture loss curvature, providing more stable updates than first-order methods. Our ablations show the Kron optimizer shines in deep RL, helping agents adapt as learning evolves ✨

💡 Our first intervention: better architectures for stable gradients. Residual connections create shortcuts that preserve gradients and avoid vanishing ⚡️. We extend this with MultiSkip: broadcasting features to all fully connected layers, ensuring direct gradient flow at scale

So what's going on?.Gradients are at the heart of this instability. We relate vanishing gradients to classic RL diagnostics:.⚫ Dead neurons.⚫ Representation collapse.⚫ Loss landscape instability (from bootstrapping)

We test the same architectures in supervised learning: gradients propagate well and performance scales ✅. But in RL, performance collapses due to non-stationarity!. We simulate this by shuffling labels periodically. The result? Gradients degrade, and bigger networks collapse! 🚫

Scaling up networks in deep RL is tricky: do it naively, and performance collapses 😵‍💫. Why? Increasing depth and width leads to severe vanishing gradients, causing unstable learning ⚠️. We diagnose this issue across several algorithms: DQN, Rainbow, PQN, PPO, SAC, and DDPG.

RT @harshit_sikchi: Behavioral Foundation Models (BFMs) trained with RL are secretly more powerful than we think. BFM’s directly output a p….

RT @pcastr: thrilled that we'll be presenting this paper as a spotlight at #ICML2025 . come by our poster in vancouver to chat with us abou….

RT @wang_jianren: (1/n) Since its publication in 2017, PPO has essentially become synonymous with RL. Today, we are excited to provide you….

RT @gan_chuang: 🤖Can world models quickly adapt to new environments with just a few interactions? . Introducing AdaWorld 🌍 — a new approac….