So you’ve trained your favorite diffusion/flow based policy, but it’s just not good enough 0-shot. Worry not, in our new work DSRL - we show how to *steer* pre-trained diffusion policies with off-policy RL, improving behavior efficiently enough for direct training in the real

RT @natashajaques: In our latest paper, we discovered a surprising result: training LLMs with self-play reinforcement learning on zero-sum….

Ok what are the takeaways?. 1. Wrap a base diffusion policy into the environment and just control initial noise to steer it. Yields a frustratingly simple, scalable and efficient algorithm that retains good exploration while enabling sample efficient improvement. 2. Train two Q.

Thanks to noise-aliasing - DSRL is also a competitive offline RL procedure: first training a diffusion/flow policy on an entire offline dataset, then applying DSRL to steer it to high-reward behavior using the offline data performs on par with state-of-the-art offline RL methods

What are the numbers? In simulation, we find that DSRL substantially outperforms all existing approaches to finetuning diffusion policies online on benchmarks such as Robomimic and Mujoco tasks. (8/10)

What’s the catch? We’ve changed the action space of the RL policy from the original actions a to initial noise z. The challenge is that offline data or interventions are collected in the (s, a, s’, r) space, not in the (s, z, s’, r) space. To allow this data to still be useful,

Why does this actually help? Since any "action" played by the noise-space RL policy is just initial noise for the denoising process, even early in training the denoised actions look like actions from a BC-trained policy, rather than an unconverged RL policy. This makes

Can we understand what’s going on a little more visually? Watching a timelapse of DSRL on a WidowX robot tasked with picking and placing an object shows the coherence of exploration and the efficiency of learning! (5/10)

Ok but can we steer policies that we didn’t actually pre-train ourselves? To test this, we applied DSRL to pi0, a state-of-the-art flow-based generalist policy from @physical_int. DSRL is able to improve pi0 in real-world deployment, on some tasks taking success from 25% to 90%

Why does this matter? - 1) retains base policy exploration, 2) enables super efficient real-world RL, efficient enough to put on a robot! Let’s look at some examples of efficient improvement of real-world diffusion policies for robotic control. We applied DSRL on several

What’s the key idea? Don’t touch the base policy, but instead train a lightweight policy via RL to select the initial noise in a diffusion (or flow) policy's denoising process. This initial noise modulates the behaviors, guiding it towards desired behaviors. Think of it as -

RT @ajwagenmaker: Diffusion policies have demonstrated impressive performance in robot control, yet are difficult to improve online when 0-….

I'm sadly unable to be at #RSS2025 this year, but my students @prodarhan, @chuning_zhu and @marceltornev will be! Find them presenting some exciting work today, 6/21: . 1) @chuning_zhu will present Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large

RT @pranav_atreya: In robotics benchmarks are rarely shared. New eval setups are created for each new project, a stark difference from eval….

Check out some of our new work on distributed robot evaluation led by @KarlPertsch, @pranav_atreya and @tonyh_lee! Hopefully folks can contribute, and help us take a step towards systematic and standardized empiricism in robot learning! :). Also check out some of the fun sim eval.

RT @yunchuzh: How should a robot perceive the world? What kind of visual representation leads to robust visuomotor policy learning for robo….

Check out @yunchuzh's new work on automatically selecting keypoints as a representation for super robust policy learning!.

Go read our paper to get into the guts of things :). Paper: Website: Here’s some higher level points, I learned from this paper. 1. Keypoints are surprisingly robust. Future policy representations may perhaps look a little.

Here’s a cool bonus - turns out keypoint trackers are pretty good at bridging large visual gaps. So the cool thing is that visuomotor policies trained in simulation, can transfer over to the real world; even when the simulation looks pretty bad! This is impactful because it

But we care about policies! Turns out, using the same imitation learning setup as standard behavior cloning, but just changing the input visual representation to the keypoints selected by ATK yields policies that are robust to significant visual variations—lighting, background,