Our latest work performs sim2real dexterous grasping using end-to-end depth RL.

We open-sourced the full pipeline! Data conversion from MimicKit, training recipe, pretrained checkpoint, and deployment instructions. Train your own spin kick with mjlab: https://t.co/KvNQn0Edzr

super excited that we won the best student paper for videomimic! unfortunately i was packing in the hotel during the awards ceremony 😅

Our most recent work on multi-fingered manipulation was built on the back of these features including tiled rendering which enabled direct end-to-end pixels to action based sim-to-real transfer. https://t.co/GJXVmmBGpr

I'm super excited to announce mjlab today! mjlab = Isaac Lab's APIs + best-in-class MuJoCo physics + massively parallel GPU acceleration Built directly on MuJoCo Warp with the abstractions you love.

Very happy to share our new Isaac Lab white-paper that @yukez just announced at CoRL! https://t.co/D5QbzXRVct This is the product of a tremendous amount of work from many teams at NVIDIA plus many folks from RAI and ETH. Congrats to the whole team!

cool work. recommend digging into details -- they show how throughput can be increased by changing the arrangement of envs and RL training across a node. this will become especially important as policy size increases with pre-trained models..

This project wouldn't have been possible without my collaborators, Karl Van Wyk, @JitendraMalikCV, @pabbeel, @robot_trainer, @ankurhandos. For more information and videos, check out our website:

In the real world, we also find that the distilled E2E teacher policies lead to the best performance.

We train end-to-end policies with depth and then distill them into RGB. Compared to state-based teachers, we find our depth policies lead to better students.

We find batch size is a critical bottleneck for the vision-based policies. Training with our disaggregated setup reliably produces better policies.

Our method is able to double the number of environments on the exact same hardware (4xL40S).

We propose to train our policies with a new method that disaggregates simulation and RL training on separate GPUs. In this way, we have several GPUs solely running the environment simulation, whereas policy inference, PPO, and experience buffers are hosted on another GPU.

This is done mainly because of the difficulty to scale up the batch size for vision RL. Most traditional simulators scale up to multiple GPUs using data parallelism. While serviceable for state-based training, it is not an efficient use of GPU memory for vision RL.

Many prior works factorize vision policies into a privileged teacher training phase and then distill into a vision student.

Should robots have eyeballs? Human eyes move constantly and use variable resolution to actively gather visual details. In EyeRobot ( https://t.co/iSL7ZLZcHu) we train a robot eyeball entirely with RL: eye movements emerge from experience driven by task-driven rewards.

Check out our full code release!!! And I am also glad to see that VideoMimic is in the best paper finalist at CoRL 2025. Seoul!! https://t.co/jKGM3lzprY

excited that VideoMimic was just nominated for an award at CoRL, see you soon in Seoul! 🇰🇷 we have also now also released our sim/real-robot deployment pipeline, including checkpoints and nice viser web viz -- check it out https://t.co/HhqnbE0F0S

Awesome research and I'm glad it's now available to the community. Sim to real learning is a tool with a ton of potential, but it's so hard to get started building the right environments and task setup!

Happy to announce that we have finally open sourced the code for DextrAH-RGB along with Geometric Fabrics: https://t.co/v7QPGgtyDi https://t.co/fHyvvKU9IA