Read the full paper here: Website:

TLDR: we found a scalable way to align the action spaces of different end-effectors. Cross-embodied diffusion policies trained with our method using latent actions enable skill-transfer and multi-robot control 🦾.

Co-training diffusion policies enables cross-embodiment skill transfer and improves spatial generalization: we demonstrate this in experiments using a Faive hand, a mimic hand and a Franka gripper. Despite the embodiment gap, the policies still exhibit skill transfer!.

This concludes the first step: now we have encoders that encode embodiment-specific end-effector poses into aligned latents, and embodiment-specific decoders that can do both self- and cross-reconstruction from these latents! Now, on to policy learning:.

To decode from the latent space, we train decoders with a simple reconstruction loss. Furthermore, the encoders are also finetuned to improve the reconstruction from the latent space.

Then, we train lightweight encoders on the paired action data using a pairwise contrastive loss, ensuring that the alignment of the end-effector poses is upheld in the latent space.

First, we learn a shared latent action space via contrastive learning. To generate paired data across end-effectors, we use embodiment-specific retargeting methods.

How can robots with different end-effectors efficiently learn from each other?. In our latest work, we propose a new approach for learning imitation learning policies across end-effectors that enables efficient cross-embodiment skill transfer through two steps:. 🧵

RT @elvisnavah: It is my opinion that if you want to learn from human demos, your hardware stack should have human-level dexterity capabili….

RT @elvisnavah: Happy to announce mimic-one: a Scalable Model Recipe for General Purpose Robot Dexterity, the culmination of years of resea….