All videos are autonomous. We also tested training "from scratch" (from a VLM initialization), but this failed on all tasks, indicating that fine-tuning our models is essential for success. For more, check out our blog post:

Event 5 🥇: the gold medal task is to wash a frying pan in the sink using soap and water. Both sides. We also tackled silver (cleaning the fingers) and bronze (wiping the counter), in our blog post. The pan is hard to clean, but the robot rose to the challenge.

And here is our attempt at the gold. The Olympics requires using the fingers, but we had to use a sharper tool. That's a disqualification for us, but the result is still really interesting.

Event 4 🥈: we tried both gold and silver: peeling an orange and using a dog poop bag. For gold, we had to "bend the rules" and use a tool, so we believe we only take silver. The dog bag is really hard, not least because it blinds the wrist camera when in use:

And the bronze: using windex to clean a window, in this case one of our phone booths. Just don't get stuck inside while it's cleaning!

We also did the silver medal task: making a peanut butter sandwich. Very long horizon (open jar, spread butter, cut the sandwich into elegant triangles, and close the jar), lots of force and deformables.

Event 3 🥇: this was our favorite! tool use, we did all three tasks. The gold medal task was to use a key to unlock a lock. Very fine manipulation, fitting the key in the lock and turning it with enough force to unlock.

Event 2🥈: the gold-medal task is to hang an inside-out dress shirt, after turning it right-side-in, which our robot can't do physically (the gripper is too wide to fit inside the sleeves). So we tackled the silver-medal task, which is to turn a sock inside-out.

Event 1🥇: we fine-tune π0.6 for the "gold medal" task, going through a self-closing lever-handle door. This is hard because the robot has to keep the door open as it goes through it.

Benjie Holson proposed "Robot Olympics" - 5 "events" with gold/silver/bronze medal tasks like washing a pan: https://t.co/6pdRrNt7GH These are not tasks we made up ourselves. They illustrate "Moravec's paradox" - everyday tasks we find easy that current robots just can't do.

We got our robots to wash pans, clean windows, make peanut butter sandwiches, and more! Fine-tuning our latest model enables all of these tasks, and this has interesting implications for robotics, Moravec's paradox, and the future of large models in embodied AI. More below!

Check out the blog post and full research paper for more details and experiments, including studies into high level vs low level transfer, comparisons to robot data, and quantifying the utility of wrist cameras. https://t.co/hLZdccTR6m

This also shows up in the representations learned by the model. We plot the model’s representations of human and robot images. As pre-training is scaled up, the representation of humans and robots become more aligned: to a scaled-up model, human videos "look" like robot demos.

We were surprised, and wanted to understand why. What about π0.5 enabled emergent human-robot transfer? We ran an experiment to test if it only appears above a certain scale. Turns out human transfer scales with the amount & diversity of robot data in VLA pre-training!

If we use our full pre-trained pi05 model, simply finetuning with human video data can double the performance on tasks that are depicted in the human videos!

We set out with the goal of understanding what it would take to make human data useful for VLAs like π0.5. We record egocentric human data with wearable cameras, and then include it in a co-training recipe with hand poses serving as actions.

We discovered an emergent property of VLAs like π0/π0.5/π0.6: as we scale up pre-training, the model learns to align human videos and robot data! This gives us a simple way to leverage human videos. Once π0.5 knows how to control robots, it can naturally learn from human video.

To learn more, see more videos, and read a full research paper about Recap and π*0.6, see our blog post here:

Quantitatively, training π*0.6 with RL can more than double throughput (number of successful task executions per hour) on the hardest tasks and cut the number of failures by as much as a factor of two.

We also trained π*0.6 to assemble boxes. Here is an hour of box building, with about two and a half minutes per box.