How to build vision-language-action models that train fast, run fast & generalize? In our new paper, we formalize & analyze the approach of our π-0.5 model & further improve it with a single stage recipe. Blog: https://t.co/IihKEmmxSB Paper: https://t.co/JfEU7pcoZk

More info about knowledge insulation here: https://t.co/h0DiPRRLge Code + checkpoints:

We open-sourced pi-05 today. All checkpoints that we release have been trained with Knowledge Insulation

TRI's latest Large Behavior Model (LBM) paper landed on arxiv last night! Check out our project website: https://t.co/n0qmDRivRH One of our main goals for this paper was to put out a very careful and thorough study on the topic to help people understand the state of the

Had a blast on the Unsupervised Learning Podcast with @hausman_k! We covered the past, present, and future of robot learning 🤖 Big thanks to @jacobeffron for being a fantastic host!

It was a really fun project with the amazing team @physical_int including @brian_ichter, Jost Tobias Springenberg, @liliyu_lili, Adrian Li-Bell, @KarlPertsch, @allenzren, @HomerWalke, @QuanVng, @lucy_x_shi, @slevine

Our paper includes many ablations & details about various modeling choices. Check it out :)

One might think that another way of “knowledge insulation” would be to just freeze the backbone. This, however, does not work as shown below, indicating that a base VLM does not contain sufficient representations for robot motions.

We call this procedure “knowledge insulation” and the resulting model π-0.5 + KI. Here are some videos of the model controlling a mobile manipulator in unseen homes.

The model follows language instructions much better, has high performance, and fast inference speed. We also train the model on web-data at the same time, which further increases generalization.

It turns out that this is the best of both worlds: The model learns really fast (7.5 times faster than π-0, as fast as π-0-FAST), but with the advantages of the action expert of π-0 at inference time.

Our insight is to stop the gradient from the action expert and instead train the VLM backbone with discretized FAST actions to learn representations. This way, we “insulate” the knowledge of the pre-trained VLM, but still ensure to adapt the backbone to robotics.

While flow-matching action experts are great for inference since they produce continuous actions and have fast inference, their gradients are a bad signal for training. Consequently, the model trains slowly and struggles with following language instructions.

Check out our new work where we dissect various aspects of chain-of-thought at both training and inference time) for robotics! Awesome work led by @verityw_

We auto-encode point tracks to automatically evaluate motion realism in generative video models. By inherently focusing on motion, our new metric (TRAJAN) correlates much better with human judgments of these models than appearance based metrics.

Scaling data diversity, transfer between data sources, and a good training recipe were the main ingredients to allow robots to generalize to new homes!

More insights: π-0.5 is trained to break tasks down into subtasks, before producing actual robot actions. It turns out that adding the subtask prediction data is useful, even if you query the model with the overall task directly.

In particular, the diverse robot in-the-wild helps a lot, even though this data is from static robots, and we evaluate on mobile manipulator tasks. Check out more in the blog post https://t.co/OismdeBWlr and paper

To achieve this, we enabled π-0.5 to be trained on many different data sources, from multi-environment mobile manipulator data, to static robot data in-the-wild, in the lab, and more classical vision-language data. Those additional data sources help the model significantly!

I think this is a very exciting result. Generally, all (except for this green line above) evaluations in our paper are done in unseen environments!