RT @kvablack: In LLM land, a slow model is annoying. In robotics, a slow model can be disastrous! Visible pauses at best, dangerously jerky….

To learn more about RTC, check our blog post, as well as the full-length research paper that we prepared with detailed results:

Introducing Grok Imagine.

Quantitatively, RTC enables high success rates even as we introduce artificially high inference delays. Even if we add 200ms of additional delays on top of what the model needs, RTC performance remains stable, while naive methods drop off sharply.

RTC not only makes motions smoother, but it actually allows the robot to move with more precision and speed, improving the performance of our models without any training-time changes at all!

The main idea in RTC is to treat generating a new action chunk as an “inpainting” problem: the actions that will be executed while the robot “thinks” are treated as fixed, while new actions are inferred via flow matching.

Our models need to run in real time on real robots, but inference with big VLAs takes a long time. We developed Real-Time Action Chunking (RTC) to enable real-time inference with flow matching for the π0 and π0.5 VLAs! More in the thread👇

RT @GoddenThomas: Quick life update; I've joined the hardware team at @physical_int and I'm moving to San Francisco!.

To learn more, check out our blog post here:

At runtime, π-0.5 with knowledge insulation runs much faster than π-0-FAST, giving us the best of all worlds: fast training, fast robot control, and better generalization.

This works really well: π-0.5 with knowledge insulation trains just as fast as π-0-FAST, 5x-7x faster than π-0, and still retains excellent performance, dexterity, and compute-efficient inference at runtime, outperforming our prior models.

Our knowledge insulation method uses discrete tokens to train the VLM backbone, so that it learns appropriate representations, even as the action expert trains to take those representations and turn them into continuous actions.

The trouble is that in practice, the action expert is cold started, and in the beginning of VLA training the gradients from the action expert corrupt the VLM backbone, causing it to forget web-scale pretraining and slow down training.

The VLM consists of a language model (LLM) and a vision encoder. The vision encoder serves as a kind of “visual cortex”, and the LLM as a kind of “prefrontal cortex”. To turn it into a VLA, we add a “motor cortex”: an action expert that can output continuous action chunks.

We figured out how to train VLAs with diffusion outputs much faster (7.5x faster), inheriting better language following from the VLM, and leading to better results. The key: protect the VLM backbone during training with knowledge insulation. Let’s talk about what we learned👇

To learn more about π-0.5 (pronounced “pi oh five”), check out our blog post here: A research paper about π-0.5 describing the model, training, and experiments is here:

97.6% of the π-0.5 data does not come from the mobile robots we use in our experiments. Most comes from other robots: cross-embodiment data in the lab, non-mobile (static) robots in the wild. It also includes web data. Each piece of data is important for good results.

π-0.5 performs hierarchical inference, inferring high-level semantic subtasks ("pick up the plate") followed by actions. It uses a co-training recipe with data from other robots, high-level commands, verbal instructions, and multimodal data from the web.

We got a robot to clean up homes that were never seen in its training data! Our new model, π-0.5, aims to tackle open-world generalization. We took our robot into homes that were not in the training data and asked it to clean kitchens and bedrooms. More below⤵️

At Pi, we seek to develop generalizable and broadly capable physical intelligence. With models that can control more robots and perform more skills, generalist cross-embodiment policies will continue to become more and more useful, practical and powerful.

Receive menu from customer