That’s all for now! This project was a long time coming, be sure to check out the full blog post and paper here:

Finally, there’s a subtle issue with non-real-time inference that’s easy to overlook: distribution shift. Pauses for inference are not in the training data! We found that RTC was not only faster, but also more precise and consistent than our old synchronous strategy.

To prepare for this future, we added up to +200ms of artificial latency to π0.5 (>300ms total), and the speed and performance of RTC were totally unaffected!

Model size is not the only contributor to latency. Personally, I’m betting that the VLAs that solve physical intelligence will not be able to fit in onboard robot computers. That means we will need centralized inference servers, and we will have network latency.

Importantly, this requires no training-time changes! It’s applicable to any diffusion- or flow-based policy at inference time. With RTC, we get smooth real-time execution.

Our solution, real-time chunking (RTC), combines action chunking with inpainting — the actions within the inference delay are frozen, while the rest are “inpainted” in a way that’s consistent with the previous plan.

For smooth execution, we need to always produce the next action as soon as it’s needed. This is called a “real-time constraint”. With high-latency models, this requires concurrency: generating new actions while executing old ones. But naive concurrency does not work.

In LLM land, a slow model is annoying. In robotics, a slow model can be disastrous! Visible pauses at best, dangerously jerky motions at worst. But large VLAs are slow by nature. What can we do about this? An in-depth 🧵:

I mean, technically the model is optimized. by the XLA compiler, not by a human!. from

This caption is a bit funny to me because we've put precisely zero effort into optimizing our model implementation. Thanks JAX!

RT @physical_int: We got a robot to clean up homes that were never seen in its training data! Our new model, π-0.5, aims to tackle open-wor….

RT @physical_int: We are excited to share new experiments with AgiBot @AgiBot_zhiyuan on multi-task, multi-embodiment VLAs! With one model….

RT @physical_int: Many of you asked for code & weights for π₀, we are happy to announce that we are releasing π₀ and pre-trained checkpoin….

RT @physical_int: @Astribot_Inc 🤝 Physical Intelligence (π)

My favorite slide that I made for my talk last weekend -- a very silly thought experiment in which we compare language datasets to robotics datasets (in the most shallow way possible). Yes it is to scale; I learned that the maximum shape size in Keynote is 20,000pts

Here's a link to the recording for anyone that's interested!.

If you're at #CoRL2024, come check out my talk at the X-Embodiment workshop at 1:30pm! Thanks to @KarlPertsch for inviting me to speak!

Overall, working at @physical_int has been a blast and joining was definitely the right decision. I can't believe it's only been 6 months and I can't wait for what comes next!.

For super cool uncut videos of evaluations and all that other good stuff, check out our blog post:.

Here's a little secret: π₀-small, which also uses flow matching but not a VLM backbone, was our "main model" for 4+ months and was outperforming many strong baselines! IMO the most exciting benefit of adding the VLM init was drastically improved language following