Chelsea Finn
@chelseabfinn
Followers
80K
Following
1K
Media
255
Statuses
606
Asst Prof of CS & EE @Stanford Co-founder of Physical Intelligence @physical_int PhD from @Berkeley_EECS, EECS BS from @MIT
Palo Alto, CA
Joined June 2014
The robot can autonomously perform a real gallbladder removal subroutine!. - Successfully completed the procedure on all 8 of 8 held-out gallbladders.- Uses same alg that we used to train robots to make trail mix, using language hierarchy. Paper + videos:
Introducing Hierarchical Surgical Robot Transformer (SRT-H), a language-guided policy for autonomous surgery🤖🏥. On the da Vinci robot, we perform a real surgical procedure on animal tissue. Collaboration b/w @JohnsHopkins & @Stanford
4
20
162
We still lack a scalable recipe for RL post-training seeded with demonstration data. Many methods add an imitation loss, but this constrains learning too much. We propose to use the demos only to perturb exploration -- It works really well!. Paper:
RL often struggles with poor sample efficiency, even with expert data. How can we address this?. One approach is to incorporate an imitation loss, but that can overconstrain the policy. We propose leveraging prior data implicitly to guide more effective exploration. (1/5).
3
37
296
How can robots problem solve in novel environments?. We combine high-level reasoning with VLMs with low-level controllers to allow test-time problem solving. Paper & code:
How can robots autonomously handle ambiguous situations that require commonsense reasoning?. *VLM-PC* provides adaptive high-level planning, so robots can get unstuck by exploring multiple strategies. Paper:
7
19
148
How do we make a scalable RL recipe for robots?. We study batch online RL w/ demos. Key findings:.- iterative filtered imitation is insufficient.- need diverse policy data, eg using diffusion policy.- policy extraction can hinder data diversity. Paper:
Robotic models are advancing rapidly—but how do we scale their improvement? 🤖. We propose a recipe for batch online RL (train offline with online rollouts) that enables policies to self-improve without complications of online RL. More: (1/8)
3
26
172
Most robot policies don't have any memory!. This is because:.- policies often perform *worse* with past observations as input.- GPU memory and compute constraints. We address both to train long-context robot diffusion policies. 🤖. Paper & code:
Giving history to our robot policies is crucial to solve a variety of daily tasks. However, diffusion policies get worse when adding history. 🤖. In our recent work we learn how adding an auxiliary loss that we name Past-Token Prediction (PTP) together with cached embeddings
4
25
235
RT @amberxie_: Introducing ✨Latent Diffusion Planning✨ (LDP)! We explore how to use expert, suboptimal, & action-free data. To do so, we le….
0
44
0
RT @Anikait_Singh_: I’m in Singapore for #ICLR2025!. Excited to present Improving Test-Time Search for LLMs with Backtracking Against In-Co….
0
9
0
I'm giving two talks today/Sunday at #ICLR2025 !. - Post-Training Robot Foundation Models (Robot Learning Workshop @ 12:50 pm). - Robot Foundation Models with Open-Ended Generalization (Foundation Models in the Wild @ 2:30 pm). Will cover π-0, Demo-SCORE, Hi Robot, & π-0.5.
8
10
246
RT @SurajNair_1: Since the first year of my PhD, every talk I’ve given has opened with a slide about the distant north star: dropping a rob….
0
5
0
RT @smithlaura1028: My goal throughout my PhD has been to take robots out of the lab and into the real world. It was so special to be a par….
0
56
0
The model is far from perfect. See some of the failures on the blog post: That just means many open challenges & more work to do!. This was a large, company-wide effort spanning hardware, software, research, and operators! Very proud of the team.❤️
1
1
38
We also had some fun messing with the robot, to test how it reacts. :). (again, all environments/objects/furniture not seen in training!)
1
2
21
We brought robots to 3 new rental homes + tested further in new mock kitchens & bedrooms (everything not seen in training!). Here are some of the results:
1
0
10
Third: the diversity of the mobile manipulation data. Diverse data from more than 100 locations is crucial for good performance in new environments. With our pre-training recipe, performance of our model matches that of a model fine-tuned on data from the target environment!
1
0
11
Second: the data mixture. Including data from static robot arms in pre-training helps unseen env performance on mobile manipulators by a large margin. This is true for both multi-task cross-embodiment data in labs (CE) and diverse multi-env data (ME).
1
0
10
This robot data alone is not enough! A couple key improvements. First: the model training recipe. Compared to the π-0 recipe (flow matching):.We find far better performance when pre-training with next-token prediction (i.e. π-0 fast recipe) and fine-tuning with flow-matching.
2
3
26
We started with data: can we collect robot data in many real-world environments?. We did this with both static robot arms and mobile manipulators. It wasn’t easy. Huge effort spanning operations, hardware, software, and research.
1
0
15
For the π-0 release, we focused on capability: can robots complete complex tasks like folding laundry?. From there, we felt that generalization to any new environment was the next most important challenge. This entails new.- furniture.- objects.- lighting.- countertops.and so on.
1
0
16
Introducing π-0.5!. The model works out of the box in completely new environments. Here the robot cleans new kitchens & bedrooms. 🤖. Detailed paper + videos in more than 10 unseen rooms:. A short thread 🧵
15
98
704
Key idea: When training on all data, policy success is indicative of whether the strategy it took is good!. Paper: Led by @_anniechen_ and @AlecLessing, with @liu_yuejiang @StanfordAILab.
0
0
12