Scalable, reproducible, and reliable robotic evaluation remains an open challenge, especially in the age of generalist robot foundation models. Can *simulation* effectively predict *real-world* robot policy performance & behavior?. Presenting SIMPLER!👇.

RT @_crockwell: Ever wish YouTube had 3D labels?. 🚀Introducing🎥DynPose-100K🎥, an Internet-scale collection of diverse videos annotated with….

Generate videos in just a few seconds. Try Grok Imagine, free for a limited time.

RT @allenzren: HNY! Lately I took a crack at implementing the pi0 model from @physical_int. PaliGemma VLM (2.3B fine-tuned) + 0.3B "action….

RT @KuanFang: How can robots learn bimanual, contact-rich manipulation tasks when scaling up expert demonstrations is so challenging? With….

RT @Scobleizer: The robots will manipulate the Holodeck.

@KuanFang @zhu_xinghao @pangtao22.

Website, paper, and videos: This is an internship project at the Boston Dynamics AI Institute in 2024 summer. Thanks Tong Zhao, Tao Pang, @KuanFang, @zhu_xinghao, Jiuguang Wang for the great collaboration!.[7/n].

We achieve good policy generalization across different object dimensions, shapes, and physical properties (e.g., mass and friction), along with minimal sim-to-real drop in policy performance. Our policy also remains robust to external human perturbations. [6/n]

To deploy our policy in the real-world, we track the pose changes for any object by first using Grounding-Dino + EfficientViT-SAM to segment the target object in the first frame. We then select keypoints within the segmented mask and track them across subsequent frames. [5/n].

We then train a task-conditioned diffusion policy via behavior cloning using uncolored point clouds. We also introduce key design choices in feature extraction, action prediction, and data augmentation that significantly improve our policy's sim-to-real transfer ability. [4/n]

We generate large-scale synthetic demonstrations in random simulation environments by building on recent advances in efficient motion planning through contact. As the planner is stochastic, we also filter out suboptimal trajectories to ensure high demonstration quality. [3/n].

We propose Generalizable PLanning-GuIded Diffusion Policy LEarning (GLIDE) and adopt a sim-to-real approach. We specifically focus on the task of bimanually reorienting any object - including heavy or bulky ones - to any user-specified SE(2) target pose on the table. [2/n]

Learning bimanual, contact-rich robot manipulation policies that generalize over diverse objects has long been a challenge. Excited to share our work: Planning-Guided Diffusion Policy Learning for Generalizable Contact-Rich Bimanual Manipulation!. 🧵1/n

RT @haosu_twitr: Check out the full demo of Hillbot Alpha's latest progress! Our robot successfully stocks shelves over a **long horizon**….

SIMPLER will be presented at #CoRL2024 at 4pm on Nov 8 (Section 4)!. While I won't be in person due to visa constraints, @xiao_ted, @KarlPertsch, and @oier_mees will be presenting the paper and are happy to chat about it in person!.

RT @Stone_Tao: just made it possible to evaluate generalist robotics models like Octo at 60-100x real world evaluation speeds via gpu simul….

RT @FangYunhao_X: Excited to announce our new paper VILA^2, an awesome collaboration with @LigengZhu and my amazing mentors @Yao__Lu @yin_h….

RT @oier_mees: Monday I will also help present "Evaluating Real-World Robot Manipulation Policies in Simulation" with @KarlPertsch and @xi….

RT @xiao_ted: Action-packed day of robotics + AI at #RSS2024 tomorrow on Monday, July 15!. Hope to see you:.- I'll be giving a talk at the….

RT @yuchen010807: While the Segment Anything Model (SAM) greatly improves 2D segmentation annotation efficiency, is there a foundation mode….

RT @brandontrabucco: With Meta's recent work on multi-token prediction, it's clear that left-to-right #LLMs have flaws, but can we do bette….