Super excited to shall our recent work! We did not cherry-pick for this "cherry picking with RL" paper!😂 Huge thanks to all my collaborators, and especially @xkelym🧡!.

RT @YiruHelenWang: 🚨Tired of binary pass/fail metrics that miss the bigger picture?. 🤖Introducing #RoboEval — an open benchmark that shows….

RT @abhishekunique7: So you’ve trained your favorite diffusion/flow based policy, but it’s just not good enough 0-shot. Worry not, in our n….

How can we continuously improve large pretrained behavior policies when 0-shot performance is not good enough? Directly finetuning the base policy via RL tends to be sample-inefficient. Can we squeeze more juice from the base policy to enable automatic and efficient performance.

Huge thanks to @shubham_kernel, Zhengyu Zhang, @xkelym, @siddhss5, @abhishekunique7, and everyone who helped along the way! 🙏.📄 Paper:🌐 Website:

Read the paper to see what makes it tick—lots of subtle design choices and insights packed in. 💡 Takeaways:. 1️⃣ Use robust yet flexible visual representations from RGB images. Keypoints are one such representation. 2️⃣ Automatically select keypoint representations that are.

We show that ATK-selected keypoints provide an effective visual representation for bridging substantial visual gaps, for instance enabling transfer of visuomotor policies from simulation to reality. As a bonus, these keypoints remain robust to variations in object positions,

Using the same imitation learning setup as standard behavior cloning, policies trained on ATK-selected keypoints remain robust to significant visual variations—lighting, background, clutter. (6/8)

We show that ATK enables automatic task-relevant feature selection. The chosen keypoints correspond to semantically meaningful elements of the task. For example, it focuses on the towel for folding, the blanket for hanging, the top of items for grasping, such as sushi or grapes,

💡 Key insight: The desired set of keypoints is the minimal set of points that can express the optimal policy. ATK automatically selects these via a task-driven masking mechanism, using just the gradient signal from supervised learning on the task. Notably, this requires no

To mitigate this fragility, we decided to use 2D keypoints—consistent image-space features—as compact and robust state representations. Why?.1️⃣ Keypoint tracking is robust to occlusion & distractions, thanks to strong pretraining on diverse Internet-scale video data.2️⃣ Keypoints

Learning from raw RGB images often leads to fragile policies that break under slight visual changes—backgrounds, lighting, distractors. (2/8)

How should a robot perceive the world? What kind of visual representation leads to robust visuomotor policy learning for robotics?. Policies trained on raw images are often fragile—easily broken by lighting, clutter, or object variations—making it challenging to deploy policies

RT @memmelma: Have some offline data lying around? Use it to robustify few-shot imitation learning! 🤖. STRAP 🎒 is a retrieval-based method….

Can't wait to see how this incredible tool changes the game for robotics and AI applications. 🚀.

RT @chuning_zhu: How can we train RL agents that transfer to any reward? In our @NeurIPSConf paper DiSPO, we propose to learn the distribut….

RT @Xiaofeng2Guo: We introduce 𝐅𝐥𝐲𝐢𝐧𝐠 𝐂𝐚𝐥𝐥𝐢𝐠𝐫𝐚𝐩𝐡𝐞𝐫, an aerial manipulation system that can draw various calligraphy artworks:. 🎯Contact-awa….

RT @abhishekunique7: So I hear that behavior cloning is all the rage now. What if we could do better, but with the same data? :) In CCIL, w….

RT @abhishekunique7: Excited about @ZoeyC17's new work on real2sim for robotics! We present URDFormer, a technique to learn models that go….