Big thanks to the entire team!.@JoeyHejna @tylerlum23 @onclk_ @AleksTaranovic @DenBless94 @geri_neumann @leto__jean @DorsaSadigh.ArXiv: Project: Code: 7/7.

Few-shot prompting makes it even better. By providing the VLM with just a few good plans, it generates more accurate trajectories - fixing common failure modes. With few-shot prompts, our success rate improves to 81% across tasks. Fridge doors finally open correctly!.6/7

We also perform zero-shot sim-to-real transfer on 3 real-world tasks:.- Pick and Place.- Pushing.- Hammering.No demos. No tuning. Just vision, language, and simulation. 5/7

We evaluate on 8 dexterous manipulation tasks covering:. - Semantic understanding (e.g., "move apple to board"). - Articulated object control. - Unstructured motion. - Precise manipulation.Avg success: 72% with no reward engineering or demonstrations. 4/7

Our approach, in detail:.The VLM generates:.1. Object keypoint trajectories – how the objects move.2. Coarse hand trajectory – how the robot moves.We use keypoint tracking as the reward and train a residual RL policy to refine hand motion and learn precise finger control. 3/7.

Dexterous robot hands offer fine control but are hard to train. Challenges:.- costly human demos.- task-specific RL rewards.We bypass this by combining VLMs (for high-level spatial/semantic reasoning) with RL, simplifying the entire training process. 2/7

Can we teach dexterous robot hands manipulation without human demos or hand-crafted rewards?.Our key insight: Use Vision-Language Models (VLMs) to scaffold coarse motion plans, then train an RL agent to execute them with 3D keypoints as the interface. 1/7