Teleoperation is slow, expensive, and difficult to scale. So how can we train our robots instead? Introducing X-Sim: a real-to-sim-to-real framework that trains image-based policies 1) learned entirely in simulation 2) using rewards from human videos. https://t.co/5yt2iTFYF4

Check out our paper and project page for more information. Prompting with the Future: Open-World Model Predictive Control with Interactive Digital Twins 📄 https://t.co/rQhV0uIETo 🌐 https://t.co/U3m6v1oSCu 💻 https://t.co/tpXOnRMgLP Huge thanks to my advisors: @KuanFang and

By explicitly modeling dynamics with an interactive digital twin, our method significantly outperforms baselines that directly prompt or fine-tune the VLM to handle both semantics and physics.

Our method enables the robot to perform diverse, previously unseen manipulation tasks, involving 6-DoF manipulation, tool use, and precise manipulation.

To robustly model the dynamics and provide informative inputs to the VLM, we build the digital twin from a real scene scan using a hybrid representation: Meshes for physics simulation Gaussians for photorealistic rendering

At the core of our method, we integrate a pretrained VLM with an interactive digital twin in a Model Predictive Control paradigm. 🌏 The digital twin serves as a dynamic model to predict the outcomes of different actions. 🏆 The VLM evaluates and selects the best action sequence

How can robots solve tasks that demand both semantic and physical reasoning, like playing real-world Angry Birds, without tons of data? We introduce Prompting with the Future: an MPC framework that fuses a pretrained VLM with an interactive digital twin for grounded, open-world

📢Thrilled to share our new paper: Esoteric Language Models (Eso-LMs) > 🔀Fuses autoregressive (AR) and masked diffusion (MDM) paradigms > 🚀First to unlock KV caching for MDMs (65x speedup!) > 🥇Sets new SOTA on generation speed-vs-quality Pareto frontier How? Dive in👇

🤯 GPT-4o knows H&M left Russia in 2022 but still recommends shopping at H&M in Moscow. 🤔 LLMs store conflicting facts from different times, leading to inconsistent responses. We dig into how to better update LLMs with fresh facts that contradict their prior knowledge. 🧵 1/6