Introducing🌍 Awesome-World-Models, a one-stop github repo of everything there is to know about world models! Here is a new, curated one-stop resource list for everyone interested in "World Models," aiming to be a go-to guide for researchers and developers in the field. 🧵(1/n)

Excited to share SoftMimic -- a new approach for learning compliant humanoid policies that interact gently with the world.

Simulation drives robotics progress, but how do we close the reality gap? Introducing GaussGym: an open-source framework for learning locomotion from pixels with ultra-fast parallelized photorealistic rendering across >4,000 iPhone, GrandTour, ARKit, and Veo scenes! Thread 🧵

I always found it puzzling how language models learn so much from next-token prediction, while video models learn so little from next frame prediction. Maybe it's because LLMs are actually brain scanners in disguise. Idle musings in my new blog post:

Is RL really scalable like other objectives? We found that just scaling up data and compute is *not* enough to enable RL to solve complex tasks. The culprit is the horizon. Paper: https://t.co/KsNZgk782S Thread ↓

What makes a robot hand design better at learning from human demonstrations? Is it being similar in size to a human hand, or matching its degrees of freedom? DexMachina lets us explore this question in simulation — and the results are quite interesting! Check it out 😉

We came up with a really simple way to train flow matching (diffusion) policies with offline RL! Flow Q-learning from @seohong_park uses a distillation (reflow-like) scheme to train flow matching actor, and works super well! Check it out: https://t.co/TYYXGuyAgI

🚀 Meet ToddlerBot 🤖– the adorable, low-cost, open-source humanoid anyone can build, use, and repair! We’re making everything open-source & hope to see more Toddys out there!

1/9 🚨 New Paper Alert: Cross-Entropy Loss is NOT What You Need! 🚨 We introduce harmonic loss as alternative to the standard CE loss for training neural networks and LLMs! Harmonic loss achieves 🛠️significantly better interpretability, ⚡faster convergence, and ⏳less grokking!

I've been waiting for this for a while. Open source procedural scene generation from NVIDIA. This kind of thing would be really useful for scaling up simulation data for robots.

We RL'ed humanoid robots to Cristiano Ronaldo, LeBron James, and Kobe Byrant! These are neural nets running on real hardware at our GEAR lab. Most robot demos you see online speed videos up. We actually *slow them down* so you can enjoy the fluid motions. I'm excited to announce

Atlas doing a quick warm up before work.

🔍This study focuses on overcoming the challenges of #ReinforcementLearning (RL) for motor control policies in complex tasks such as #dexterousmanipulation. 🔗Check it out: https://t.co/a1ht4xotL9 @GaganKhandate @XiaoYangLiu10 @ColumbiaCompSci @CUSEAS @Columbia

Over 50 researchers in the robot learning community joining forces on a mission to scale up robot learning to an unprecedented level 🚀 It’s amazing to see what we can achieve as a team! I made so many new friends in the process and I’m truly grateful for that ❤️

Introducing ZeroRF, where we reconstruct high-quality radiance fields from *sparse views* at an order of magnitude faster than previous methods (30 secs for 320x320) without any pretraining or additional regularization! https://t.co/9XCRqrgX0r

Represent robot policies as trajectories through space. Naturally allows for cross-embodiment transfer and learning from human video!

Let me clear a *huge* misunderstanding here. The generation of mostly realistic-looking videos from prompts *does not* indicate that a system understands the physical world. Generation is very different from causal prediction from a world model. The space of plausible videos is

Can we collect robot data without any robots? Introducing Universal Manipulation Interface (UMI) An open-source $400 system from @Stanford designed to democratize robot data collection 0 teleop -> autonomously wash dishes (precise), toss (dynamic), and fold clothes (bimanual)

Can we use VLMs out of the box to solve robot control and embodied tasks? Our new work PIVOT shows how this can be done! We used PIVOT to sort food, find you a conference room, and even help you make a cute smiley face out of fruits :) Check it out: https://t.co/68toEa5Ndc

Current works are restricted to short sequences of texts and images, limiting their ability to model the world. Presenting Large World Model (LWM): capable of processing long text, images, videos of over 1M tokens (and *no* lost in the middle!) Project: