Introducing POD ! Predict-Optimize-Distill : A Self-Improving Cycle for 4D Object Understanding ! Inputs: a multi-view scan of an object + casually captured, long-form human interaction monocular videos (from your phone) ! Outputs: 3D part poses over time .

Check out the project page for more! https://t.co/jH1kZqbtqC Great collab with @ChungMinKim @akanazawa @Ken_Goldberg @QianqianWang5 Brent and Mingxuan. Also shoutout to the research and visualization code at @nerfstudioteam !

4D-DPM decomposes objects into parts with GARField and assigns each a trainable pose. Using DiffRend, we optimize part motion with a combination of pixel-based losses and 3D regularizations. DINO features as a target provide good 2D-3D semantic correspondence!

Robot Do 🦾 Because robot manipulators are too clunky to mimic our hands, we instead optimize robot trajectories to match part motions after detecting the object's pose. From a single demo, RSRD can adapt to multiple object orientations during deployment!

Robot See 👁️ RSRD reconstructs object motion instead of hand motion to bridge the embodiment gap. To do this we introduce 4D Differentiable Part Models, a representation that can recover 3D part motion from a scan + monocular RGB video. See results in 4D on our site!

Robot See, Robot Do allows you to teach a robot articulated manipulation with just your hands and a phone! RSRD imitates from 1) an object scan and 2) a human demonstration video, reconstructing 3D motion to plan a robot trajectory. https://t.co/jH1kZqbtqC #CoRL2024 (Oral)

🎉 V1.0.0 officially released! * New splatfacto model, nerfstudio's implementation of Gaussian splatting * Complete rewrite of the viewer, now built entirely on viser

After a long time in the making, it’s finally live! It’s been so fun building this with the @nerfstudioteam, and we can’t wait for what’s next. Reach out if you are curious for our future plans. 🥳