Many #robot_learning works use human videos but need lots of data/retraining. We present #HRT1 — a robot learns from just one human video and performs mobile manipulation tasks in new environments with relocated objects — via trajectory transfer.🔗 https://t.co/zbOGWrSHAF (1/11)

(11/11) #HRT1 shows that robots don't always need huge training pipelines. With the right system, a single human demo goes a long way. One demo → mobile manipulation → real-world execution. @saihaneesh_allu @jishnu_jaykumar @YuXiang_IRVL

(10/11) The standout aspect: No Training Needed: Full mobile manipulation with no learned policy — one shot, no RL, no finetuning. One human demo → real robot execution. And while most progress is on tabletop tasks, #HRT1 shows how far unified mobility + manipulation can go.

(9/11) A current limitation: performance depends on object pose estimation, creating an open-loop step. A brief manual pose check ensures stability. Depth also degrades in high-light or low-light scenes.

(8/11) After reaching the optimized base pose, we run a nonlinear constrained optimization to compute the joint-space trajectory that accurately imitates the rollout path.

(7/11) We solve a constrained optimization to find the optimal base pose (x, y, θ) so the robot can feasibly follow the rollout trajectory using trajectory optimization.

(6/11) We express the demonstration in the object frame. During rollout, BundleSDF estimates the relative pose change from RGB-D demo + rollout frames, adapting the demo to the new scene and producing the rollout trajectory.

(5/11) We convert the 3D human hand mesh trajectory into a robot end-effector trajectory, forming the robot's demonstration motion.

(4/11) From the RGB-D demo, we (i) reconstruct a 3D human hand mesh using #HaMeR: https://t.co/QWvTSaGCpL and (ii) refine its alignment w.r.t. the human hand via a depth-based optimization. Thus, we extract the precise 3D hand mesh trajectory that represents the demonstration.

(3/11) Data collection shouldn't be a bottleneck. Wear an AR headset, perform the task once from the robot's viewpoint, and HRT1 extracts all needed info from a single RGB-D demo.

(2/11) We didn't reinvent the wheel — we unified it. With MR-based, hands-free data collection, HRT1 unifies scene understanding (detection, segmentation, pose estimation), 3D hand tracking + grasp transfer, and motion optimization, yielding a coherent mobile-manipulation system.

It was great to have Jiafei @DJiafei visiting us today @IRVLUTD! He gave an inspiring talk on “Towards Robotics Foundation Models that can Reason.” I especially liked two aspects of MolmoAct: • predicting robot trajectories for manipulation • using depth inside the network

Are you interested in Vision-Language-Action (VLA) models? We had an excellent guest lecture today by Ankit Goyal @imankitgoyal from NVIDIA on VLAs and their role in robot manipulation 🎥 Watch the recording here 👇 https://t.co/fhl15iKGr8 Slides: https://t.co/9lxXTXtX9r

We have two papers accepted to #IROS2025 🎉 Unfortunately, due to EOGA48 in Texas, we won’t be able to attend the conference. Please feel free to reach out to the authors if you’re interested in our work!

Congrats Dr. Yangxiao Lu @parker_sean_L . Happy farewell and all the very best for your next phase !!

Recognizing unseen objects is a challenge our lab keeps working on. This dataset from Amazon looks exciting to explore! Our latest work @parker_sean_L @ruosenli @jingliqiang6 @JwRobotics combines few-shot object detection with LLMs for visual grounding https://t.co/wz6pSvA3mC

A fantastic way to invoke the curious young minds! Part of research is to show the work through demos !!

Check out the awesome work https://t.co/lRverD96JF

Our post-ICRA party was at the FC Dallas soccer game ⚽ @IRVLUTD

Good luck Jishnu