Excited to share with everyone an accessible, intuitive tutorial on diffusion models! If you're curious about the math behind diffusion models and how their different interpretations can be unified, please check it out!. Stay tuned for a blog post soon!.

RT @alexwei_: 1/N I’m excited to share that our latest @OpenAI experimental reasoning LLM has achieved a longstanding grand challenge in AI….

RT @yidingjiang: A mental model I find useful: all data acquisition (web scrapes, synthetic data, RL rollouts, etc.) is really an explorati….

RT @GillmanLab: Ever wish you could turn your video generator into a controllable physics simulator? . We're thrilled to introduce Force Pr….

This is joint first-author work with Zilai Zeng @zilaizeng, and advised by Yilun Du @du_yilun and Chen Sun @jesu9. Project: Code: arXiv: Check out our #ICLR2025 poster on April 25th at 3:30pm (Poster #418)!.

We also discover that internet-scale pretraining can bridge the suboptimality gap through probabilistic adaptation and its inverse. Even for an in-domain model trained only on failed trajectories, successful video plans can be synthesized through adaptation even for novel tasks.

We particularly highlight probabilistic adaptation, which does not finetune the internet-pretrained model, but instead does adaptation via score composition. We also propose its inverse, which we demonstrate is robust and generalizable across environments, tasks and embodiments.

How can we combine benefits of internet-scale pretraining with in-domain information to create a video model for performant novel robotic task generalization?. We investigate a suite of adaptation techniques: direct finetuning, subject customization, and probabilistic adaptation.

Video models trained on small-scale sets of in-domain data capture domain-specific visual details and interaction dynamics, but have limited task generalization. Internet-scale pretrained video models exhibit zero-shot generalization, but may not model in-domain characteristics.

Video models can be applied to perform text-conditioned decision making for robotic tasks in two ways: as visual planners that generate actions to execute and as policy supervisors that critique trajectories. But what kind of training data makes for a good robotic video model?

Internet-scale datasets of videos and natural language are a rich training source!. But can they be used to facilitate novel downstream robotic behaviors across embodiments and environments?. Our new #ICLR2025 paper, Adapt2Act, shows how.

RT @scychan_brains: New work led by @Aaditya6284:."Strategy coopetition explains the emergence and transience of in-context learning in tra….

RT @dylanjsam: Excited to share new work from my internship @GoogleAI !. Curious as to how we should measure the similarity between example….

RT @ssokota: Model-free deep RL algorithms like NFSP, PSRO, ESCHER, & R-NaD are tailor-made for games with hidden information (e.g. poker).….

RT @poolio: Brush🖌️ is now a competitive 3D Gaussian Splatting engine for real-world data and supports dynamic scenes too! Check out the re….

RT @XiaomengXu11: Can robots leverage their entire body to sense and interact with their environment, rather than just relying on a central….

RT @dylanjsam: To trust LLMs in deployment (e.g., agentic frameworks or for generating synthetic data), we should predict how well they wil….

This is work with co-authors Mandy He, Zilai Zeng @zilaizeng, and Chen Sun @jesu9. Project Page: arXiv: Code: Come check out our poster at #NeurIPS2024 on Thursday, December 12th at 4:30pm (Poster #6706)!.

TADPoLe leverages text-conditioned diffusion models generally pretrained on large-scale data, enabling its application for policy learning across robotic agents and environments without any modification or finetuning - including for visually synthetic settings such as MetaWorld.

TADPoLe enables the learning of text-conditioned policies in a zero-shot manner, avoiding the need to handcraft reward functions for novel behaviors. Crucially, it also demonstrates sensitivity to subtle details of provided text prompts, such as pose specifications.

In practice, TADPoLe does not require completely denoising over many timesteps to a clean reconstruction; rather, we devise a way to extract a reward using just one inference step. We thus provide dense, text-conditioned rewards to agents during training in an efficient manner.