We’re releasing the RoboArena today!🤖🦾. Fair & scalable evaluation is a major bottleneck for research on generalist policies. We’re hoping that RoboArena can help!. We provide data, model code & sim evals for debugging! Submit your policies today and join the leaderboard! :).🧵

I'll give a talk about benchmarking generalist policies today at RSS (4:30p, RTH 526, in the benchmarking workshop)!.I will discuss sim eval, auto eval, and distributed real-world eval (ie RoboArena) -- swing by :)

RT @RobobertoMM: It was time to improve our evaluations in robot learning! We introduce a methodology based on anonymous A/B testing: faire….

RT @abhishekunique7: Check out some of our new work on distributed robot evaluation led by @KarlPertsch, @pranav_atreya and @tonyh_lee! Hop….

RT @tonyh_lee: 🚀 We just launched RoboArena — a real-world evaluation platform for robot policies!.Think Chatbot Arena, but for robotics.….

RT @pranav_atreya: In robotics benchmarks are rarely shared. New eval setups are created for each new project, a stark difference from eval….

@cyrusneary @edward_s_hu @ShivinDass @JieWang_ZJUI @chelseabfinn @svlevine.

Thanks to my co-leads @pranav_atreya @tonyh_lee!.And thanks to the many collaborators from across the robotics community who agreed to help out with evals! .@moo_jin_kim @prodarhan @dineshjayaraman @RobobertoMM @GlenBerseth @abhishekunique7 @YoungwoonLee @percyliang.

You can join RoboArena both, by submitting policies, and by contributing evals! . Check our website for more details: Paper: Join RoboArena and show off your best policies! :). 7/.

RoboArena is based on the DROID platform. We provide all the resources you need to start training state of the art policies: open-source data, policy training code for DROID VLAs (now added to openpi), and DROID sim evals for debugging!. 6/

All eval episodes + scores are publicly accessible on our website + we create leaderboards and LLM-generated reports for each policy, with episode “citations” as evidence for strengths and weaknesses — check it out!. 5/

Like in Chatbot Arena, we can aggregate many such pairwise evals to compute a global policy ranking. As a result, RoboArena evals are comprehensive (many tasks + scenes), and trustworthy (no cherry-picking of tasks, evaluators don’t know which policies they evaluate). 4/.

We crowdsource evals across many institutions, and evaluators can freely choose to test *any task* in *any scene*. Only requirement: each eval runs two policies back-to-back, and provides feedback on which policy performed better. 3/

Everyone knows robot benchmarking is hard: it’s tedious & challenging to reproduce. Prior benchmarks try to standardize (objects, viewpoints, …) but it’s a loosing battle, esp for generalist policies that need eval in many tasks/scenes. RoboArena takes a different approach….2/.

RT @LerrelPinto: Final note: It is easier to work on robotics now than any point in the past.

RT @kvablack: In LLM land, a slow model is annoying. In robotics, a slow model can be disastrous! Visible pauses at best, dangerously jerky….

RT @polkirichenko: Join us at #CVPR2025 Demographic Diversity in Computer Vision workshop tomorrow!.📅 Wednesday, June 11, 9am-6pm.📍 room 21….

Check out Danny's paper on a single-stage VLA recipe that trains fast, has fast inference, and follows language commands well. ⚡️⚡️⚡️.The key: combine FAST tokens + flow-matching expert, and make sure those pesky diffusion gradients don't mess up your beautiful VLM backbone! :).

Here's the link to our original Embodied CoT work: Also lots of other works have since shown that grounded reasoning can help generalization (eg Gemini robotics, HAMSTER. ) -- I think we still have only scratched the surface on these approaches!.

Our embodied CoT work (ECoT) showed that policies generalize better when allowed to reason step-by-step, at the expense of slower inference. Will's new work investigates *why* ECoT policies work better & develops "ECoT-Lite" recipes that run much faster & still generalize well!👇.