Happy to share the work from my internship at @nianticspatial: ACE-G, which we will present at ICCV next week in Hawaii 🌴 ACE-G introduces large-scale pre-training to previously mainly per-scene-optimized scene coordinate regressors. (1/n)

Bruns et al., "ACE-G: Improving Generalization of Scene Coordinate Regression Through Query Pre-Training" Train a scene coordinate regressor with "map codes" (ie, trainable inputs) so that you can train one generalizable regressor. Then, find these "map codes" to localize.

ACE-G: Improving Generalization of Scene Coordinate Regression Through Query Pre-Training @leonard_bruns @axelbarrosotw , Tommaso Cavallari, @AMonszpart Sowmya Munukutla, Victor Adrian Prisacariu @eric_brachmann tl;dr: SCR=attention(image x map code) https://t.co/ZTU4fKIiS3

ACE-G: Improving Generalization of Scene Coordinate Regression Through Query Pre-Training @leonard_bruns, @axelbarrosotw, Tommaso Cavallari, @AMonszpart, @scriptide, @viprad, @eric_brachmann tl;dr: transformer-based scene-agnostic coordinate regressor+map codes; separate

Shoutout to everyone involved in this project: @axelbarrosotw, Tommaso Cavallari, @AMonszpart, @scriptide, @viprad, @eric_brachmann (7/n)

Check out our project page for more explanations, examples, and the paper! (6/n) https://t.co/E3mexB6SIA

Here's an example comparing ACE-G's estimate (red) to ACE (green) and the ground-truth (blue). ACE fails to estimate the correct pose for significant changes in lighting, viewpoint, and objects. (5/n)

During pre-training, we optimize many scenes in parallel and repeatedly replace scenes after they have converged using precomputed mapping & query data. This way, our network can be pre-trained across 1000s of mapping-query configuration. (4/n)

We pre-train the scene-agnostic regressor specifically for mapping → query generalization by intertwining mapping pre-training (optimize map codes and regressor) and query pre-training (freeze map codes and only optimize regressor). (3/n)

To do so, we split up the scene-specific MLP of prior works into scene-specific map codes and a scene-agnostic coordinate regressor. The map codes takes the place of the weights in the MLP and the regressor estimates the scene coordinates of images given the map code. (2/n)

I increasingly think OpenAI is trying to destroy the concept of copyright through desensitization. Copyright is a huge threat to their profits. They are facing multiple massive copyright lawsuits. So they release Sora, knowing it can output copyrighted IP and knowing people

Vista is a generative driving world model by Gao et al. Built on Stable Video Diffusion, it can generate driving scenes conditioned on a single input image and optional, additional control inputs. In this example, we visualize the latent diffusion steps and the generated, decoded

HOT3D in Rerun! 🫱⌨️🫲 HOT3D is a new benchmark dataset for vision-based understanding of 3D hand-object interactions by @TweetEdMiller and the Project Aria team at @AIatMeta.

Check out the Neural Graph Mapping for Dense SLAM with Efficient Loop Closure paper by @leonard_bruns, Jun Zhang, and Patric Jensfelt, visualized with Rerun. “Existing neural field-based SLAM methods typically employ a single monolithic field as their scene representation. This

Project Aria data in Rerun! 👓 Project Aria is a research program from @AIatMeta to accelerate AR and AI from a human perspective. Project Aria glasses are a sensor-packed device that helps researchers gather multimodal data (video, audio eye-tracking, and location) from the

Go2 in Rerun: visualizing @UnitreeRobotics Go2 quadruped using Rerun’s ROS 2 bridge PoC. This example visualizes the Go2 robot, its onboard sensor data, and how it sees the environment. It’s built using Rerun’s new proof-of-concept C++ ROS 2 bridge, which currently supports a

Thanks for sharing @ducha_aiki It's actually none or multiple fields per keyframe. Depending on how much in a new keyframe is not covered by the existing fields yet. Will share a few more details once I'm back from IROS 🙂

Thanks for sharing our work 🙂

Sim-to-Rerun: Using Rerun to visualize CARLA simulator output through #ROS 2. This example visualizes simulated sensor data from the CARLA simulator for autonomous driving in Rerun. It’s built using Rerun’s new proof-of-concept C++ ROS2 bridge, which currently has support for a

A Rerun Viewer for the DROID Dataset! DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset is a robot manipulation dataset by @SashaKhazatsky et al. with 76k demonstration trajectories or 350h of interaction data, collected across 564 scenes and 86 tasks.