Presenting our EmbodiedAI Workshop paper in collaboration with @UCU_Faculty_of_APPS and @rpartsey "What Do We Learn from Using Text Captions as a Form of 3D Scene Representation?" poster 76 ARCH 4E at #CVPR2024 from 1pm to 2pm today.

Excited to be at #ICLR2024! If you're Ukrainian or interested in hearing about science and education in Ukraine, I'd love to connect and talk to you.

Excited to be among core Habitat contributors for this release. Habitat 3.0 is out!

The future of robot butlers starts with mobile manipulation. We’re announcing the NeurIPS 2023 Open-Vocabulary Mobile Manipulation Challenge! - Full robot stack ✅ - Parallel sim and real evaluation ✅ - No robot required ✅👀 https://t.co/mggAbRhrLP

https://t.co/7MB7aITPM5 💔

Nice coverage of sim2real work using Habitat.

Embodied AI researcher’s Christmas/New Year experiments in @ai_habitat:

I’m delighted to be listed among the Habitat current contributors on the AI Habitat website https://t.co/icD8vLSnBS.

Even though this is not the first release I contributed to. But it is special for me as I was co-migrating Habitat’s configuration system to Hydra (that’s one of the biggest enhancements).

Habitat v0.2.3 with a lot of new features and enhancements is released!

Paper Club with the author of the paper, what could be better? Let's meet to discuss the new paper and answer the question: Is Mapping Necessary for Realistic PointGoal Navigation? Lecturer and research author — @rpartsey. ⚡️Register at https://t.co/OHGpdZmWRz

🔥

Featuring #CVPR2022 paper by @rpartsey @erikwijmans Naoki Yokoyama @dobosevych @DhruvBatraDB @o_maksymets https://t.co/liV5RdVbIP

In zero-shot sim2real experiments, the agent does well at avoiding obstacles, but the most challenging part seems to be stopping within the success threshold. Across 9 episodes, it achieves 11% Success, 65% SoftSPL, and makes it 92% of the way to the goal (SoftSuccess).

Trained on Gibson train split, our agent follows a near-perfect path on both Gibson and MP3D val scenes. It scores 96% Success, 77% SPL, 76% SoftSPL on Gibson and 79% Success, 60% SPL, 69% SoftSPL on Matterport3D.

Our visual odometry additions advance the state of art on the Habitat Realistic Challenge PointNav Track from 71% to 94% Success (+32% relative) compared to the strongest published work and from 91% to 94% Success (+3% relative) compared to the strongest concurrent work.

Specifically, we apply Flip and Swap augmentations during the navigation time to achieve an ensemble effect and incorporate the information about the action taken as a fixed action-specific vector concatenated to the input of all fully connected layers.

First, we identify that the lack of localization is the core bottleneck by demonstrating that an agent with perfect localization can overcome the above-mentioned challenges (row 2). Then, we focus on robust odometry estimation solely from perceptual input (row 3).

Our pipeline consists of two components: an RNN-based navigation policy that decides which action to take to reach the goal and a CNN-based visual odometry module that estimates relative pose change that is used to update the goal coordinates to be wrt. current pose.

We study a problem of PointGoal navigation (“Go to (x, y)”) under realistic conditions: 1) imperfect sensors (noisy RGB-D), 2) inaccurate motors (noisy actuation), 3) no ground-truth localization (GPS+Compass sensor).