Happy to share our latest work on efficient sensor tokenization for end-to-end driving architectures! We introduce a novel way to tokenize multi-camera input for AV Transformers that is resolution- and camera-count-agnostic, yet geometry-aware. 🧵👇.

5/ Overall: Triplane-based multi-camera tokenization enables more efficient end-to-end AV policy training and deployment!. 📄 Paper: 👥 Authors: @iamborisi, @CristianoSalto, @YurongYou, @yan_wang_9, @WenjieLuo, @drmapavone @NVIDIAAI @NVIDIADRIVE.

4/ We benchmark on massive internal driving datasets as well as well-known open datasets (e.g., nuScenes, Waymo). As one particular example: Despite using a much smaller 1B LLM, we can match or outperform 7B models on nuScenes thanks to our efficient geometry-aware tokens!.

3/ Our method is:.- Self-supervised.- Compatible with any image encoder.- Resolution-agnostic.- Camera-count-agnostic.- Trained via simple reconstruction losses (no GAN discriminators needed!).- Trainable with (optional) auxiliary signals, like depth or foundation model features

2/ We use a triplane representation to encode multiple camera inputs into a fixed number of tokens, producing.✅ Up to 72% fewer tokens.✅ Up to 50% faster inference.✅ Similar or better driving performance vs baselines.✅ Seamless scaling to more cameras and higher resolutions.

1/ Tokenization is a core bottleneck for deploying large Transformer-based AV models. Existing patch-based ViTs and autoencoders produce token counts that scale with the number of input cameras and their resolution. Enter Triplanes: an efficient volumetric latent representation

RT @chen_yiting_TW: 📢 The first X-Sense Workshop: Ego-Exo Sensing for Smart Mobility at #ICCV2025! . 🎤 We’re honored to host an outstandin….

RT @NVIDIADRIVE: 🛣️ New NVIDIA DRIVE Labs video on the future of mapless driving!. High-definition (HD) maps have been essential for autono….

RT @sephy_li: Announcing the 2025 NAVSIM Challenge! What's new? We're testing not only on real recordings—but also imaginary futures genera….

RT @MaxiIgl: Our @CVPR paper on training traffic models in closed loop is an Oral at CVPR!! The work was done by Zhejun Zhang ( https://t.c….

RT @drmapavone: At #GTC2025, Jensen unveiled Halos, a comprehensive safety system for AVs and Physical AI. Halos integrates numerous techno….

Don’t miss this deep dive into the future of autonomous vehicles! . Excited to present about how foundation models are transforming AV technology with @ALVAREZ_JOSEM at #GTC25!. Check out all the session details below 👇.

RT @drmapavone: For the first time ever, @nvidia is hosting an AV Safety Day at GTC - a multi-session workshop on AV safety. We will shar….

RT @drmapavone: Complementing DreamDrive, I am thrilled to introduce STORM, which enables fast scene reconstruction with a single feed-forw….

RT @drmapavone: Introducing DreamDrive, which combines the complementary strengths of generative AI (video diffusion) and neural reconstruc….

RT @elad_sharony: ❓ How can we significantly boost local-optimizers performance under strict runtime constraints in robotics and autonomous….

RT @MaxiIgl: Despite of what the posting says, these can be international (e.g. Europe)! Same for intern positions!.

RT @drmapavone: We also have several open positions for interns: To learn more about our work, check out our websi….

RT @drmapavone: The Autonomous Vehicle (AV) Research group @nvidia is now hiring. From AV foundation models to AI safety and generative sim….

RT @drmapavone: This week we will be presenting 3 papers @eccvconf, on online mapping (@iamborisi), traffic scenario generation, and VLM-ba….