Introducing SMERF: a streamable, memory-efficient method for real-time exploration of large, multi-room scenes on everyday devices. Our method brings the realism of Zip-NeRF to your phone or laptop!. Project page: ArXiv: . (1/n)

I had the pleasure last month of giving slightly provocative talk at Bliss AI, a a wonderful student-run organization here in Berlin. The best part: the talk is online for everyOne to enjoy! Behold: "NeRF is dead" .

RT @ChrisJReiser: Are you at #SIGGRAPH2024 and want to learn how to reconstruct meshes from multi-view images that contain details like in….

RT @zhenjun_zhao: InterNeRF: Scaling Radiance Fields via Parameter Interpolation. @clintonjwang, @PeterHedman3, Polina Golland, @jon_barron….

RT @gigazine: Googleがリアルタイムに高精度なレンダリングを実現する新技術「SMERF」を発表.

RT @MartinNebelong: Seeing the amazing new SMERF technology immediately made me imagine a time when we can walk around in environments like….

RT @bilawalsidhu: Thoughts NeRFs were dead? Google DeepMind just dropped SMERF — streamable, multi-room NeRFs with cm-level detail. Oh and….

RT @RadianceFields: SMERF from Google Research (again) achieves Zip-NeRF quality, operating at a remarkable 60fps on everyday devices like….

This has been a joint work with my amazing collaborators: @PeterHedman3, @chrisjreiser, @PeterZhizhin, @jfthibert, @mariolucic_, @RSzeliski, and @jon_barron. Learn more and try SMERF out yourself at (8/n).

The result: a set of compact, streaming-ready submodels ready to run at up to 60 fps in your browser. The best part: you can try it out yourself: (7/n)

Only a single submodel needs to be in memory at a time, and while the user explores the space, we swap out old submodels and stream in new ones. We train submodels to be mutually consistent, making transitions barely noticeable. (6/n)

We also modify MERF to significantly improve visual fidelity on small-to-medium size scenes. Our submodels capture thin geometry, high-resolution textures, and specular highlights better than ever before. (5/n)

How do we achieve this? We distill a teacher model into a family of MERF-like student submodels, each of which specializes to a different part of the scene. Each submodel captures the entire scene, so rendering stays fast and GPU memory consumption stays low. (4/n)

SMERF has the best of both worlds: we produce renders nearly indistinguishable from Zip-NeRF while rendering at 60 fps or more on desktops, laptops, and even recent smartphones, all while scaling to scenes as big as a house!. (3/n)

Existing approaches for view-synthesis are torn between two conflicting goals: high quality and fast rendering. Most methods only achieve one or the other. (2/n)

While this blog post may only have two authors, the project itself is the hard work of a number of amazing teammates. Take a peak at the "Acknowledgments" section -- you may spot a few familiar names :).

From the moment NeRF was first published, the research community knew it would be something game-changing. I'm proud to be part of the team turning this amazing line of work into a real product experience!.

RT @BenMildenhall: Code finally released for our CVPR 2022 papers (mip-NeRF 360/Ref-NeRF/RawNeRF)! You can also find links for each paper's….

I'm stoked to be a contributor on Object SRT, a new method for unsupervised, posed-images-to-3D-scene representation and segmentation! It's crazy fast and, while far from perfect, is leaps and bounds better than anything I've seen yet :).

None of this would be possible without my amazing collaborators! @negative_result, @sschoenholz, @ethansdyer, and @jaschasd.