SMERF: Streamable Memory Efficient Radiance Fields

Original Article ↗ Hacker News Discussion ↗

Google’s SMERF project showcases streamable, memory‑efficient neural radiance fields that let users walk through highly realistic 3D reconstructions of real spaces in a web browser, even on older smartphones. Commenters are impressed by the quality, reflections, and “mirror worlds,” while probing trade‑offs such as spatial resolution in large scenes, large payload sizes, and lengthy multi‑GPU training. The thread also explores how SMERF compares to 3D Gaussian Splatting and traditional photogrammetry, potential applications in VR and real estate, and the prospects for open‑sourcing the training code and tooling.

Overall reactions & performance

  • Many commenters found the web demos “stunning” and surprisingly smooth, even on older smartphones and mid‑range devices.
  • Desktop performance was also reported as excellent, with real‑time frame rates at high quality.
  • Some users had issues in Firefox due to GPU/ANGLE/Direct3D problems; authors note the viewer was not well‑tested there and recommend WebKit/Chromium for now.

Availability, models & tooling

  • Pretrained models are already served via the demo; running the page downloads them.
  • The web viewer source is on GitHub under Apache 2.0; people are encouraged to hack on it (e.g., add VR, better controls).
  • Full training code is not yet released due to dependencies on internal libraries; authors hope to open‑source it after disentangling.
  • Current capture pipeline: camera (often DSLR), structure‑from‑motion (e.g., COLMAP) for camera poses, teacher NeRF (Zip‑NeRF), then SMERF training, then asset export for the viewer.

Technical details & limitations

  • Representation: a neural radiance field backed by a sparse low‑res voxel grid plus dense high‑res triplanes (“glowing fog” ray‑marched).
  • Quality in large spaces is limited by spatial resolution; more volume requires more voxels, so big scenes trade detail for coverage.
  • SMERF quality is also upper‑bounded by the Zip‑NeRF teacher.
  • Reflective surfaces (mirrors, glossy metal, TVs) behave like “window into mirror‑world” artifacts; users report eerie mirror/fridge/TV “interiors.”
  • Scenes are static; training is heavy: roughly 12–48 hours on large multi‑GPU setups per scene.
  • Viewer streams multiple sub‑models and swaps them in/out of memory; network payloads are large, and compression is an open problem.

Comparisons to other methods

  • Closely related to NeRF; Gaussian Splatting is “heavily inspired” and uses similar inputs/outputs but a different representation.
  • SMERF reportedly has visibly higher quality than 3D Gaussian Splatting on large scenes and slightly better on small scenes, but takes much longer to train; rendering speed on CUDA GPUs is comparable.
  • Related large‑scale work (e.g., Block‑style NeRFs) uses many regional models for big environments.

Applications, ideas & critiques

  • Strong interest in VR/AR (including standalone headsets) and in real‑estate walkthroughs as a significant upgrade over current 360‑photo tours.
  • Suggestions include motion/gyro controls on phones, progressive/priority loading, and compositing rasterized game characters into SMERF environments.
  • Some skepticism appears around marketing tone and lack of detailed limitations discussion, and around the current need for heavy optimization and per‑scene reconstruction.