18 ms·

SMERF: Streamable Memory Efficient Radiance Fields

630 points by duckworthd 3y ago

We built SMERF, a new way for exploring NeRFs in real-time in your web browser. Try it out yourself! Over the last few months, my collaborators and I have put together a new, real-time method that makes NeRF models accessible from smartphones, laptops, and low-power desktops, and we think we’ve done a pretty stellar job! SMERF, as we like to call it, distills a large, high quality NeRF into a real-time, streaming-ready representation that’s easily deployed to devices as small as a smartphone via the web browser. On top of that, our models look great! Compared to other real-time methods, SMERF has higher accuracy than ever before. On large multi-room scenes, SMERF renders are nearly indistinguishable from state-of-the-art offline models like Zip-NeRF and a solid leap ahead of other approaches. The best part: you can try it out yourself! Check out our project website for demos and more. If you have any questions or feedback, don’t hesitate to reach out by email (smerf@google.com) or Twitter (@duck).

azinman2 3y ago
Will there be any notebooks or other code released to train our own models?
- duckworthd 3y ago
  I hope so, but it'll be a good while before we can release anything. We have tight dependencies to other not-yet-OSS libraries, and until they're released, our won't work either.
asgerhb 3y ago
Wow! What am I even looking at here? Polygons, voxels, or something else entirely? How were the benchmarks recorded?
- duckworthd 3y ago
  You're looking at something called a "neural radiance field" backed by a sparse, low resolution voxel grid and a dense high resolution triplane grid. That's a bit of a word soup, but you can think of it like a glowing fog rendered with ray marching. The benchmark details are a bit complicated. Check out the technical paper's experiment section for the nitty gritty details.
  - asgerhb 3y ago
    Hahah I'm in formal methods, not graphics, so I'm picturing a literal green glowing fog. But what I'm gathering is polygons partitioned into cube chunks. This is of course not including the particulars of this impressive contribution, but I fear that's beyond my ken.
RagnarD 3y ago
I'm curious how the creators would compare this to the capabilities of Unreal Engine 5 (as far as the display technology goes.)
mdrzn 3y ago
Impressive is not a big enough statement! This is incredibly smooth on my phone and crazy good on a desktop pc. Keep it up!
- duckworthd 3y ago
  Thank you :)
jsk95505 3y ago
[flagged]
kolja005 3y ago
Been following Jon Barron et al’s work for a little while now. The speed of improvement given the complexity of these types of systems is mind-boggling. I wonder how long it takes before Google street view gets replaced by some NeRF variant.
boppo1 3y ago
How long until I can make my own?
deleted 3y ago
[deleted]
sim7c00 3y ago
this looks really amazing. i have a relatively old smartphone (2019) and its really surprisingly smooth and high fidently. amazing job!
- duckworthd 3y ago
  Thank you :). I'm glad to hear it! Which model are you using?
  - sim7c00 3y ago
    samsung galaxy 10se
guywithabowtie 3y ago
Any plans to release the models ?
- duckworthd 3y ago
  The pretrained models are already available online! Check out the "demo" section of the website. Your browser is fetching the model when you run the demo.
  - ilaksh 3y ago
    Will the code be released, or an API endpoint? Otherwise it will be impossible for us to use it for anything.. since it's Google I assume it will just end up in a black hole like most of the research.. or five years later some AI researchers leave and finally create a startup.
    - duckworthd 3y ago
      I hope to release code in the new year, but it'll take a while. The codebase is heavily wired into other not-yet-open-sourced libraries, and it'll take a while to disentangle them.
      - ilaksh 3y ago
        That sounds terrific! I really appreciate your effort. It's amazing work and so great of you to share it.
zeusk 3y ago
Are radiance fields related to Gaussian splattering?
- duckworthd 3y ago
  Gaussian Splatting is heavily inspired by work in radiance fields (or NeRF) models. They use much of the same technology!
- corysama 3y ago
  Similar inputs, similar outputs, different representation.
joeldg 3y ago
[dead]
aappleby 3y ago
Very impressive demo.
- duckworthd 3y ago
  Thank you!
refulgentis 3y ago
This is __really__ stunning work, huge, huge, deal that I'm seeing this in a web browser on my phone. Congratulations! When I look at the NYC scene in the highest quality on desktop, I'm surprised by how low-quality ex. the stuff on the counter and shelves is. So then I load the lego model, and see that's _very_ detailed, so it doesn't seem inherent to the method. Is it a consequence of input photo quality, or something else?
- duckworthd 3y ago
  > This is __really__ stunning work Thank you :) > Is it a consequence of input photo quality, or something else? It's more a consequence of spatial resolution: the bigger the space, the more voxels you need to maintain a fixed resolution (e.g. 1 mm^3). At some point, we have to give up spatial resolution to represent larger scenes. A second limitation is the teacher model we're distilling. Zip-NeRF (https://jonbarron.info/zipnerf/ https://jonbarron.info/zipnerf/) is good, but it's not _perfect_. SMERF reconstruction quality is upper-bounded by its Zip-NeRF teacher.
  - deleted 3y ago
    [deleted]
CuriousCosmic 3y ago
Is there a relatively easy way to apply these kinds of techniques (either NeRFs or gaussian splats) to larger environments even if it's lower precision? Like say small towns/a few blocks worth of env.
- duckworthd 3y ago
  In principle, there's no reason you can't fit multiple City blocks at the same time with Instant NGP on a regular desktop. The challenge is in estimating the camera and lens parameters over such a large space. I expect such a reconstruction to be quite fuzzy given the low space resolution.
- ibrarmalik 3y ago
  You’re under the right paper for doing this. Instead of one big model, they have several smaller ones for regions in the scene. This way rendering is fast for large scenes. This is similar to Block-NeRF [0], in their project page they show some videos of what you’re asking. As for an easy way of doing this, nothing out-of-the-box. You can keep an eye on nerfstudio [1], and if you feel brave you could implement this paper and make a PR! [0] https://waymo.com/intl/es/research/block-nerf/ https://waymo.com/intl/es/research/block-nerf/ [1] https://github.com/nerfstudio-project/nerfstudio https://github.com/nerfstudio-project/nerfstudio
  - CuriousCosmic 3y ago
    Oooh fun. I'm glad it seems possible nowadays. I might take a swing at putting together an out of the box tool at some point if nobody beats me to it first.
barrkel 3y ago
The mirror on the wall of the bathroom in the Berlin location looks through to the kitchen in the next room. I guess the depth gauging algorithm uses parallax, and mirrors confuse it, seeming like windows. The kitchen has a blob of blurriness as the rear of the mirror intrudes into kitchen, but you can see through the blurriness to either room. The effect is a bit spooky. I felt like a ghost going through walls.
- rpastuszak 3y ago
  Try noclipping through the TV in the Berlin living room. It gets pleasantly creepy.
- rzzzt 3y ago
  You can also get inside the bookcase for the ultimate Matthew McConaughey experience.
- nightpool 3y ago
  The refigerator in the NYC scene has a very slick specular lighting effect based on the angle you're viewing it from, and if you go "into" the fridge you can see it's actually generating a whole 3d scene with blurry grey and white colors that turn out to precisely mimic the effects of the light from the windows bouncing off the metal, and you can look "out" from the fridge into the rest of the room. Same as the full-length mirror in the bedroom in the same scene—there's a whole virtual "mirror room" that's been built out behind the mirror to give the illusion of depth as you look through it. Very cool and unique consequence of the technology
  - alkonaut 3y ago
    What does the reconstructed space look like when there are opposing mirrors? It’ll just be a long corridor of ever more blurry rooms?
  - pjc50 3y ago
    Funnily enough, this is how reflections are usually emulated in game engines that do not support raytracing: another copy of the world behind the mirror. Also used in films in a few places (e.g. Terminator)
promiseofbeans 3y ago
It runs impressively well on my 2yo s21fe. It was super impressive how it streamed in more images as I explored the space. The tv reflections in the Berlin demo were super impressive. My one note is that it look a really long time to load all the images - the scene wouldn't render until all ~40 initial images loaded. Would it be possible to start partially rendering as the images arrive, or do you need to wait for all of them before you can do the first big render?
- duckworthd 3y ago
  Pardon our dust: "images" is a bad name for what's being loaded. Past versions of this approach (MERF) stored feature vectors in PNG images. We replace them with binary arrays. Unfortunately, all such arrays need to be loaded before the first frame can be rendered. You do however point out one weakness of SMERF: large payload sizes. If we can figure out how to compress them by 10x, it'll be a very different experience!
  - promiseofbeans 3y ago
    Or even just breaking them down into smaller chunks (prioritise loading the ones closer to where the user is looking) could help
    - duckworthd 3y ago
      The viewer biases towards assets closer to user's camera (otherwise you'd have to load the whole scene!). We tried training SMERF with a larger number of smaller submodels, but at some point, it becomes too onerous to train and quality begins to suffer.
VikingCoder 3y ago
Wow. Some questions: Take for instance the fulllivingroom demo. (I prefer fps mode.) 1) How many images are input? 2) How long does it take to compute these models? 3) How long does it take to prepare these models for this browser, with all levels, etc? 4) Have you tried this in VR yet?
- vyrotek 3y ago
  Not exactly what you asked for. But I recently came across this VR example using Gaussian Splatting instead. Exciting times. https://twitter.com/gracia_vr/status/1731731549886787634 https://twitter.com/gracia_vr/status/1731731549886787634 https://www.gracia.ai https://www.gracia.ai
- duckworthd 3y ago
  Glad you liked our work! 1) Around 100-150 if memory serves. This scene is part of the mip-NeRF 360 benchmark, which you can download from the corresponding project website: https://jonbarron.info/mipnerf360/ https://jonbarron.info/mipnerf360/ 2) Between 12 and 48 hours, depending on the scene. We train on 8x V100s or 16x A100s. 3) The time for preparing assets is included in 2). I don't have a breakdown for you, but it's something like 50/50. 4) Nope! A keen hacker might be able to do this themselves by editing the JavaScript code. Open your browser's DevTools and have a look -- the code is all there!
  - dougmwne 3y ago
    Do you need position data to go along with the photos or just the photos? For VR, there’s going to be some very weird depth data from those reflections, but maybe they would not be so bad when you are in headset.
    - duckworthd 3y ago
      > Do you need position data to go along with the photos or just the photos? Short answer: Yes. Long answer: Yes, but it can typically be derived from images. Structure-from-motion methods are typically used to derive lens and position information for each photo in the training set. These are then used by Zip-NeRF (our teacher) and SMERF (our model) to train a model.
durag 3y ago
Any plans to do this in VR? I would love to try this.
- duckworthd 3y ago
  Not at the moment but an intrepid hacker could surely extend our JavaScript code and put something together. UPDATE: The code for our web viewer is here: https://github.com/smerf-3d/smerf-3d.github.io/blob/main/viewer/index.html https://github.com/smerf-3d/smerf-3d.github.io/blob/main/vie...
deleted 3y ago
[deleted]
- deleted 3y ago
  [deleted]
blovescoffee 3y ago
Since you're here @author :) Do you mind giving a quick rundown on how this competes with the quality of zip-nerf?
- duckworthd 3y ago
  Check out our explainer video for answers to this question and more! https://www.youtube.com/watch?v=zhO8iUBpnCc https://www.youtube.com/watch?v=zhO8iUBpnCc
heliophobicdude 3y ago
Great work!! Question for the authors, are there opportunities, where they exist, to not use optimization or tuning methods for reconstructing a model of a scene? We are refining efficient ways of rendering a view of a scene from these models but the scenes remain static. The scenes also take a while to reconstruct too. Can we still achieve the great look and details of RF and GS without paying for an expensive reconstruction per instance of the scene? Are there ways of greedily reconstructing a scene with traditional CG methods into these new representations now that they are fast to render? Please forgive any misconceptions that I may have in advanced! We really appreciate the work y'all are advancing!
- duckworthd 3y ago
  > Are there opportunities, where they exist, to not use optimization or tuning methods for reconstructing a model of a scene? If you know a way, let me know! Every system I'm aware of involves optimization in one way or another, from COLMAP to 3D Gaussian Splatting to Instant NGP and more. Optimization is a powerful workhorse that gives us a far wider range of models than a direct solver ever could. > Can we still achieve the great look and details of RF and GS without paying for an expensive reconstruction per instance of the scene? In the future I hope so. We don't have a convincing way to generate 3D scenes yet, but given the progress in 2D, I think it's only a matter of time. > Are there ways of greedily reconstructing a scene with traditional CG methods into these new representations now that they are fast to render? Not that I'm aware of! If there were, I think these works should be on the front page instead of SMERF.
annoyingnoob 3y ago
There is a market here for Realtors to upload pictures and produce walk-throughs of homes for sale.
- esafak 3y ago
  https://matterport.com/ https://matterport.com/
- ibrarmalik 3y ago
  The Luma folks made something similar: https://apps.apple.com/app/luma-flythroughs/id6450376609?l=en-GB https://apps.apple.com/app/luma-flythroughs/id6450376609?l=e...
  - duckworthd 3y ago
    Be careful with this one! Luma's offering requires that the camera follow the recorded video path. Our method lets the camera go wherever you desire!
SubiculumCode 3y ago
Im not sure why this demo runs so horribly in Firefox but not other browsers..anyone else having this?
- daemonologist 3y ago
  Runs pretty well (20-100 fps depending on the scene) for me on both Firefox 120.1.1 on Android 14 (Pixel 7; smartphone preset) and Firefox 120.0.1 on Fedora 39 (R7 5800, 64 GB memory, RX 6600 XT; 1440p; desktop preset).
  - SubiculumCode 3y ago
    It seems that for some reason, my firefox is stuck in software compositor. I am getting: WebRender initialization failed Blocklisted; failure code RcANGLE(no compositor device for EGLDisplay)(Create)_FIRST 3D11_COMPOSITING runtime failed Failed to acquire a D3D11 device Blocklisted; failure code FEATURE_FAILURE_D3D11_DEVICE2 I'm running a 3060
- duckworthd 3y ago
  We unfortunately haven't tested our web viewer in Firefox. Let us know which platform you're running and we'll do our best to take a look in the new year (holiday vacation!). In the meantime, give it a shot in a Webkit- or Chromium-based browser. I've had good results on Safari on iPhone, Chrome on Android/Macbook/Windows.
jerpint 3y ago
Just ran this on my phone through a browser, this is very impressive
- duckworthd 3y ago
  Thank you :)
catskul2 3y ago
When might we see this in consumer VR? I'm surprised we don't already but I was suspecting it was a computation constraint. Does this relieve the computation constraint enough to run on Quest 2/3? Is there something else that would prevent binocular use?
- doctoboggan 3y ago
  I recently got a new quest and I am wondering the same thing. The fact that this is currently running in a browser (and can run on a mobile device) gives me hope that we will see something like this in VR sooner rather than later.
- duckworthd 3y ago
  I can't predict the future, but I imagine soon: all of the tools are there. The reason we didn't develop for VR is actually simpler than you'd think: we just don't have the developer time! At the end of the day, only a handful of people actively wrote code for this project.
  - nojvek 3y ago
    Any plans to open source the code?
    - duckworthd 3y ago
      Yes, I hope so! But it'll take at least a few months of work. We have some tight dependencies to not-yet-open-sourced code, and until that's released, any code we put out will be dead on arrival. In the meantime, feel free to explore the live viewer code! https://github.com/smerf-3d/smerf-3d.github.io/blob/main/viewer/index.html https://github.com/smerf-3d/smerf-3d.github.io/blob/main/vie...
nox100 3y ago
memory efficient? It downloaded 500meg!
- bongodongobob 3y ago
  A. Storage isn't memory B. That's hardly anything in 2023.
  - duckworthd 3y ago
    Right-o. The web viewer is swapping assets in and out of memory as the user explores the scene. The Network and disc requirements are high but memory usage is low.
monlockandkey 3y ago
Get this on a VR headset and you have a game changer literally.
modeless 3y ago
How long until you can stitch Street View into a seamless streaming NeRF of every street in the world? I hope that's the goal you're working towards!
- duckworthd 3y ago
  ;)
  - modeless 3y ago
    Haha, too bad the Earth VR team was disbanded because that would be the Holy Grail. If someone can get the budget to work on that I'd be tempted to come back to Google just to help get it done! It's what I always wanted when I was building the first Earth VR demo...
- deelowe 3y ago
  I read another article talking about what waymo was working on and this looks oddly similar... My understanding is that the goal is to use this to reconstruct 3d models of street view images in real time.
  - duckworthd 3y ago
    Block-NeRF is a predecessor work that helped inspire SMERF, in fact! https://waymo.com/research/block-nerf/ https://waymo.com/research/block-nerf/
    - deelowe 3y ago
      Very cool. Thanks!
- xnx 3y ago
  Not every street yet: https://waymo.com/research/block-nerf/ https://waymo.com/research/block-nerf/
yarg 3y ago
What I'm seeing from all of these things is very accurate single navigable 3D images. What I haven't seen anything of is feature and object detection, blocking and extraction. Hopefully a more efficient and streamable codec necessitates the sort of structure that lends itself more easily to analysis.
- yorwba 3y ago
  You mean something like this? https://jumpat.github.io/SA3D/ https://jumpat.github.io/SA3D/ Found by putting "nerf sam segment 3d" into DuckDuckGo.
- goldenshale 3y ago
  Checkout the LERF work from the NerfStudio team at UC Berkeley. SMERF is addressing a different problem, but there are definitely ways to incorporate semantics and detection as well.
- duckworthd 3y ago
  3D understanding as a field is very much in its infancy. Good work is being done in this area, but we've got a long ways to go yet. SMERF is all about "view synthesis" -- rendering realistic images -- with no attempt at semantic understanding or segmentation.
  - cooper_ganglia 3y ago
    "It's my VR-deployed SMERF CLIP model with LLM integration, and I want it now!" It is funny how quickly goalposts move! I love to see progress though, and wow, is progress happening fast!
    - yarg 3y ago
      It's not always moving goalposts - sometimes a new technology progresses on some aspects and regresses in others. This technology is a significant step forward in some ways - but people are going to compare it to state of the art 3D renders and think that it's more impressive than it actually is. Eventually this sort of thing will have understanding of lighting (delumination and light source manipulation) and spatial structure (and eventually spatio-temporal structure). Right now it has none of that, but a layman will look at the output and think that what they're seeing is significantly closer due to largely cosmetic similarities.
fngjdflmdflg 3y ago
>Google DeepMind Google Research Google Inc. What a variety of groups! How did this come about?
- duckworthd 3y ago
  Collaboration is a thing at the Big G :)
tomatotomato31 3y ago
I'm following this through two minutes paper and I'm looking forward to using it. My grandpa died 2 years ago and in hindsight I took pictures for using them as in your demo. Awesome thanks:)
- duckworthd 3y ago
  It would be my dream to make capturing 3D memories as easy and natural as taking a 2D photos with your smartphone today. Someday!
twelfthnight 3y ago
Hope this doesn't come as snarky, but does Google pressure researchers to do PR in their papers? This really is cool, but there is a lot of self-promotion in this paper and very little discussion of limitations (and the discussion of them is bookended by qualifications why they really aren't limitations). It makes it harder for me to trust the paper if I feel like the paper is trying to persuade me of something rather than describe the complete findings.
- tomatotomato31 3y ago
  People are not allowed to be proud of their work anymore?
  - twelfthnight 3y ago
    Oh absolutely. I guess I just got the feeling reading this that there was more than the standard pride here and that there was professional PR going on. If no one else is getting that vibe I'm okay to accept it's just me.
- duckworthd 3y ago
  I won't say too much about this, but the amount of buzz around articles these days is more of "research today" sort of thing. Top conferences like CVPR receives thousands of submissions each year, and there's a lot of upside to getting your work in front of as many eyeballs as possible. By no means do I claim that SMERF is the be-all-end-all in real-time rendering, but I do believe it's a sold step in the right direction. There are all kinds of ways to improve this work and others in the field: smaller representation sizes, faster training, higher quality, and fewer input images would all make this technology more accessible.
yieldcrv 3y ago
I had read about a competing technology that was suggesting NeRF's were a dead end but perhaps that was biased?
- duckworthd 3y ago
  You're probably thinking of 3D Gaussian Splatting (3DGS), another fantastic approach to real-time novel view synthesis. There's tons of fantastic work being built on 3DGS right now, and the dust has yet to settle with respect to which method is "better". Right now, I can say that SMERF has slightly higher quality on than 3DGS on small scenes and visibly higher quality on big scenes and runs on a wider variety of devices, but takes much longer than 3DGS to train. https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
rzzzt 3y ago
What kind of modes does the viewer cycle through when I press the space key?
- duckworthd 3y ago
  Nice discovery :). Check the developer console: it'll tell you.
smusamashah 3y ago
This is very impressive but given its by Google, will some code ever be released?
- duckworthd 3y ago
  I hope to release the code in the new year, but we have some big dependencies that need to be released worse. In the meantime, you can already begin hacking on the live viewer, https://github.com/smerf-3d/smerf-3d.github.io/blob/main/viewer/index.html https://github.com/smerf-3d/smerf-3d.github.io/blob/main/vie...
cubefox 3y ago
Very impressive! Any information on how this compares to 3D Gaussian splatting in terms of performance, quality or data size?
- duckworthd 3y ago
  All these details and more in our technical paper! In short: SMERF training takes much longer, SMERF rendering is nearly as fast as 3DGS when a CUDA GPU is available, and quality is visibly higher than 3DGS on large scenes and slightly higher on smaller scenes. https://arxiv.org/abs/2312.07541 https://arxiv.org/abs/2312.07541
  - zyang 3y ago
    Is it possible to use zip-nerf to train GS to eliminate the floaters.
    - duckworthd 3y ago
      Maybe! That's the seed of a completely different research paper :)
xnx 3y ago
Does the an open source toolchain exist for capturing, processing, and hosting navigable 3D walkthroughs like this (e.g. something like an open-source Matterport)?
- duckworthd 3y ago
  Not yet, as far as I'm aware. The current flow involves a DSLR for capture, COLMAP for camera parameter estimation, one codebase for training a teacher model, our codebase for training SMERF, and our web viewer for rendering models. Sounds like an opportunity!
  - strofocles 3y ago
    Is there a significant advantage for capturing using DSLRs vs using the phone camera of a decent phone?
    - duckworthd 3y ago
      The big difference is access to fisheye lenses a burst mode that can be run for minutes at a time, and the ability to minimize the amount of camera post processing. In principle, the capture could be done with a smartphone, but the experience of doing so is pretty time consuming right now.
- gorkish 3y ago
  You don't need a toolchain for capturing; you just need the data. Get it now; process it when better tools become available. There are guides for shooting for Photogrammetry and NeRF that are generally applicable to what you need to do.
germandiago 3y ago
Amazing, impressive, almost unbelievable :O
- duckworthd 3y ago
  Thank you!
varelse 3y ago
[dead]
edrxty 3y ago
Is there any relation between this class of rendering techniques and the way the BD scenes in Cyberpunk 2077 were created? The behavior of the volume and the "voxels" seem eerily similar.
- 9dev 3y ago
  I doubt Cyberpunk uses more than a special shader for the BD sequences, but what’s a lot more remarkable to me is how similar the idea is at heart. Maybe we’re actually going to see this (maybe sans the brain-implant to record them, but hey) after all. Amazing technology, that’s for sure.
- duckworthd 3y ago
  I can't say. I'm not familiar with BD in Cyberpunk.
  - edrxty 3y ago
    https://youtu.be/KXXGS3MGCro?t=118 https://youtu.be/KXXGS3MGCro?t=118 It's a sort of replayable cutscene that happens a couple times in the game where you can wander through it. The noteworthy bit is it's rendered out of voxels that look very similar to the demos but at a much lower resolution and if you push the frustrum into any objects, you get the same kind of effect where the surface breaks into blocks.
    - duckworthd 3y ago
      Interesting effect. It does look very voxel-y. I'm not a video game developer at heart, so I can only guess how it was implemented. I doubt NeRF models were involved, but I wouldn't be surprised if some sort of voxel discretization was.
      - promiseofbeans 3y ago
        It seems like it might even just be some kind of shader
westurner 3y ago
"Researchers create open-source platform for Neural Radiance Field development" (2023) https://news.ycombinator.com/item?id=36966076 https://news.ycombinator.com/item?id=36966076 NeRF Studio > Included Methods, Third-party Methods: https://docs.nerf.studio/#supported-methods https://docs.nerf.studio/#supported-methods Neural Radiance Field: https://en.wikipedia.org/wiki/Neural_radiance_field https://en.wikipedia.org/wiki/Neural_radiance_field
zyang 3y ago
Why is there a 300m^2 footprint limit if the sub-models are dynamically loaded. Is this constrained by training, rasterizing, or both?
- duckworthd 3y ago
  In terms of the live viewer, there's actually no limit on footprint size. 300 m^2 is simply the biggest indoor capture we had!
nojvek 3y ago
Holy mother of god. Wow! Either matterport takes and runs with this or this is a startup waiting to disrupt Realestate. I can’t believe how smooth this ran on my smartphone. Feedback: if there was a mode to use the phone compass and gyro for navigation, it’d feel natural. Felt weird to navigate with fingers and figure how to move in xyz dimension. As others have said, VR mode would be epic.
- tobr 3y ago
  Is this really something the real estate market wants though? The point of using styled and meticulously chosen images is to entice people to visit the property in person. I think it’s hard to fall for a home because you saw it through virtual reality.
  - iandanforth 3y ago
    It is, yes. If you browse zillow you'll find many homes have 3D views attached. These are often image-sphere captures that you can painfully move through by clicking. While I agree that full res photos can be more appealing, the user experience with SMERF is so much better it might leave end users with a more positive feeling about a property and thus increase the chances of a sale.
  - hobofan 3y ago
    > I think it’s hard to fall for a home because you saw it through virtual reality. I think if you take this 1-2 steps further and combine this with halucinating already owned furniture, or furniture that matches the prospective buyers taste into the property, this will make it a lot easier to fall for a home.
- duckworthd 3y ago
  Thanks for the feedback! I agree, we could do better with the movement UX. A challenge for another day.
  - nojvek 3y ago
    Since the viewer is on GitHub, I’ll take it for a spin. Are you accepting pull requests?
digdugdirk 3y ago
Can you recommend a good entry point into the theory/math behind these? This is one of those true "wtf, we can do this now?" moments, I'm super curious about how these are generated/created.
- duckworthd 3y ago
  Oof, there's a lot of machinery here. It depends a lot on your academic background. I'd recommend starting with a tutorial on neural radiance fields, aka NeRF, (https://sites.google.com/berkeley.edu/nerf-tutorial/home https://sites.google.com/berkeley.edu/nerf-tutorial/home) and an applied overview of Deep Learning with tools like PyTorch or JAX. This line of work is still "cutting edge" research, so a lot of knowledge hasn't been rolled up into textbook or article form yet.
slalomskiing 3y ago
I wonder since this runs at real time framerate if it would be possible for someone to composite a regular rasterized frame on top of something like this (with correct depth testing) to make a game For example a 3rd person game where the character you control and the NPCs/enemies is raster but the environment is all radiance fields
- jasonwatkinspdx 3y ago
  If you go to one of the demos the space bar will cycle through some debug modes. One shows a surface reconstruction. It comes from the usual structure from motion techniques I presume, so it's coarse and noisy, but I think the fundamental idea is viable.
- duckworthd 3y ago
  This should absolutely be possible! The hard part is making it look natural: NeRF models (including SMERF) have no explicit materials or lighting. That means that any character inserted into the game will look out of place.
  - vanderZwan 3y ago
    Why bother to make it look natural when you can have a really awkward greenscreen-like effect for nostalgic and "artistic" purposes?

630 points by duckworthd 3y ago

azinman2 3y ago

Will there be any notebooks or other code released to train our own models?

duckworthd 3y ago
I hope so, but it'll be a good while before we can release anything. We have tight dependencies to other not-yet-OSS libraries, and until they're released, our won't work either.

asgerhb 3y ago

Wow! What am I even looking at here? Polygons, voxels, or something else entirely? How were the benchmarks recorded?

duckworthd 3y ago
You're looking at something called a "neural radiance field" backed by a sparse, low resolution voxel grid and a dense high resolution triplane grid. That's a bit of a word soup, but you can think of it like a glowing fog rendered with ray marching. The benchmark details are a bit complicated. Check out the technical paper's experiment section for the nitty gritty details.
- asgerhb 3y ago
  Hahah I'm in formal methods, not graphics, so I'm picturing a literal green glowing fog. But what I'm gathering is polygons partitioned into cube chunks. This is of course not including the particulars of this impressive contribution, but I fear that's beyond my ken.

RagnarD 3y ago

I'm curious how the creators would compare this to the capabilities of Unreal Engine 5 (as far as the display technology goes.)

mdrzn 3y ago

Impressive is not a big enough statement! This is incredibly smooth on my phone and crazy good on a desktop pc. Keep it up!

duckworthd 3y ago
Thank you :)

jsk95505 3y ago

[flagged]

kolja005 3y ago

Been following Jon Barron et al’s work for a little while now. The speed of improvement given the complexity of these types of systems is mind-boggling. I wonder how long it takes before Google street view gets replaced by some NeRF variant.

boppo1 3y ago

How long until I can make my own?

deleted 3y ago

[deleted]

sim7c00 3y ago

this looks really amazing. i have a relatively old smartphone (2019) and its really surprisingly smooth and high fidently. amazing job!

duckworthd 3y ago
Thank you :). I'm glad to hear it! Which model are you using?
- sim7c00 3y ago
  samsung galaxy 10se

guywithabowtie 3y ago

Any plans to release the models ?

duckworthd 3y ago
The pretrained models are already available online! Check out the "demo" section of the website. Your browser is fetching the model when you run the demo.
- ilaksh 3y ago
  Will the code be released, or an API endpoint? Otherwise it will be impossible for us to use it for anything.. since it's Google I assume it will just end up in a black hole like most of the research.. or five years later some AI researchers leave and finally create a startup.
  - duckworthd 3y ago
    I hope to release code in the new year, but it'll take a while. The codebase is heavily wired into other not-yet-open-sourced libraries, and it'll take a while to disentangle them.
    - ilaksh 3y ago
      That sounds terrific! I really appreciate your effort. It's amazing work and so great of you to share it.

zeusk 3y ago

Are radiance fields related to Gaussian splattering?

duckworthd 3y ago
Gaussian Splatting is heavily inspired by work in radiance fields (or NeRF) models. They use much of the same technology!
corysama 3y ago
Similar inputs, similar outputs, different representation.

joeldg 3y ago

[dead]

aappleby 3y ago

Very impressive demo.

duckworthd 3y ago
Thank you!

refulgentis 3y ago

This is __really__ stunning work, huge, huge, deal that I'm seeing this in a web browser on my phone. Congratulations! When I look at the NYC scene in the highest quality on desktop, I'm surprised by how low-quality ex. the stuff on the counter and shelves is. So then I load the lego model, and see that's _very_ detailed, so it doesn't seem inherent to the method. Is it a consequence of input photo quality, or something else?

duckworthd 3y ago
> This is __really__ stunning work Thank you :) > Is it a consequence of input photo quality, or something else? It's more a consequence of spatial resolution: the bigger the space, the more voxels you need to maintain a fixed resolution (e.g. 1 mm^3). At some point, we have to give up spatial resolution to represent larger scenes. A second limitation is the teacher model we're distilling. Zip-NeRF (https://jonbarron.info/zipnerf/ https://jonbarron.info/zipnerf/) is good, but it's not _perfect_. SMERF reconstruction quality is upper-bounded by its Zip-NeRF teacher.
- deleted 3y ago
  [deleted]

CuriousCosmic 3y ago

Is there a relatively easy way to apply these kinds of techniques (either NeRFs or gaussian splats) to larger environments even if it's lower precision? Like say small towns/a few blocks worth of env.

duckworthd 3y ago
In principle, there's no reason you can't fit multiple City blocks at the same time with Instant NGP on a regular desktop. The challenge is in estimating the camera and lens parameters over such a large space. I expect such a reconstruction to be quite fuzzy given the low space resolution.
ibrarmalik 3y ago
You’re under the right paper for doing this. Instead of one big model, they have several smaller ones for regions in the scene. This way rendering is fast for large scenes. This is similar to Block-NeRF [0], in their project page they show some videos of what you’re asking. As for an easy way of doing this, nothing out-of-the-box. You can keep an eye on nerfstudio [1], and if you feel brave you could implement this paper and make a PR! [0] https://waymo.com/intl/es/research/block-nerf/ https://waymo.com/intl/es/research/block-nerf/ [1] https://github.com/nerfstudio-project/nerfstudio https://github.com/nerfstudio-project/nerfstudio
- CuriousCosmic 3y ago
  Oooh fun. I'm glad it seems possible nowadays. I might take a swing at putting together an out of the box tool at some point if nobody beats me to it first.

barrkel 3y ago

The mirror on the wall of the bathroom in the Berlin location looks through to the kitchen in the next room. I guess the depth gauging algorithm uses parallax, and mirrors confuse it, seeming like windows. The kitchen has a blob of blurriness as the rear of the mirror intrudes into kitchen, but you can see through the blurriness to either room. The effect is a bit spooky. I felt like a ghost going through walls.

rpastuszak 3y ago
Try noclipping through the TV in the Berlin living room. It gets pleasantly creepy.
rzzzt 3y ago
You can also get inside the bookcase for the ultimate Matthew McConaughey experience.
nightpool 3y ago
The refigerator in the NYC scene has a very slick specular lighting effect based on the angle you're viewing it from, and if you go "into" the fridge you can see it's actually generating a whole 3d scene with blurry grey and white colors that turn out to precisely mimic the effects of the light from the windows bouncing off the metal, and you can look "out" from the fridge into the rest of the room. Same as the full-length mirror in the bedroom in the same scene—there's a whole virtual "mirror room" that's been built out behind the mirror to give the illusion of depth as you look through it. Very cool and unique consequence of the technology
- alkonaut 3y ago
  What does the reconstructed space look like when there are opposing mirrors? It’ll just be a long corridor of ever more blurry rooms?
- pjc50 3y ago
  Funnily enough, this is how reflections are usually emulated in game engines that do not support raytracing: another copy of the world behind the mirror. Also used in films in a few places (e.g. Terminator)

promiseofbeans 3y ago

It runs impressively well on my 2yo s21fe. It was super impressive how it streamed in more images as I explored the space. The tv reflections in the Berlin demo were super impressive. My one note is that it look a really long time to load all the images - the scene wouldn't render until all ~40 initial images loaded. Would it be possible to start partially rendering as the images arrive, or do you need to wait for all of them before you can do the first big render?

duckworthd 3y ago
Pardon our dust: "images" is a bad name for what's being loaded. Past versions of this approach (MERF) stored feature vectors in PNG images. We replace them with binary arrays. Unfortunately, all such arrays need to be loaded before the first frame can be rendered. You do however point out one weakness of SMERF: large payload sizes. If we can figure out how to compress them by 10x, it'll be a very different experience!
- promiseofbeans 3y ago
  Or even just breaking them down into smaller chunks (prioritise loading the ones closer to where the user is looking) could help
  - duckworthd 3y ago
    The viewer biases towards assets closer to user's camera (otherwise you'd have to load the whole scene!). We tried training SMERF with a larger number of smaller submodels, but at some point, it becomes too onerous to train and quality begins to suffer.

VikingCoder 3y ago

Wow. Some questions: Take for instance the fulllivingroom demo. (I prefer fps mode.) 1) How many images are input? 2) How long does it take to compute these models? 3) How long does it take to prepare these models for this browser, with all levels, etc? 4) Have you tried this in VR yet?

vyrotek 3y ago
Not exactly what you asked for. But I recently came across this VR example using Gaussian Splatting instead. Exciting times. https://twitter.com/gracia_vr/status/1731731549886787634 https://twitter.com/gracia_vr/status/1731731549886787634 https://www.gracia.ai https://www.gracia.ai
duckworthd 3y ago
Glad you liked our work! 1) Around 100-150 if memory serves. This scene is part of the mip-NeRF 360 benchmark, which you can download from the corresponding project website: https://jonbarron.info/mipnerf360/ https://jonbarron.info/mipnerf360/ 2) Between 12 and 48 hours, depending on the scene. We train on 8x V100s or 16x A100s. 3) The time for preparing assets is included in 2). I don't have a breakdown for you, but it's something like 50/50. 4) Nope! A keen hacker might be able to do this themselves by editing the JavaScript code. Open your browser's DevTools and have a look -- the code is all there!
- dougmwne 3y ago
  Do you need position data to go along with the photos or just the photos? For VR, there’s going to be some very weird depth data from those reflections, but maybe they would not be so bad when you are in headset.
  - duckworthd 3y ago
    > Do you need position data to go along with the photos or just the photos? Short answer: Yes. Long answer: Yes, but it can typically be derived from images. Structure-from-motion methods are typically used to derive lens and position information for each photo in the training set. These are then used by Zip-NeRF (our teacher) and SMERF (our model) to train a model.

durag 3y ago

Any plans to do this in VR? I would love to try this.

duckworthd 3y ago
Not at the moment but an intrepid hacker could surely extend our JavaScript code and put something together. UPDATE: The code for our web viewer is here: https://github.com/smerf-3d/smerf-3d.github.io/blob/main/viewer/index.html https://github.com/smerf-3d/smerf-3d.github.io/blob/main/vie...

deleted 3y ago

[deleted]

deleted 3y ago
[deleted]

blovescoffee 3y ago

Since you're here @author :) Do you mind giving a quick rundown on how this competes with the quality of zip-nerf?

duckworthd 3y ago
Check out our explainer video for answers to this question and more! https://www.youtube.com/watch?v=zhO8iUBpnCc https://www.youtube.com/watch?v=zhO8iUBpnCc

heliophobicdude 3y ago

Great work!! Question for the authors, are there opportunities, where they exist, to not use optimization or tuning methods for reconstructing a model of a scene? We are refining efficient ways of rendering a view of a scene from these models but the scenes remain static. The scenes also take a while to reconstruct too. Can we still achieve the great look and details of RF and GS without paying for an expensive reconstruction per instance of the scene? Are there ways of greedily reconstructing a scene with traditional CG methods into these new representations now that they are fast to render? Please forgive any misconceptions that I may have in advanced! We really appreciate the work y'all are advancing!

duckworthd 3y ago
> Are there opportunities, where they exist, to not use optimization or tuning methods for reconstructing a model of a scene? If you know a way, let me know! Every system I'm aware of involves optimization in one way or another, from COLMAP to 3D Gaussian Splatting to Instant NGP and more. Optimization is a powerful workhorse that gives us a far wider range of models than a direct solver ever could. > Can we still achieve the great look and details of RF and GS without paying for an expensive reconstruction per instance of the scene? In the future I hope so. We don't have a convincing way to generate 3D scenes yet, but given the progress in 2D, I think it's only a matter of time. > Are there ways of greedily reconstructing a scene with traditional CG methods into these new representations now that they are fast to render? Not that I'm aware of! If there were, I think these works should be on the front page instead of SMERF.

annoyingnoob 3y ago

There is a market here for Realtors to upload pictures and produce walk-throughs of homes for sale.

esafak 3y ago
https://matterport.com/ https://matterport.com/
ibrarmalik 3y ago
The Luma folks made something similar: https://apps.apple.com/app/luma-flythroughs/id6450376609?l=en-GB https://apps.apple.com/app/luma-flythroughs/id6450376609?l=e...
- duckworthd 3y ago
  Be careful with this one! Luma's offering requires that the camera follow the recorded video path. Our method lets the camera go wherever you desire!

SubiculumCode 3y ago

Im not sure why this demo runs so horribly in Firefox but not other browsers..anyone else having this?

daemonologist 3y ago
Runs pretty well (20-100 fps depending on the scene) for me on both Firefox 120.1.1 on Android 14 (Pixel 7; smartphone preset) and Firefox 120.0.1 on Fedora 39 (R7 5800, 64 GB memory, RX 6600 XT; 1440p; desktop preset).
- SubiculumCode 3y ago
  It seems that for some reason, my firefox is stuck in software compositor. I am getting: WebRender initialization failed Blocklisted; failure code RcANGLE(no compositor device for EGLDisplay)(Create)_FIRST 3D11_COMPOSITING runtime failed Failed to acquire a D3D11 device Blocklisted; failure code FEATURE_FAILURE_D3D11_DEVICE2 I'm running a 3060
duckworthd 3y ago
We unfortunately haven't tested our web viewer in Firefox. Let us know which platform you're running and we'll do our best to take a look in the new year (holiday vacation!). In the meantime, give it a shot in a Webkit- or Chromium-based browser. I've had good results on Safari on iPhone, Chrome on Android/Macbook/Windows.

jerpint 3y ago

Just ran this on my phone through a browser, this is very impressive

duckworthd 3y ago
Thank you :)

catskul2 3y ago

When might we see this in consumer VR? I'm surprised we don't already but I was suspecting it was a computation constraint. Does this relieve the computation constraint enough to run on Quest 2/3? Is there something else that would prevent binocular use?

doctoboggan 3y ago
I recently got a new quest and I am wondering the same thing. The fact that this is currently running in a browser (and can run on a mobile device) gives me hope that we will see something like this in VR sooner rather than later.
duckworthd 3y ago
I can't predict the future, but I imagine soon: all of the tools are there. The reason we didn't develop for VR is actually simpler than you'd think: we just don't have the developer time! At the end of the day, only a handful of people actively wrote code for this project.
- nojvek 3y ago
  Any plans to open source the code?
  - duckworthd 3y ago
    Yes, I hope so! But it'll take at least a few months of work. We have some tight dependencies to not-yet-open-sourced code, and until that's released, any code we put out will be dead on arrival. In the meantime, feel free to explore the live viewer code! https://github.com/smerf-3d/smerf-3d.github.io/blob/main/viewer/index.html https://github.com/smerf-3d/smerf-3d.github.io/blob/main/vie...

nox100 3y ago

memory efficient? It downloaded 500meg!

bongodongobob 3y ago
A. Storage isn't memory B. That's hardly anything in 2023.
- duckworthd 3y ago
  Right-o. The web viewer is swapping assets in and out of memory as the user explores the scene. The Network and disc requirements are high but memory usage is low.

monlockandkey 3y ago

Get this on a VR headset and you have a game changer literally.

modeless 3y ago

How long until you can stitch Street View into a seamless streaming NeRF of every street in the world? I hope that's the goal you're working towards!

duckworthd 3y ago
;)
- modeless 3y ago
  Haha, too bad the Earth VR team was disbanded because that would be the Holy Grail. If someone can get the budget to work on that I'd be tempted to come back to Google just to help get it done! It's what I always wanted when I was building the first Earth VR demo...
deelowe 3y ago
I read another article talking about what waymo was working on and this looks oddly similar... My understanding is that the goal is to use this to reconstruct 3d models of street view images in real time.
- duckworthd 3y ago
  Block-NeRF is a predecessor work that helped inspire SMERF, in fact! https://waymo.com/research/block-nerf/ https://waymo.com/research/block-nerf/
  - deelowe 3y ago
    Very cool. Thanks!
xnx 3y ago
Not every street yet: https://waymo.com/research/block-nerf/ https://waymo.com/research/block-nerf/

yarg 3y ago

What I'm seeing from all of these things is very accurate single navigable 3D images. What I haven't seen anything of is feature and object detection, blocking and extraction. Hopefully a more efficient and streamable codec necessitates the sort of structure that lends itself more easily to analysis.

yorwba 3y ago
You mean something like this? https://jumpat.github.io/SA3D/ https://jumpat.github.io/SA3D/ Found by putting "nerf sam segment 3d" into DuckDuckGo.
goldenshale 3y ago
Checkout the LERF work from the NerfStudio team at UC Berkeley. SMERF is addressing a different problem, but there are definitely ways to incorporate semantics and detection as well.
duckworthd 3y ago
3D understanding as a field is very much in its infancy. Good work is being done in this area, but we've got a long ways to go yet. SMERF is all about "view synthesis" -- rendering realistic images -- with no attempt at semantic understanding or segmentation.
- cooper_ganglia 3y ago
  "It's my VR-deployed SMERF CLIP model with LLM integration, and I want it now!" It is funny how quickly goalposts move! I love to see progress though, and wow, is progress happening fast!
  - yarg 3y ago
    It's not always moving goalposts - sometimes a new technology progresses on some aspects and regresses in others. This technology is a significant step forward in some ways - but people are going to compare it to state of the art 3D renders and think that it's more impressive than it actually is. Eventually this sort of thing will have understanding of lighting (delumination and light source manipulation) and spatial structure (and eventually spatio-temporal structure). Right now it has none of that, but a layman will look at the output and think that what they're seeing is significantly closer due to largely cosmetic similarities.

fngjdflmdflg 3y ago

>Google DeepMind Google Research Google Inc. What a variety of groups! How did this come about?

duckworthd 3y ago
Collaboration is a thing at the Big G :)

tomatotomato31 3y ago

I'm following this through two minutes paper and I'm looking forward to using it. My grandpa died 2 years ago and in hindsight I took pictures for using them as in your demo. Awesome thanks:)

duckworthd 3y ago
It would be my dream to make capturing 3D memories as easy and natural as taking a 2D photos with your smartphone today. Someday!

twelfthnight 3y ago

Hope this doesn't come as snarky, but does Google pressure researchers to do PR in their papers? This really is cool, but there is a lot of self-promotion in this paper and very little discussion of limitations (and the discussion of them is bookended by qualifications why they really aren't limitations). It makes it harder for me to trust the paper if I feel like the paper is trying to persuade me of something rather than describe the complete findings.

tomatotomato31 3y ago
People are not allowed to be proud of their work anymore?
- twelfthnight 3y ago
  Oh absolutely. I guess I just got the feeling reading this that there was more than the standard pride here and that there was professional PR going on. If no one else is getting that vibe I'm okay to accept it's just me.
duckworthd 3y ago
I won't say too much about this, but the amount of buzz around articles these days is more of "research today" sort of thing. Top conferences like CVPR receives thousands of submissions each year, and there's a lot of upside to getting your work in front of as many eyeballs as possible. By no means do I claim that SMERF is the be-all-end-all in real-time rendering, but I do believe it's a sold step in the right direction. There are all kinds of ways to improve this work and others in the field: smaller representation sizes, faster training, higher quality, and fewer input images would all make this technology more accessible.

yieldcrv 3y ago

I had read about a competing technology that was suggesting NeRF's were a dead end but perhaps that was biased?

duckworthd 3y ago
You're probably thinking of 3D Gaussian Splatting (3DGS), another fantastic approach to real-time novel view synthesis. There's tons of fantastic work being built on 3DGS right now, and the dust has yet to settle with respect to which method is "better". Right now, I can say that SMERF has slightly higher quality on than 3DGS on small scenes and visibly higher quality on big scenes and runs on a wider variety of devices, but takes much longer than 3DGS to train. https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/

rzzzt 3y ago

What kind of modes does the viewer cycle through when I press the space key?

duckworthd 3y ago
Nice discovery :). Check the developer console: it'll tell you.

smusamashah 3y ago

This is very impressive but given its by Google, will some code ever be released?

duckworthd 3y ago
I hope to release the code in the new year, but we have some big dependencies that need to be released worse. In the meantime, you can already begin hacking on the live viewer, https://github.com/smerf-3d/smerf-3d.github.io/blob/main/viewer/index.html https://github.com/smerf-3d/smerf-3d.github.io/blob/main/vie...

cubefox 3y ago

Very impressive! Any information on how this compares to 3D Gaussian splatting in terms of performance, quality or data size?

duckworthd 3y ago
All these details and more in our technical paper! In short: SMERF training takes much longer, SMERF rendering is nearly as fast as 3DGS when a CUDA GPU is available, and quality is visibly higher than 3DGS on large scenes and slightly higher on smaller scenes. https://arxiv.org/abs/2312.07541 https://arxiv.org/abs/2312.07541
- zyang 3y ago
  Is it possible to use zip-nerf to train GS to eliminate the floaters.
  - duckworthd 3y ago
    Maybe! That's the seed of a completely different research paper :)

xnx 3y ago

Does the an open source toolchain exist for capturing, processing, and hosting navigable 3D walkthroughs like this (e.g. something like an open-source Matterport)?

duckworthd 3y ago
Not yet, as far as I'm aware. The current flow involves a DSLR for capture, COLMAP for camera parameter estimation, one codebase for training a teacher model, our codebase for training SMERF, and our web viewer for rendering models. Sounds like an opportunity!
- strofocles 3y ago
  Is there a significant advantage for capturing using DSLRs vs using the phone camera of a decent phone?
  - duckworthd 3y ago
    The big difference is access to fisheye lenses a burst mode that can be run for minutes at a time, and the ability to minimize the amount of camera post processing. In principle, the capture could be done with a smartphone, but the experience of doing so is pretty time consuming right now.
gorkish 3y ago
You don't need a toolchain for capturing; you just need the data. Get it now; process it when better tools become available. There are guides for shooting for Photogrammetry and NeRF that are generally applicable to what you need to do.

germandiago 3y ago

Amazing, impressive, almost unbelievable :O

duckworthd 3y ago
Thank you!

varelse 3y ago

[dead]

edrxty 3y ago

Is there any relation between this class of rendering techniques and the way the BD scenes in Cyberpunk 2077 were created? The behavior of the volume and the "voxels" seem eerily similar.

9dev 3y ago
I doubt Cyberpunk uses more than a special shader for the BD sequences, but what’s a lot more remarkable to me is how similar the idea is at heart. Maybe we’re actually going to see this (maybe sans the brain-implant to record them, but hey) after all. Amazing technology, that’s for sure.
duckworthd 3y ago
I can't say. I'm not familiar with BD in Cyberpunk.
- edrxty 3y ago
  https://youtu.be/KXXGS3MGCro?t=118 https://youtu.be/KXXGS3MGCro?t=118 It's a sort of replayable cutscene that happens a couple times in the game where you can wander through it. The noteworthy bit is it's rendered out of voxels that look very similar to the demos but at a much lower resolution and if you push the frustrum into any objects, you get the same kind of effect where the surface breaks into blocks.
  - duckworthd 3y ago
    Interesting effect. It does look very voxel-y. I'm not a video game developer at heart, so I can only guess how it was implemented. I doubt NeRF models were involved, but I wouldn't be surprised if some sort of voxel discretization was.
    - promiseofbeans 3y ago
      It seems like it might even just be some kind of shader

westurner 3y ago

"Researchers create open-source platform for Neural Radiance Field development" (2023) https://news.ycombinator.com/item?id=36966076 https://news.ycombinator.com/item?id=36966076 NeRF Studio > Included Methods, Third-party Methods: https://docs.nerf.studio/#supported-methods https://docs.nerf.studio/#supported-methods Neural Radiance Field: https://en.wikipedia.org/wiki/Neural_radiance_field https://en.wikipedia.org/wiki/Neural_radiance_field

zyang 3y ago

Why is there a 300m^2 footprint limit if the sub-models are dynamically loaded. Is this constrained by training, rasterizing, or both?

duckworthd 3y ago
In terms of the live viewer, there's actually no limit on footprint size. 300 m^2 is simply the biggest indoor capture we had!

nojvek 3y ago

Holy mother of god. Wow! Either matterport takes and runs with this or this is a startup waiting to disrupt Realestate. I can’t believe how smooth this ran on my smartphone. Feedback: if there was a mode to use the phone compass and gyro for navigation, it’d feel natural. Felt weird to navigate with fingers and figure how to move in xyz dimension. As others have said, VR mode would be epic.

tobr 3y ago
Is this really something the real estate market wants though? The point of using styled and meticulously chosen images is to entice people to visit the property in person. I think it’s hard to fall for a home because you saw it through virtual reality.
- iandanforth 3y ago
  It is, yes. If you browse zillow you'll find many homes have 3D views attached. These are often image-sphere captures that you can painfully move through by clicking. While I agree that full res photos can be more appealing, the user experience with SMERF is so much better it might leave end users with a more positive feeling about a property and thus increase the chances of a sale.
- hobofan 3y ago
  > I think it’s hard to fall for a home because you saw it through virtual reality. I think if you take this 1-2 steps further and combine this with halucinating already owned furniture, or furniture that matches the prospective buyers taste into the property, this will make it a lot easier to fall for a home.
duckworthd 3y ago
Thanks for the feedback! I agree, we could do better with the movement UX. A challenge for another day.
- nojvek 3y ago
  Since the viewer is on GitHub, I’ll take it for a spin. Are you accepting pull requests?

digdugdirk 3y ago

Can you recommend a good entry point into the theory/math behind these? This is one of those true "wtf, we can do this now?" moments, I'm super curious about how these are generated/created.

duckworthd 3y ago
Oof, there's a lot of machinery here. It depends a lot on your academic background. I'd recommend starting with a tutorial on neural radiance fields, aka NeRF, (https://sites.google.com/berkeley.edu/nerf-tutorial/home https://sites.google.com/berkeley.edu/nerf-tutorial/home) and an applied overview of Deep Learning with tools like PyTorch or JAX. This line of work is still "cutting edge" research, so a lot of knowledge hasn't been rolled up into textbook or article form yet.

slalomskiing 3y ago

I wonder since this runs at real time framerate if it would be possible for someone to composite a regular rasterized frame on top of something like this (with correct depth testing) to make a game For example a 3rd person game where the character you control and the NPCs/enemies is raster but the environment is all radiance fields

jasonwatkinspdx 3y ago
If you go to one of the demos the space bar will cycle through some debug modes. One shows a surface reconstruction. It comes from the usual structure from motion techniques I presume, so it's coarse and noisy, but I think the fundamental idea is viable.
duckworthd 3y ago
This should absolutely be possible! The hard part is making it look natural: NeRF models (including SMERF) have no explicit materials or lighting. That means that any character inserted into the game will look out of place.
- vanderZwan 3y ago
  Why bother to make it look natural when you can have a really awkward greenscreen-like effect for nostalgic and "artistic" purposes?