3 ms·

> I'm pretty sure that structure from motion + mesh reduction + normal mapping can reconstruct the original images almost pixel-perfect This is actually incorr

> I'm pretty sure that structure from motion + mesh reduction + normal mapping can reconstruct the original images almost pixel-perfect This is actually incorrect, NeRF is the SotA method for these novel view synthesis benchmarks. If you could get better results using meshes, you’d be able to publish a paper and blow all these methods out of the water! That said as you point out there are still major limitations, particularly rendering speed. But the technique is very new and progressing rapidly.

wokwokwok 5y ago
I'm not going to mince words: NeRF is rubbish compared to traditional photogrammetry. > NeRF is the SotA method for these novel view synthesis benchmarks That's an interesting statement; uh, I don't know what 'novel view synthesis benchmarks' you're referring to, but the parent post didn't mention them, and like me, probably doesn't care what they are. If the state of the art is an 800x800 pixel image... uh, well, bluntly, that's really very unimpressive. > Compared to that, I find "AI rendering" which is blurry and much slower (15fps @ 800px) somewhat underwhelming. ^ This. It's very much a 'watch this space' technology, because it does have some really interesting and promising features, and it's changing quickly, but I think finding it 'somewhat underwhelming' is a pretty fair response.
- dTal 5y ago
  No, I'm sorry but this blows photogrammetry out of the water. The original NeRF paper photorealistically handled complex occlusions (like foliage) and reflective and refractive caustics. No other technique comes close. That is the entire reason it's interesting, and believe it or not there are practical applications for it right now. Forget gaming - this lets you capture lightfields for VR with a cell phone in 5 minutes. And if the NeRFs themselves can be rendered in real time, it solves the problem of compressed light field scene representation. Buckle up for photorealistic VR.
  - wokwokwok 5y ago
    > there are practical applications for it right now I'm flat out skeptical you're not just waving your hands in the air vaguely. Provide concrete examples of how it's used right now then, specifically in a scenario where traditional techniques don't work. Not, "look at this youtube video that took 80 hours to render a 800x800 pixel image for our paper"; an actual practical application. I've never seen NeRF deployed in anything other than a proof of concept or toy scenario.
    - dTal 5y ago
      I just gave you one. You can now cheaply and rapidly capture dense lightfields of highly specular objects for VR display. Get yourself a camera array (100 cameras is not infeasible!) and you can capture them instantaneously. That's totally game changing compared to the current state of the art of scanning camera gantries (slow) or photogrammetry (fails on complex or highly specular geometry). If you're asking me for an example of it being publicly used in production, well I think you're asking a lot considering the technique is only a few months old.
      - wokwokwok 5y ago
        > If you're asking me for an example of it being publicly used in production, well I think you're asking a lot considering the technique is only a few months old That is what I explicitly asked for. You’re failure to provide an example is not because it’s new it’s because it’s actually not useful practically at the moment. NeRF has been around since March 2020 (https://arxiv.org/abs/2003.08934); https://arxiv.org/abs/2003.08934); you are simply wrong; traditional techniques are better right now, have better implementations and are widely used. NeRF is a promising technology that is categorically worse in its current implementation and maturity. I don’t know what else to say.
fxtentacle 5y ago
The big fallacy with AI research is that people treat it as "completely new", so in their mind it doesn't make sense to compare the AI to traditional methods. But most traditional methods have also been created by highly advanced intelligences ... us humans. FYI we once got 1st place on the Sintel AI benachmark in "Clean & EPE matched" with a 2004 paper... By now it's down to 10th place, but AI is by no means far ahead of traditional methods. As for "novel view synthesis benchmarks", photogrammetry is used in many hollywood productions for virtual actors and/or for virtual environment destruction. In my opinion, having Hollywood use your technique for billion-dollar blockbuster movies is probably a hint that it works well in practice ;)

wokwokwok 5y ago

I'm not going to mince words: NeRF is rubbish compared to traditional photogrammetry. > NeRF is the SotA method for these novel view synthesis benchmarks That's an interesting statement; uh, I don't know what 'novel view synthesis benchmarks' you're referring to, but the parent post didn't mention them, and like me, probably doesn't care what they are. If the state of the art is an 800x800 pixel image... uh, well, bluntly, that's really very unimpressive. > Compared to that, I find "AI rendering" which is blurry and much slower (15fps @ 800px) somewhat underwhelming. ^ This. It's very much a 'watch this space' technology, because it does have some really interesting and promising features, and it's changing quickly, but I think finding it 'somewhat underwhelming' is a pretty fair response.

dTal 5y ago
No, I'm sorry but this blows photogrammetry out of the water. The original NeRF paper photorealistically handled complex occlusions (like foliage) and reflective and refractive caustics. No other technique comes close. That is the entire reason it's interesting, and believe it or not there are practical applications for it right now. Forget gaming - this lets you capture lightfields for VR with a cell phone in 5 minutes. And if the NeRFs themselves can be rendered in real time, it solves the problem of compressed light field scene representation. Buckle up for photorealistic VR.
- wokwokwok 5y ago
  > there are practical applications for it right now I'm flat out skeptical you're not just waving your hands in the air vaguely. Provide concrete examples of how it's used right now then, specifically in a scenario where traditional techniques don't work. Not, "look at this youtube video that took 80 hours to render a 800x800 pixel image for our paper"; an actual practical application. I've never seen NeRF deployed in anything other than a proof of concept or toy scenario.
  - dTal 5y ago
    I just gave you one. You can now cheaply and rapidly capture dense lightfields of highly specular objects for VR display. Get yourself a camera array (100 cameras is not infeasible!) and you can capture them instantaneously. That's totally game changing compared to the current state of the art of scanning camera gantries (slow) or photogrammetry (fails on complex or highly specular geometry). If you're asking me for an example of it being publicly used in production, well I think you're asking a lot considering the technique is only a few months old.
    - wokwokwok 5y ago
      > If you're asking me for an example of it being publicly used in production, well I think you're asking a lot considering the technique is only a few months old That is what I explicitly asked for. You’re failure to provide an example is not because it’s new it’s because it’s actually not useful practically at the moment. NeRF has been around since March 2020 (https://arxiv.org/abs/2003.08934); https://arxiv.org/abs/2003.08934); you are simply wrong; traditional techniques are better right now, have better implementations and are widely used. NeRF is a promising technology that is categorically worse in its current implementation and maturity. I don’t know what else to say.

fxtentacle 5y ago

The big fallacy with AI research is that people treat it as "completely new", so in their mind it doesn't make sense to compare the AI to traditional methods. But most traditional methods have also been created by highly advanced intelligences ... us humans. FYI we once got 1st place on the Sintel AI benachmark in "Clean & EPE matched" with a 2004 paper... By now it's down to 10th place, but AI is by no means far ahead of traditional methods. As for "novel view synthesis benchmarks", photogrammetry is used in many hollywood productions for virtual actors and/or for virtual environment destruction. In my opinion, having Hollywood use your technique for billion-dollar blockbuster movies is probably a hint that it works well in practice ;)