6 ms·

I don't really understand why NeRFs would be particularly useful in more than a few niche cases, perhaps because I don't fully understand what they really are.

I don't really understand why NeRFs would be particularly useful in more than a few niche cases, perhaps because I don't fully understand what they really are. My impression is that you take a bunch of photos in various places and directions, then you use those as samples of a 3D function that describes the full scene, and optimize a neural network to minimize the difference between the true light field and what's described by the network. An approximation of the actual function, that fits the training data. The millions of coefficients are seen as a black box that somehow describes the scene when combined in a certain way, I guess mapping a camera pose to a rendered image? But why would that be better than some other data structure, like a mesh, a point cloud, or signed distance field, where you have the scene as structured data you can reason about? What happens if you want to animate part of a NeRF, or crop it, or change it in any way? Do you have to throw away all trained coefficients and start again from training data? Can you use this method as a part of a more traditional photogrammetry pipeline and extract the result as a regular mesh? Nvidia seems to suggest that NeRFs are in some way better than meshes, but according to my flawed understanding they just seem unwieldy.

p1esk 5y ago
What happens if you want to animate part of a NeRF, or crop it, or change it in any way? Do you have to throw away all trained coefficients and start again from training data? You don’t change NeRF (the model). You change the point of view of an observer.
- wokwokwok 5y ago
  I mean, this is the parent posts point; the use case for a static photo or a static 3d nerf is pretty limited. With other structured data compositing and animating is relatively trivial. It turns out that people have approached this problem before and you can composite nerf too (1) by sampling different functions over the volume. …but, let’s not pretend. The complaint is entirely valid. You’re taking a high resolution voxel grid and encoding it into a model. Working with simple voxel data let’s you do all kinds of normal image manipulation techniques, and it’s not clear how you would do some of those with a nerf. Practically speaking, the applications you can use this for are therefore reasonably limited right now. [1] - https://www.unite.ai/st-nerf-compositing-and-editing-for-video-synthesis/ https://www.unite.ai/st-nerf-compositing-and-editing-for-vid...
  - yorwba 5y ago
    > Working with simple voxel data let’s you do all kinds of normal image manipulation techniques, and it’s not clear how you would do some of those with a nerf. Any image transformation you can do on voxels you can straightforwardly transfer to nerfs. Voxel data is just a lookup table from discrete positions to material properties like color and density. When you apply a transform, you change the inputs (e.g. multiplying them with a rotation matrix) or the outputs (e.g. changing the color). If you want to do the same thing with a nerf that maps continuous positions and directions to material properties like color and density, just transform the inputs or the outputs. The major difference is that with voxel data you can easily do output-modifying transformations directly on the stored representation, while for nerfs it might be cheaper to do it on the fly instead of redoing the training procedure to bake the change into the model.
    - wokwokwok 5y ago
      > Any image transformation you can do on voxels you can straightforwardly transfer to nerfs No. > it might be cheaper to do it on the fly instead of redoing the training procedure to bake the change into the model. I think it’s a bit more complex than you imagine; it’s not “cheaper/not cheaper”; it’s literally the only way of doing it. If you have a transformation f(x) that takes a pixel array as in input and returns a pixel array as an output, that is a trivial transformation. If you have a transformation that takes a vector input f(x) and returns a pixel output, it’s seriously bloody hard to convert it to a “good” vector again. Consider taking a layered svg and applying a box blur. Now you want an svg again. It’s not a trivial problem. Lines blur and merge, you have reconstruct an entirely new svg. Now you add the constraint in 3d; you can never have a full voxel representation in memory even temporarily because of memory constraints. At best you’re looking at applying voxel level transformations on the fly to render specific views, and then retrain those into a new nerf model. I think that counts as … not straightforward. Doing all your transformations on the fly is a lovely idea, but you gotta understand reason nerf exists is that the raw voxel data is too big to store in memory. It’s simply not possible you can dynamically run a image processing pipeline over that volume data in real-time. You have to bake it into nerfs to use it at all.
corysama 5y ago
If you tried to repro these results (including time & space constraints) using traditional photogrammetry, you would be sorely disappointed. Photogrammetry is great if you have a very solid object that is not shiny or translucent at all. You get a lot of surface color micro-detail and a bit of bumpy meso-detail. But, if something is fuzzy, hairy, or lacey or smokey you are straight-up out of luck. Don't even try. If it is shiny, it can be difficult to capture at all --let alone capture the shine. Material capture techniques that are not "chalk sculpture" are rare, very limited and usually experimental. NeRFs however are pretty much a photograph that you can walk around in. They have about as much structure as a photograph ;) But, that lets them not care about the mathematical definition of your scuffed-up, lacquered, iridescent, carbon fiber mirror frame and just show it as it looks from whatever angle.
- sorenjan 5y ago
  Maybe NeRFs can be used as an intermediate step to reconstruct the scene, and then extract the surfaces and their materials to more conventional representations like meshes, textures, refraction index, etc. I guess the main benefit is that it fills in the undersampled areas in a scene, whether that's an occlusion, a reflection angle, or something else. My main problem with them is that it seems as if all the data is unstructured and interdependent, not like pixels, voxels, or similar where you can clearly extract and manipulate parts of the data and know what it means. To use your photograph example, a digital photo is a simple grid of colored points, and it's easy to change them individually. A regular 3D scene is a collection of well defined vertices, triangles, materials etc, that is then rendered into a digital photo using a easy to describe process. A NeRF on the other hand seems to be more like enter camera pose => magic => inferred image. Maybe I'm overthinking it and it doesn't have to be as general as out current formats, maybe a binary blob that can represent a static scene is fine for plenty of applications. But it feels needlessly complicated.
  - corysama 5y ago
    https://www.matthewtancik.com/nerf https://www.matthewtancik.com/nerf is able to sample the 3D volume somehow. "We can also convert the NeRF to a mesh using marching cubes."
deleted 5y ago
[deleted]
fxtentacle 5y ago
The pros in the vfx industry still all use reconstructed geometry. And yes, animating or cropping a Nerf is painful. In my opinion, Nerf is more about showing progress in making AI memorize 3D scenes and the hope is that this will lead to actual understanding sometime in the future.
- deleted 5y ago
  [deleted]
crubier 5y ago
You are right, this is why approaches like plenoxels are vastly faster than nerf. They combine the optimization approach of neural nets, but combine it with a simple and regular data representation of the scene. https://arxiv.org/pdf/2112.05131.pdf https://arxiv.org/pdf/2112.05131.pdf

p1esk 5y ago

What happens if you want to animate part of a NeRF, or crop it, or change it in any way? Do you have to throw away all trained coefficients and start again from training data? You don’t change NeRF (the model). You change the point of view of an observer.

wokwokwok 5y ago
I mean, this is the parent posts point; the use case for a static photo or a static 3d nerf is pretty limited. With other structured data compositing and animating is relatively trivial. It turns out that people have approached this problem before and you can composite nerf too (1) by sampling different functions over the volume. …but, let’s not pretend. The complaint is entirely valid. You’re taking a high resolution voxel grid and encoding it into a model. Working with simple voxel data let’s you do all kinds of normal image manipulation techniques, and it’s not clear how you would do some of those with a nerf. Practically speaking, the applications you can use this for are therefore reasonably limited right now. [1] - https://www.unite.ai/st-nerf-compositing-and-editing-for-video-synthesis/ https://www.unite.ai/st-nerf-compositing-and-editing-for-vid...
- yorwba 5y ago
  > Working with simple voxel data let’s you do all kinds of normal image manipulation techniques, and it’s not clear how you would do some of those with a nerf. Any image transformation you can do on voxels you can straightforwardly transfer to nerfs. Voxel data is just a lookup table from discrete positions to material properties like color and density. When you apply a transform, you change the inputs (e.g. multiplying them with a rotation matrix) or the outputs (e.g. changing the color). If you want to do the same thing with a nerf that maps continuous positions and directions to material properties like color and density, just transform the inputs or the outputs. The major difference is that with voxel data you can easily do output-modifying transformations directly on the stored representation, while for nerfs it might be cheaper to do it on the fly instead of redoing the training procedure to bake the change into the model.
  - wokwokwok 5y ago
    > Any image transformation you can do on voxels you can straightforwardly transfer to nerfs No. > it might be cheaper to do it on the fly instead of redoing the training procedure to bake the change into the model. I think it’s a bit more complex than you imagine; it’s not “cheaper/not cheaper”; it’s literally the only way of doing it. If you have a transformation f(x) that takes a pixel array as in input and returns a pixel array as an output, that is a trivial transformation. If you have a transformation that takes a vector input f(x) and returns a pixel output, it’s seriously bloody hard to convert it to a “good” vector again. Consider taking a layered svg and applying a box blur. Now you want an svg again. It’s not a trivial problem. Lines blur and merge, you have reconstruct an entirely new svg. Now you add the constraint in 3d; you can never have a full voxel representation in memory even temporarily because of memory constraints. At best you’re looking at applying voxel level transformations on the fly to render specific views, and then retrain those into a new nerf model. I think that counts as … not straightforward. Doing all your transformations on the fly is a lovely idea, but you gotta understand reason nerf exists is that the raw voxel data is too big to store in memory. It’s simply not possible you can dynamically run a image processing pipeline over that volume data in real-time. You have to bake it into nerfs to use it at all.

corysama 5y ago

If you tried to repro these results (including time & space constraints) using traditional photogrammetry, you would be sorely disappointed. Photogrammetry is great if you have a very solid object that is not shiny or translucent at all. You get a lot of surface color micro-detail and a bit of bumpy meso-detail. But, if something is fuzzy, hairy, or lacey or smokey you are straight-up out of luck. Don't even try. If it is shiny, it can be difficult to capture at all --let alone capture the shine. Material capture techniques that are not "chalk sculpture" are rare, very limited and usually experimental. NeRFs however are pretty much a photograph that you can walk around in. They have about as much structure as a photograph ;) But, that lets them not care about the mathematical definition of your scuffed-up, lacquered, iridescent, carbon fiber mirror frame and just show it as it looks from whatever angle.

sorenjan 5y ago
Maybe NeRFs can be used as an intermediate step to reconstruct the scene, and then extract the surfaces and their materials to more conventional representations like meshes, textures, refraction index, etc. I guess the main benefit is that it fills in the undersampled areas in a scene, whether that's an occlusion, a reflection angle, or something else. My main problem with them is that it seems as if all the data is unstructured and interdependent, not like pixels, voxels, or similar where you can clearly extract and manipulate parts of the data and know what it means. To use your photograph example, a digital photo is a simple grid of colored points, and it's easy to change them individually. A regular 3D scene is a collection of well defined vertices, triangles, materials etc, that is then rendered into a digital photo using a easy to describe process. A NeRF on the other hand seems to be more like enter camera pose => magic => inferred image. Maybe I'm overthinking it and it doesn't have to be as general as out current formats, maybe a binary blob that can represent a static scene is fine for plenty of applications. But it feels needlessly complicated.
- corysama 5y ago
  https://www.matthewtancik.com/nerf https://www.matthewtancik.com/nerf is able to sample the 3D volume somehow. "We can also convert the NeRF to a mesh using marching cubes."

deleted 5y ago

[deleted]

fxtentacle 5y ago

The pros in the vfx industry still all use reconstructed geometry. And yes, animating or cropping a Nerf is painful. In my opinion, Nerf is more about showing progress in making AI memorize 3D scenes and the hope is that this will lead to actual understanding sometime in the future.

deleted 5y ago
[deleted]

crubier 5y ago

You are right, this is why approaches like plenoxels are vastly faster than nerf. They combine the optimization approach of neural nets, but combine it with a simple and regular data representation of the scene. https://arxiv.org/pdf/2112.05131.pdf https://arxiv.org/pdf/2112.05131.pdf