3 ms·

If you tried to repro these results (including time & space constraints) using traditional photogrammetry, you would be sorely disappointed. Photogrammetry is

If you tried to repro these results (including time & space constraints) using traditional photogrammetry, you would be sorely disappointed. Photogrammetry is great if you have a very solid object that is not shiny or translucent at all. You get a lot of surface color micro-detail and a bit of bumpy meso-detail. But, if something is fuzzy, hairy, or lacey or smokey you are straight-up out of luck. Don't even try. If it is shiny, it can be difficult to capture at all --let alone capture the shine. Material capture techniques that are not "chalk sculpture" are rare, very limited and usually experimental. NeRFs however are pretty much a photograph that you can walk around in. They have about as much structure as a photograph ;) But, that lets them not care about the mathematical definition of your scuffed-up, lacquered, iridescent, carbon fiber mirror frame and just show it as it looks from whatever angle.

sorenjan 5y ago
Maybe NeRFs can be used as an intermediate step to reconstruct the scene, and then extract the surfaces and their materials to more conventional representations like meshes, textures, refraction index, etc. I guess the main benefit is that it fills in the undersampled areas in a scene, whether that's an occlusion, a reflection angle, or something else. My main problem with them is that it seems as if all the data is unstructured and interdependent, not like pixels, voxels, or similar where you can clearly extract and manipulate parts of the data and know what it means. To use your photograph example, a digital photo is a simple grid of colored points, and it's easy to change them individually. A regular 3D scene is a collection of well defined vertices, triangles, materials etc, that is then rendered into a digital photo using a easy to describe process. A NeRF on the other hand seems to be more like enter camera pose => magic => inferred image. Maybe I'm overthinking it and it doesn't have to be as general as out current formats, maybe a binary blob that can represent a static scene is fine for plenty of applications. But it feels needlessly complicated.
- corysama 5y ago
  https://www.matthewtancik.com/nerf https://www.matthewtancik.com/nerf is able to sample the 3D volume somehow. "We can also convert the NeRF to a mesh using marching cubes."
CobrastanJorji 5y ago
That's more interesting than I realized. In this example, I assumed that the model was generating some sort of 3D mesh representing the woman. Is that not at all the case? Would this technique be unable to generate a model or volumetric information despite being able to reasonably render her from many directions?
- Mandelmus 5y ago
  No, there is no mesh. A NeRF is a neural network trained to work as a function f(x, y, z, θ, φ). You put in your viewing position (x, y, z) in 3D space and the direction (θ, φ) you're looking into (where θ and φ are the angles for up/down and left/right, respectively), and the function will output a tuple (r, g, b, σ) of the colour (r, g, b) and the material density (σ) of whatever you see at the pixel in that direction and from that position. You can generate a mesh from the density information this function gives you, but for that you need to discretise the continuous densities you get out.
  - ghgr 5y ago
    Minor correction: it's not your (i.e. the camera's) XYZ position what you input, but the position of the point whose RGBA you're trying to render.
amelius 5y ago
Also, the method is lossy as hell. Not something you'd ever use for an intermediate representation.

sorenjan 5y ago

Maybe NeRFs can be used as an intermediate step to reconstruct the scene, and then extract the surfaces and their materials to more conventional representations like meshes, textures, refraction index, etc. I guess the main benefit is that it fills in the undersampled areas in a scene, whether that's an occlusion, a reflection angle, or something else. My main problem with them is that it seems as if all the data is unstructured and interdependent, not like pixels, voxels, or similar where you can clearly extract and manipulate parts of the data and know what it means. To use your photograph example, a digital photo is a simple grid of colored points, and it's easy to change them individually. A regular 3D scene is a collection of well defined vertices, triangles, materials etc, that is then rendered into a digital photo using a easy to describe process. A NeRF on the other hand seems to be more like enter camera pose => magic => inferred image. Maybe I'm overthinking it and it doesn't have to be as general as out current formats, maybe a binary blob that can represent a static scene is fine for plenty of applications. But it feels needlessly complicated.

corysama 5y ago
https://www.matthewtancik.com/nerf https://www.matthewtancik.com/nerf is able to sample the 3D volume somehow. "We can also convert the NeRF to a mesh using marching cubes."

CobrastanJorji 5y ago

That's more interesting than I realized. In this example, I assumed that the model was generating some sort of 3D mesh representing the woman. Is that not at all the case? Would this technique be unable to generate a model or volumetric information despite being able to reasonably render her from many directions?

Mandelmus 5y ago
No, there is no mesh. A NeRF is a neural network trained to work as a function f(x, y, z, θ, φ). You put in your viewing position (x, y, z) in 3D space and the direction (θ, φ) you're looking into (where θ and φ are the angles for up/down and left/right, respectively), and the function will output a tuple (r, g, b, σ) of the colour (r, g, b) and the material density (σ) of whatever you see at the pixel in that direction and from that position. You can generate a mesh from the density information this function gives you, but for that you need to discretise the continuous densities you get out.
- ghgr 5y ago
  Minor correction: it's not your (i.e. the camera's) XYZ position what you input, but the position of the point whose RGBA you're trying to render.

amelius 5y ago

Also, the method is lossy as hell. Not something you'd ever use for an intermediate representation.