3 ms·

This sort of stuff (generating 3D assets from photographs of real objects) has been common for quite a while via photogrammetry. NeRFs are interesting because (

by jowday 5y ago

This sort of stuff (generating 3D assets from photographs of real objects) has been common for quite a while via photogrammetry. NeRFs are interesting because (in some cases) they can create renders that look higher quality with fewer photos, and they hint at the potential of future learned rendering models.

SV_BubbleTime 5y ago
I did it for a project… it was SLOW! Gave decent rules for a large area, but poor results for small details. The real trick is textures. When the photo is laid over and wrapped on the model, it looks great. But once you remove that, the raw mesh underneath is not as impressive. I would really like to see the examples they had here without the texture laid on top.
- hwers 5y ago
  Given the same amount of compute as this used, photogrammetry would be about as fast.
- hansworst 5y ago
  There is no mesh here. Nerfs are 5d (colours are computed based on a 3D position vector + a view direction vector) fields that are rendered volumetrically. So the “texture” is an integral part of the neural representation of the scene, not just an image applied to a mesh. The cool part is that this also allows for capturing transparency, and any effects caused by lighting (including complex specular reflections) are embedded into the representation.
  - soylentgraham 5y ago
    Nitpicking, but for GP; Nerf is the internal representation, but the output doesnt have to be 2D (ray traced basically) There are examples of people outputting SDF (and by extension geometry) with nerf, and projecting original texture onto that would give some nice effects; (live volumetric works best this way) though there would be some disparity where edges/occlusion isnt perfect, so youd want to sample nerf's rgb anyway... although a lot of that is fuzzy at the edges too. A lot of incorrect transparency at edges looks great in the 2D renders (so much anti aliasing and noise!) but less good for texturing
    - ath92 5y ago
      A NeRF is not the same as an SDF though. NGP (the paper by Nvidia linked here) can train NeRFs and SDFs, but I don't know of any straightforward way of extracting an SDF from a NeRF. And while it's true that there are methods for extracting a surface from a NeRF, achieving a high quality result can be challenging because you have to figure out what to do with regions that have low occupancy (i.e. regions that are translucent). Should you consider those regions as contained within the surface, or outside of it? Especially when dealing with things like hair, it's not obvious how to construct a surface based on a NeRF.

by jowday 5y ago

SV_BubbleTime 5y ago

I did it for a project… it was SLOW! Gave decent rules for a large area, but poor results for small details. The real trick is textures. When the photo is laid over and wrapped on the model, it looks great. But once you remove that, the raw mesh underneath is not as impressive. I would really like to see the examples they had here without the texture laid on top.

hwers 5y ago
Given the same amount of compute as this used, photogrammetry would be about as fast.
hansworst 5y ago
There is no mesh here. Nerfs are 5d (colours are computed based on a 3D position vector + a view direction vector) fields that are rendered volumetrically. So the “texture” is an integral part of the neural representation of the scene, not just an image applied to a mesh. The cool part is that this also allows for capturing transparency, and any effects caused by lighting (including complex specular reflections) are embedded into the representation.
- soylentgraham 5y ago
  Nitpicking, but for GP; Nerf is the internal representation, but the output doesnt have to be 2D (ray traced basically) There are examples of people outputting SDF (and by extension geometry) with nerf, and projecting original texture onto that would give some nice effects; (live volumetric works best this way) though there would be some disparity where edges/occlusion isnt perfect, so youd want to sample nerf's rgb anyway... although a lot of that is fuzzy at the edges too. A lot of incorrect transparency at edges looks great in the 2D renders (so much anti aliasing and noise!) but less good for texturing
  - ath92 5y ago
    A NeRF is not the same as an SDF though. NGP (the paper by Nvidia linked here) can train NeRFs and SDFs, but I don't know of any straightforward way of extracting an SDF from a NeRF. And while it's true that there are methods for extracting a surface from a NeRF, achieving a high quality result can be challenging because you have to figure out what to do with regions that have low occupancy (i.e. regions that are translucent). Should you consider those regions as contained within the surface, or outside of it? Especially when dealing with things like hair, it's not obvious how to construct a surface based on a NeRF.