3 ms·

The magic of this method is that we don't construct a "geometry" the same way one might think. There are no triangles or textures here. Instead, we train a mach

The magic of this method is that we don't construct a "geometry" the same way one might think. There are no triangles or textures here. Instead, we train a machine learning model to predict the derivative of the color and opacity at every point in 3D space. We then integrate along rays emitted from the camera to render an image. It's similar to what's used in CT scans!

heyitsguay 6y ago
That's very cool, but also makes it sound more challenging to integrate into the existing 3D modeling ecosystem vs, say, photogrammetry approaches. Is it possible to generate approximate textured meshes from the color and opacity information? [Edit] After a little Googling I do see this has been done, using marching cubes (https://www.matthewtancik.com/nerf https://www.matthewtancik.com/nerf).
jayd16 6y ago
This means you don't have an occlusion mesh or any other depth information, correct?
- teraflop 6y ago
  There is depth information, just not in the form of a mesh. The model learns to compute a function that takes an XYZ position within a volume as input, and returns color and opacity. You can then render images by tracing rays through this volume. You can pretty easily compute the distance to the first sufficiently-opaque region, or the "average" depth (weighted by each sample's contribution to the final pixel color), at the same time. Another recent Google project figured out a way to approximate these radiance fields with layered, partially transparent images for efficient rendering: https://augmentedperception.github.io/deepviewvideo/ https://augmentedperception.github.io/deepviewvideo/
  - duckworthd 6y ago
    Another related project by our friends in NYC: https://twitter.com/Jimantha/status/1289184432553734144 https://twitter.com/Jimantha/status/1289184432553734144
- duckworthd 6y ago
  We have depth! Check out our depth video: https://youtu.be/yPKIxoN2Vf0?t=146 https://youtu.be/yPKIxoN2Vf0?t=146
schemescape 6y ago
Thanks. So are the trained models for the examples available with code to generate 2D images from them?

heyitsguay 6y ago

That's very cool, but also makes it sound more challenging to integrate into the existing 3D modeling ecosystem vs, say, photogrammetry approaches. Is it possible to generate approximate textured meshes from the color and opacity information? [Edit] After a little Googling I do see this has been done, using marching cubes (https://www.matthewtancik.com/nerf https://www.matthewtancik.com/nerf).

jayd16 6y ago

This means you don't have an occlusion mesh or any other depth information, correct?

teraflop 6y ago
There is depth information, just not in the form of a mesh. The model learns to compute a function that takes an XYZ position within a volume as input, and returns color and opacity. You can then render images by tracing rays through this volume. You can pretty easily compute the distance to the first sufficiently-opaque region, or the "average" depth (weighted by each sample's contribution to the final pixel color), at the same time. Another recent Google project figured out a way to approximate these radiance fields with layered, partially transparent images for efficient rendering: https://augmentedperception.github.io/deepviewvideo/ https://augmentedperception.github.io/deepviewvideo/
- duckworthd 6y ago
  Another related project by our friends in NYC: https://twitter.com/Jimantha/status/1289184432553734144 https://twitter.com/Jimantha/status/1289184432553734144
duckworthd 6y ago
We have depth! Check out our depth video: https://youtu.be/yPKIxoN2Vf0?t=146 https://youtu.be/yPKIxoN2Vf0?t=146

schemescape 6y ago

Thanks. So are the trained models for the examples available with code to generate 2D images from them?