4 ms·

I've noticed a lot of interest in differential rendering/reverse rendering recently Does anyone know what the end goal of this kind of research is or why there

I've noticed a lot of interest in differential rendering/reverse rendering recently Does anyone know what the end goal of this kind of research is or why there is so much interest? Its definitely cool but is the idea just to make photogrammetry cheaper/easier? Or are there other use cases I'm missing

cperciva 4y ago
One major use case for "convert a bunch of photos into a model" is 3D printing. Often you have an object -- maybe a part which broke and needs to be replaced, maybe a tooth surface which you're making a crown for -- and you need to turn the object into a model and then create a new object.
soylentgraham 4y ago
I see it as a (potential) storage format. As we move away from triangles and towards ray/path/etc tracing (in whatever forms) we want to still have content (be it captured or crafted) in it, that we can render with things like "infinite" details, bounces, and other meta (reflectancr etc). All current forms of triangulation/meshing from fractcals, sdf, point clouds, etc, are relatively terrible and constrained, to fit existing pipelines (hence why most AR is just lackluster model viewers) Make it differentiable is a step towards cramming it into a small model and output other forms, or trace directly, or just compress before say, extracting to some Tree-renderable format on the gpu
- djmips 4y ago
  I did see that in the last NERF paper I scanned. The final representation was smaller than the sparse inputs by a large factor. It got me to thinking along the same lines that this could be like the jpeg/png for 3D scenes online.
gfodor 4y ago
There's an incredibly interesting track I've been following trying to find a good latent embedding for AI to generate 3d objects, and SDFs seem like a very good representational format for this. This would, among other things, allow for a DALL-E 2 like generation framework for 3d. Google DeepSDF.
berkut 4y ago
There are other things it's been used for like optimising roughening/regularisation parameters for light transport in order to provide better caustics with forwards path tracing: https://diglib.eg.org/handle/10.1111/cgf14347 https://diglib.eg.org/handle/10.1111/cgf14347

cperciva 4y ago

One major use case for "convert a bunch of photos into a model" is 3D printing. Often you have an object -- maybe a part which broke and needs to be replaced, maybe a tooth surface which you're making a crown for -- and you need to turn the object into a model and then create a new object.

soylentgraham 4y ago

I see it as a (potential) storage format. As we move away from triangles and towards ray/path/etc tracing (in whatever forms) we want to still have content (be it captured or crafted) in it, that we can render with things like "infinite" details, bounces, and other meta (reflectancr etc). All current forms of triangulation/meshing from fractcals, sdf, point clouds, etc, are relatively terrible and constrained, to fit existing pipelines (hence why most AR is just lackluster model viewers) Make it differentiable is a step towards cramming it into a small model and output other forms, or trace directly, or just compress before say, extracting to some Tree-renderable format on the gpu

djmips 4y ago
I did see that in the last NERF paper I scanned. The final representation was smaller than the sparse inputs by a large factor. It got me to thinking along the same lines that this could be like the jpeg/png for 3D scenes online.

gfodor 4y ago

There's an incredibly interesting track I've been following trying to find a good latent embedding for AI to generate 3d objects, and SDFs seem like a very good representational format for this. This would, among other things, allow for a DALL-E 2 like generation framework for 3d. Google DeepSDF.

berkut 4y ago

There are other things it's been used for like optimising roughening/regularisation parameters for light transport in order to provide better caustics with forwards path tracing: https://diglib.eg.org/handle/10.1111/cgf14347 https://diglib.eg.org/handle/10.1111/cgf14347