4 ms·

I haven't kept up the latest in graphics research. Is the idea that you have a machine-learning system that learns a sort of rough internal representation of t

by elihu 4y ago

I haven't kept up the latest in graphics research. Is the idea that you have a machine-learning system that learns a sort of rough internal representation of the 3D model based on the rendered image (or multiple successive images), and then uses that to help denoise, or interpolate features that are missing from the original dataset? Something I've wondered about is whether these denoising algorithms are always purely used as a last stage of post-processing at the end of the graphics pipeline, or whether they're able to, in effect, go back and say "hey, this area right here I'm pretty uncertain about, so I think we should trace some more rays" and so you might go back and forth between the raytracing engine and the denoising engine a couple times before you get a final image. It seems like the latter could be pretty effective.

jasonwatkinspdx 4y ago
So here's the paper that got everyone fired up: https://www.matthewtancik.com/nerf https://www.matthewtancik.com/nerf My very short summary: Let's model the scene as a function (x,y,z,u,v)->(R,G,B,d). That is for a given ray origin and direction, give us the corresponding radiance and density. Take a deep network, and train it to approximate this function on a scene produced from renderings or video/photos. Now you can use techniques from volumetric rendering (ray marching as used for clouds and such) to render novel views of the scene. I'd really suggest watching the video part way down the page though. These representations do shockingly good at occlusion, which is the difficult problem in rendering. I think this paradigm opens up a lot of interesting possibilities. The elephant in the room is training time, but with good enough compositing of static assets this could still make sense. The trained models are extremely compact. Edit: Oh forgot your second idea. With offline path tracers there are techniques like that. The most elegant is perhaps Metropolis Light Transport, which uses a random walk through the space of possible paths, adjusting step size based on how "interesting" the region its in is. There's also Importance Sampling... lots and lots of ideas in the research literature. But also for cinema rendering a lot of the time it's just "oh we shoot 64 paths per pixel no matter what" because that produces very even noise grain and very good antialiasing. If you use some technique that has temporal biases about noise across sequences of frames it tends to get picked up by the viewer easily.
taylorius 4y ago
The neural representations that I think the parent poster was referring to, are things like NERF - (Neural Radiance Field). These operate on finished images of a scene, and are more akin to a lightfield encoding method than anything to do with ray tracing.
- jasonwatkinspdx 4y ago
  Yup, I made a reply myself but also just to elaborate a bit: I do think this paradigm is relevant to ray/path tracing. There's already work on compositing and relighting them. The parameterization may be interesting in lots of niches, basically anywhere the long training time is justified by having a very compressed representation of the radiance within a scene/instance.
  - taylorius 4y ago
    They are super useful for compositing a 3D object into a scene where there are things close by that will interact with the object for sure. It's like a 5D ray oracle - I think it will transform film CGI pipelines (probably already is). As for scene relighting - that's extra neural witchcraft piled on top! :-)

by elihu 4y ago

jasonwatkinspdx 4y ago

So here's the paper that got everyone fired up: https://www.matthewtancik.com/nerf https://www.matthewtancik.com/nerf My very short summary: Let's model the scene as a function (x,y,z,u,v)->(R,G,B,d). That is for a given ray origin and direction, give us the corresponding radiance and density. Take a deep network, and train it to approximate this function on a scene produced from renderings or video/photos. Now you can use techniques from volumetric rendering (ray marching as used for clouds and such) to render novel views of the scene. I'd really suggest watching the video part way down the page though. These representations do shockingly good at occlusion, which is the difficult problem in rendering. I think this paradigm opens up a lot of interesting possibilities. The elephant in the room is training time, but with good enough compositing of static assets this could still make sense. The trained models are extremely compact. Edit: Oh forgot your second idea. With offline path tracers there are techniques like that. The most elegant is perhaps Metropolis Light Transport, which uses a random walk through the space of possible paths, adjusting step size based on how "interesting" the region its in is. There's also Importance Sampling... lots and lots of ideas in the research literature. But also for cinema rendering a lot of the time it's just "oh we shoot 64 paths per pixel no matter what" because that produces very even noise grain and very good antialiasing. If you use some technique that has temporal biases about noise across sequences of frames it tends to get picked up by the viewer easily.

taylorius 4y ago

The neural representations that I think the parent poster was referring to, are things like NERF - (Neural Radiance Field). These operate on finished images of a scene, and are more akin to a lightfield encoding method than anything to do with ray tracing.

jasonwatkinspdx 4y ago
Yup, I made a reply myself but also just to elaborate a bit: I do think this paradigm is relevant to ray/path tracing. There's already work on compositing and relighting them. The parameterization may be interesting in lots of niches, basically anywhere the long training time is justified by having a very compressed representation of the radiance within a scene/instance.
- taylorius 4y ago
  They are super useful for compositing a 3D object into a scene where there are things close by that will interact with the object for sure. It's like a 5D ray oracle - I think it will transform film CGI pipelines (probably already is). As for scene relighting - that's extra neural witchcraft piled on top! :-)