4 ms·

> using the latest GPUs and machine learning techniques I have no expertise in either graphics or machine learning, but is this just buzzword for its own sake,

> using the latest GPUs and machine learning techniques I have no expertise in either graphics or machine learning, but is this just buzzword for its own sake, or is there actually any way that machine learning is applicable to a mathematical problem like rendering? Again, from my position of relative ignorance, it seems like graphics is largely a well defined problem space that needs mathematical ingenuity and bigger/better hardware thrown at it, rather than building a black box function by training it on sample inputs and outputs (what would that even look like?)

Asooka 8y ago
They're using Nvidia's denoiser which is based on a pretrained neural network. Edit: look up "optix denoiser"
Shikadi 8y ago
To my knowledge, it's a buzzword. That being said, I'm also not an expert, and would like to hear from one to be sure, since it would be interesting af if they actually are using machine learning for Ray tracing
meheleventyone 8y ago
For raytracing it's been used to de-noise images: https://blogs.nvidia.com/blog/2017/05/10/ai-for-ray-tracing/ https://blogs.nvidia.com/blog/2017/05/10/ai-for-ray-tracing/
- jessewmc 8y ago
  TIL, thanks. That's a more prosaic answer than I was expecting but clever and makes sense.
- John_KZ 8y ago
  If you look closely, at around 0:22 it causes some artifacts in the rifle's red light matrix thingy. It could be regular compression artifacts, but they look like denoising artifacts to me.
  - faragon 8y ago
    Could it be just an intentional focus effect?
- amelius 8y ago
  How "local" are such techniques? Is the DL network doing things like "this is a car, and this is a patch of it, so it must look smooth"? Or is it more like "this patch is noisy, so let's make it smooth and hope for the best"?
  - haldean 8y ago
    More on the local end. Traditional denoising is edge finding + blur; you find the edges that you want to keep crisp and you blur on both sides of the edges. ML has been really good at somehow replicating and improving upon that result without explicitly building any sort of understanding about what it is that it's denoising.
moultano 8y ago
I don't think this is what they are actually using, but path tracing can be reduced to a really huge sparse sampling problem. This means that given any predictive model of how much paths through the scene will add light to a pixel, you can make that model unbiased by using https://en.wikipedia.org/wiki/Importance_sampling https://en.wikipedia.org/wiki/Importance_sampling So a model trained to take the samples you have so far and predict paths that will also connect eye to light sources could hugely speed up rendering. I don't know of any software that does this, so free idea if someone wants to try it.
- pixel_fcker 8y ago
  there was a paper recently caching sparse radiance results and using ML to predict important paths from that. I believe the latest version of Octane might do something like that if you read between the lines of their marketing guff.
VladimirGolovin 8y ago
Their current raytracer is limited by the available compute power, so it's forced to use as few samples per pixel as possible, which results in a noisy picture. However, there's a synergy between their on-chip ML cores and their raytracing engine: they use ML cores (which are typically useless in graphic workloads) to de-noise ray-traced images.
Sjenk 8y ago
Take a look at the youtube channel 2minutepapers [0], it covers various ML/DL papers and their results. The person creating the videos has a background in writing rendering software. He has various videos covering papers that use ML to improve rendering and graphics. Not long ago he created a own paper about teaching an AI the concept of metallic, translucent materials and more [1]. I really recommend checking his channel, videos are short and fun and easy to understand even without ML background. You can really sense his passion and excitement for this topic. [0] https://www.youtube.com/user/keeroyz https://www.youtube.com/user/keeroyz [1] https://www.youtube.com/watch?v=6FzVhIV_t3s https://www.youtube.com/watch?v=6FzVhIV_t3s

Asooka 8y ago

They're using Nvidia's denoiser which is based on a pretrained neural network. Edit: look up "optix denoiser"

Shikadi 8y ago

To my knowledge, it's a buzzword. That being said, I'm also not an expert, and would like to hear from one to be sure, since it would be interesting af if they actually are using machine learning for Ray tracing

meheleventyone 8y ago

For raytracing it's been used to de-noise images: https://blogs.nvidia.com/blog/2017/05/10/ai-for-ray-tracing/ https://blogs.nvidia.com/blog/2017/05/10/ai-for-ray-tracing/

jessewmc 8y ago
TIL, thanks. That's a more prosaic answer than I was expecting but clever and makes sense.
John_KZ 8y ago
If you look closely, at around 0:22 it causes some artifacts in the rifle's red light matrix thingy. It could be regular compression artifacts, but they look like denoising artifacts to me.
- faragon 8y ago
  Could it be just an intentional focus effect?
amelius 8y ago
How "local" are such techniques? Is the DL network doing things like "this is a car, and this is a patch of it, so it must look smooth"? Or is it more like "this patch is noisy, so let's make it smooth and hope for the best"?
- haldean 8y ago
  More on the local end. Traditional denoising is edge finding + blur; you find the edges that you want to keep crisp and you blur on both sides of the edges. ML has been really good at somehow replicating and improving upon that result without explicitly building any sort of understanding about what it is that it's denoising.

moultano 8y ago

I don't think this is what they are actually using, but path tracing can be reduced to a really huge sparse sampling problem. This means that given any predictive model of how much paths through the scene will add light to a pixel, you can make that model unbiased by using https://en.wikipedia.org/wiki/Importance_sampling https://en.wikipedia.org/wiki/Importance_sampling So a model trained to take the samples you have so far and predict paths that will also connect eye to light sources could hugely speed up rendering. I don't know of any software that does this, so free idea if someone wants to try it.

pixel_fcker 8y ago
there was a paper recently caching sparse radiance results and using ML to predict important paths from that. I believe the latest version of Octane might do something like that if you read between the lines of their marketing guff.

VladimirGolovin 8y ago

Their current raytracer is limited by the available compute power, so it's forced to use as few samples per pixel as possible, which results in a noisy picture. However, there's a synergy between their on-chip ML cores and their raytracing engine: they use ML cores (which are typically useless in graphic workloads) to de-noise ray-traced images.

Sjenk 8y ago

Take a look at the youtube channel 2minutepapers [0], it covers various ML/DL papers and their results. The person creating the videos has a background in writing rendering software. He has various videos covering papers that use ML to improve rendering and graphics. Not long ago he created a own paper about teaching an AI the concept of metallic, translucent materials and more [1]. I really recommend checking his channel, videos are short and fun and easy to understand even without ML background. You can really sense his passion and excitement for this topic. [0] https://www.youtube.com/user/keeroyz https://www.youtube.com/user/keeroyz [1] https://www.youtube.com/watch?v=6FzVhIV_t3s https://www.youtube.com/watch?v=6FzVhIV_t3s