3 ms·

Denoising is one of those machine learning applications where it just seems to work too well. I was thinking about the idea of using machine learning for denois

Denoising is one of those machine learning applications where it just seems to work too well. I was thinking about the idea of using machine learning for denoising a few weeks ago and then I stumbled on Nvidia’s work in the area and was totally blown away. The strategy Nvidia are adopting for their GPU where they utilise both ray tracing and deep learning cores for real-time applications is excellent.

ladberg 6y ago
I could be wrong, but I feel like deep learning denoising is mostly just blurring in the right places. Deep learning is good at segmenting each part of the image that's supposed to be a different color, and then denoising can be done by blurring within the segmented regions. I'm very curious how a DNN denoiser compares to DNN segmenter + gaussian blur, or even a classical segmenter + blur.
- llukas 6y ago
  See comparisons of DLSS 1.0 vs DLSS 2.0. 1.0 seemed to behave like you describe but 2.0 no longer does... https://www.nvidia.com/en-us/geforce/news/nvidia-dlss-2-0-a-big-leap-in-ai-rendering/ https://www.nvidia.com/en-us/geforce/news/nvidia-dlss-2-0-a-...
  - TomVDB 6y ago
    DLSS is for anti-aliasing and sharpening, not for denoising. And the technique is different than fixed frame ray tracing denoising because DLSS makes use of multiple frames and motion vectors.
    - llukas 6y ago
      If a neural network can build an idea of what it looks at over a few frames (or one frame ie. YOLO) then use it to improve quality of an image is it not denoising? Hallucinating extra pixels definitely. Denoising, maybe, I'd say it fits the bill but I'm not a domain expert.
      - TomVDB 6y ago
        With a static undersampled ray tracing image, the only thing you're doing is hallucinating extra pixels out of nothing but the surrounding pixels that you already have of the same frame. With DLSS, it's more a matter of creatively combining pixels that actually exist. When you're upscaling two 1080p consecutive images + motion vectors to one 1440p image, the amount of source pixels is higher than the final number of pixels. That said, with some config file hacking, people have been able to DLSS upscale from 540p to 1080p (with still surprisingly good results). In that case, the amount of source pixels is still lower than the number of destination pixels. So my point is only semi-valid...
- rustybolt 6y ago
  Inigo Quilez has written a tiny bit on something he calls 'variance-based blur' which seems to do something similar. See the last section of https://www.iquilezles.org/www/articles/wavelet/wavelet.htm https://www.iquilezles.org/www/articles/wavelet/wavelet.htm
- deleted 6y ago
  [deleted]
Fronzie 6y ago
It is my understanding that for ray-tracing de-noising, the de-noiser also has access to the noise-free textures (in addition to the noisy raytraced frame) This gives it far more information than typical screenshots hint at.
thdrdt 6y ago
I believe Blender also uses this denoiser. It is great! Especially for preview renders. The results are comparible to renders which take around 3 times longer.

ladberg 6y ago

I could be wrong, but I feel like deep learning denoising is mostly just blurring in the right places. Deep learning is good at segmenting each part of the image that's supposed to be a different color, and then denoising can be done by blurring within the segmented regions. I'm very curious how a DNN denoiser compares to DNN segmenter + gaussian blur, or even a classical segmenter + blur.

llukas 6y ago
See comparisons of DLSS 1.0 vs DLSS 2.0. 1.0 seemed to behave like you describe but 2.0 no longer does... https://www.nvidia.com/en-us/geforce/news/nvidia-dlss-2-0-a-big-leap-in-ai-rendering/ https://www.nvidia.com/en-us/geforce/news/nvidia-dlss-2-0-a-...
- TomVDB 6y ago
  DLSS is for anti-aliasing and sharpening, not for denoising. And the technique is different than fixed frame ray tracing denoising because DLSS makes use of multiple frames and motion vectors.
  - llukas 6y ago
    If a neural network can build an idea of what it looks at over a few frames (or one frame ie. YOLO) then use it to improve quality of an image is it not denoising? Hallucinating extra pixels definitely. Denoising, maybe, I'd say it fits the bill but I'm not a domain expert.
    - TomVDB 6y ago
      With a static undersampled ray tracing image, the only thing you're doing is hallucinating extra pixels out of nothing but the surrounding pixels that you already have of the same frame. With DLSS, it's more a matter of creatively combining pixels that actually exist. When you're upscaling two 1080p consecutive images + motion vectors to one 1440p image, the amount of source pixels is higher than the final number of pixels. That said, with some config file hacking, people have been able to DLSS upscale from 540p to 1080p (with still surprisingly good results). In that case, the amount of source pixels is still lower than the number of destination pixels. So my point is only semi-valid...
rustybolt 6y ago
Inigo Quilez has written a tiny bit on something he calls 'variance-based blur' which seems to do something similar. See the last section of https://www.iquilezles.org/www/articles/wavelet/wavelet.htm https://www.iquilezles.org/www/articles/wavelet/wavelet.htm
deleted 6y ago
[deleted]

Fronzie 6y ago

It is my understanding that for ray-tracing de-noising, the de-noiser also has access to the noise-free textures (in addition to the noisy raytraced frame) This gives it far more information than typical screenshots hint at.

thdrdt 6y ago

I believe Blender also uses this denoiser. It is great! Especially for preview renders. The results are comparible to renders which take around 3 times longer.