4 ms·

How does denoising work, exactly and why does it seem necessary to get good results?

• ShamelessC 4y ago
  tl;dr it allows the model to focus on different frequencies of pixel-space at different time steps. Time step zero being the low frequency/low-detail features and time step 1000 being a completely “filled out”, detailed image including all the high frequency features. Doing so in discretely many steps makes the problem tractable and differentiable. Gross oversimplification, but that’s the gist.
  • Natsu 4y ago
    Different frequencies? Does that mean it's doing something like an FFT?
    • ShamelessC 4y ago
      Bit of a delayed response - but no (well, not necessarily, anyways). Operating in the frequency domain is a useful technique in machine learning. With this method however, the reason that different "timesteps" amount to different frequencies is because you start with: 0: full gaussian noise 1: less gaussian noise, more generated pixels ... 998: barely any noise, mostly generated pixels 999: no noise. generation complete. It's actually the properties of gaussian noise that allow the network to learn the importance of the frequency domain indirectly. When there is a high frequency of noise, you can expect a low frequency in pixel space. When there is less noise, you can similarly expect the pixel space to now include both low _and_ high frequency details. By the end of denoising, the full range is meant to be learned (ideally, anyway). Having said that - they may also use a conversion to fourier space - I'm aware that at least nvidia does something like this in StyleGAN3.
      • Natsu 4y ago
        Why does the noise being Gaussian matter? Would it even work with other types of noise like Perlin or white noise?