4 ms·

To make it really clear, Its not the raw amount of noise or even the noise/signal ratio that I'm pointing to but rather the spectral properties of randomness. I

To make it really clear, Its not the raw amount of noise or even the noise/signal ratio that I'm pointing to but rather the spectral properties of randomness. It matters if you use white noise, pink noise, brown noise, gaussian, uniform etc. These variations in the coarseness of noise pattern will each obscure patterns of the same coarseness. Put another way, if your noise pattern is trees, you won't be able to discern if a forest was there prior to the noise. You'll miss the forest for the trees. You can sort of get out of it by switching away from single pixel uncorrelated noise, but that's hard to think about and hard to code and it all looks random when you test it anyway. I don't know how well if at all the adversary side of the process adjusts the probabilities.

GaggiX 4y ago
I wrote a response yesterday but did not post it or send it, ops. I still don't understand the problem, if you ask model trained on a noise pattern "trees" for a forest it will still give you a random forest, that's what it was trained on, also: https://arxiv.org/abs/2208.09392 https://arxiv.org/abs/2208.09392, to see the diffusion process applied to processes other than Gaussian noise.
- IIAOPSW 4y ago
  That was an illustrative example don't take the trees literally. Here is an actual illustration of dithering which I believe is relevant. https://surma.dev/things/ditherpunk/ https://surma.dev/things/ditherpunk/ Imagine asking Dakke2 for a picture of the bridge through the fog. The problem is an image of fog is graphically indistinguishable from random noise. Whats the difference between a fog patch and something the adversary drew in? Good luck training out of that one. Less tortured understanding: think about my playing cards example. Consider a face down card with the ordinary geometric lacing patterns drawn on the back side. So we are all looking at the same thing: https://www.wopc.co.uk/images/countries/usa/standard/standard-pcc-2.jpg https://www.wopc.co.uk/images/countries/usa/standard/standar... Now think about what happens to that red card as I randomly add white noise static on top. The original fine patterning on the red side is soon distorted beyond recognition because those thin white lines are obscured by equiprobable white dots and coincidental random patterns.
  - GaggiX 4y ago
    >Now think about what happens to that red card as I randomly add white noise static on top. The original fine patterning on the red side is soon distorted beyond recognition because those thin white lines are obscured by equiprobable white dots and coincidental random patterns. I don't see the problem, the pattern will remove the static white noise at the top and generate a new fine patterning on the red side (in case it did not go through the forward process to the end), of course it will not recover the lost information. >Imagine asking Dakke2 for a picture of the bridge through the fog. The problem is an image of fog is graphically indistinguishable from random noise. Whats the difference between a fog patch and something the adversary drew in? Good luck training out of that one. Ignoring the fact that Gaussian noise is really different from a patch of fog, it is obvious that if you have a high noise-to-signal ratio you will not be able to fully recover the signal, during the forward diffusion process all information is lost, however, the model will learn the distribution of samples. Also Dalle 2*
    - IIAOPSW 4y ago
      >I don't see the problem, the pattern will remove the static white noise at the top and generate a new fine patterning on the red side (in case it did not go through the forward process to the end), of course it will not recover the lost information. You said it yourself, it erases information. Pivotally it isn't just a blunt eraser. It erases a very particular grain of detail. The problem is it looks like your on hallucinogenics. https://i.imgur.com/YlPHzgY.png https://i.imgur.com/YlPHzgY.png My hypothesis here is that the distortion manifests when you have fine grain patterns like you would find on a playing card because that is the length scale that is most affected by the noise process. The presence of noise doesn't effect long range objects. The same process would never cause the red card to conmpletely go away. It destroys information about that fine, thin, white line and patterns of a similar characteristic. That's really the key to what I'm saying.
      - GaggiX 4y ago
        >The same process would never cause the red card to conmpletely go away The forward diffusion process completely destroy all information, in fact when the model is trained we simply sample directly from the gaussian distribution instead of actually applying the forward diffusion process on a sample, so the fact that you lose information in the diffusion process is not really a problem because with the diffusion process you lose all information but the model clearly can still generate images.
Jack000 4y ago
The spectral properties of the noise does matter a bit (see the stable diffusion offset noise issue for example) but > These variations in the coarseness of noise pattern will each obscure patterns of the same coarseness. is not accurate. During the forward noise process the high frequency information is obscured first, and the coarse information last. During the reverse process, the neural net learns to denoise the coarse visual elements first, and adds high frequency details at the end. With the original clip guided diffusion notebook you can skip the last 10% of the denoising steps to get a smoother image. This is also the root cause of the noise offset problem (the coarse information is not completely obscured by the forward process)
- IIAOPSW 4y ago
  You're making exactly the same point I am making. I was just substituting the word "coarseness" for "frequency" in an attempt to make it more accessible for people who have never studied this topic. I'm glad someone else sees the frequency space folly of the diffusion process. If I had time, I would test the hypothesis of doing all the learning in frequency space rather than trying to shape the profile of the noise. But I don't have time so feel free to steal my idea.

GaggiX 4y ago

I wrote a response yesterday but did not post it or send it, ops. I still don't understand the problem, if you ask model trained on a noise pattern "trees" for a forest it will still give you a random forest, that's what it was trained on, also: https://arxiv.org/abs/2208.09392 https://arxiv.org/abs/2208.09392, to see the diffusion process applied to processes other than Gaussian noise.

IIAOPSW 4y ago
That was an illustrative example don't take the trees literally. Here is an actual illustration of dithering which I believe is relevant. https://surma.dev/things/ditherpunk/ https://surma.dev/things/ditherpunk/ Imagine asking Dakke2 for a picture of the bridge through the fog. The problem is an image of fog is graphically indistinguishable from random noise. Whats the difference between a fog patch and something the adversary drew in? Good luck training out of that one. Less tortured understanding: think about my playing cards example. Consider a face down card with the ordinary geometric lacing patterns drawn on the back side. So we are all looking at the same thing: https://www.wopc.co.uk/images/countries/usa/standard/standard-pcc-2.jpg https://www.wopc.co.uk/images/countries/usa/standard/standar... Now think about what happens to that red card as I randomly add white noise static on top. The original fine patterning on the red side is soon distorted beyond recognition because those thin white lines are obscured by equiprobable white dots and coincidental random patterns.
- GaggiX 4y ago
  >Now think about what happens to that red card as I randomly add white noise static on top. The original fine patterning on the red side is soon distorted beyond recognition because those thin white lines are obscured by equiprobable white dots and coincidental random patterns. I don't see the problem, the pattern will remove the static white noise at the top and generate a new fine patterning on the red side (in case it did not go through the forward process to the end), of course it will not recover the lost information. >Imagine asking Dakke2 for a picture of the bridge through the fog. The problem is an image of fog is graphically indistinguishable from random noise. Whats the difference between a fog patch and something the adversary drew in? Good luck training out of that one. Ignoring the fact that Gaussian noise is really different from a patch of fog, it is obvious that if you have a high noise-to-signal ratio you will not be able to fully recover the signal, during the forward diffusion process all information is lost, however, the model will learn the distribution of samples. Also Dalle 2*
  - IIAOPSW 4y ago
    >I don't see the problem, the pattern will remove the static white noise at the top and generate a new fine patterning on the red side (in case it did not go through the forward process to the end), of course it will not recover the lost information. You said it yourself, it erases information. Pivotally it isn't just a blunt eraser. It erases a very particular grain of detail. The problem is it looks like your on hallucinogenics. https://i.imgur.com/YlPHzgY.png https://i.imgur.com/YlPHzgY.png My hypothesis here is that the distortion manifests when you have fine grain patterns like you would find on a playing card because that is the length scale that is most affected by the noise process. The presence of noise doesn't effect long range objects. The same process would never cause the red card to conmpletely go away. It destroys information about that fine, thin, white line and patterns of a similar characteristic. That's really the key to what I'm saying.
    - GaggiX 4y ago
      >The same process would never cause the red card to conmpletely go away The forward diffusion process completely destroy all information, in fact when the model is trained we simply sample directly from the gaussian distribution instead of actually applying the forward diffusion process on a sample, so the fact that you lose information in the diffusion process is not really a problem because with the diffusion process you lose all information but the model clearly can still generate images.

Jack000 4y ago

The spectral properties of the noise does matter a bit (see the stable diffusion offset noise issue for example) but > These variations in the coarseness of noise pattern will each obscure patterns of the same coarseness. is not accurate. During the forward noise process the high frequency information is obscured first, and the coarse information last. During the reverse process, the neural net learns to denoise the coarse visual elements first, and adds high frequency details at the end. With the original clip guided diffusion notebook you can skip the last 10% of the denoising steps to get a smoother image. This is also the root cause of the noise offset problem (the coarse information is not completely obscured by the forward process)

IIAOPSW 4y ago
You're making exactly the same point I am making. I was just substituting the word "coarseness" for "frequency" in an attempt to make it more accessible for people who have never studied this topic. I'm glad someone else sees the frequency space folly of the diffusion process. If I had time, I would test the hypothesis of doing all the learning in frequency space rather than trying to shape the profile of the noise. But I don't have time so feel free to steal my idea.