3 ms·

>I don't see the problem, the pattern will remove the static white noise at the top and generate a new fine patterning on the red side (in case it did not go th

>I don't see the problem, the pattern will remove the static white noise at the top and generate a new fine patterning on the red side (in case it did not go through the forward process to the end), of course it will not recover the lost information. You said it yourself, it erases information. Pivotally it isn't just a blunt eraser. It erases a very particular grain of detail. The problem is it looks like your on hallucinogenics. https://i.imgur.com/YlPHzgY.png https://i.imgur.com/YlPHzgY.png My hypothesis here is that the distortion manifests when you have fine grain patterns like you would find on a playing card because that is the length scale that is most affected by the noise process. The presence of noise doesn't effect long range objects. The same process would never cause the red card to conmpletely go away. It destroys information about that fine, thin, white line and patterns of a similar characteristic. That's really the key to what I'm saying.

GaggiX 4y ago
>The same process would never cause the red card to conmpletely go away The forward diffusion process completely destroy all information, in fact when the model is trained we simply sample directly from the gaussian distribution instead of actually applying the forward diffusion process on a sample, so the fact that you lose information in the diffusion process is not really a problem because with the diffusion process you lose all information but the model clearly can still generate images.
- IIAOPSW 4y ago
  I meant over a single step of the diffusion. Obviously if you keep adding noise it eventually all becomes noise. My point is the noise process is biased towards erasing some types of details faster than others. This works well when the length scale of features are proportional to the relevance and noticability, as is often the case (smaller size details usually matter less). But when the fine grain level actually has some sort of geometric patterning (such as the white lacing on the playing cards), stable diffusion can't even see it after the first steps of diffusion. It takes a lot more steps for the blob of red card overall to be eliminated than for the fine white lines. Its exactly the same sort of problem as in the dithering example. An original image of 2-pixel-thin stripes is very recognizable to us as a "thin striped object". Trying to color it in with random dithering will render it unrecognizable. However a much wider 10-pixel-stripe pattern can be safely colored with random dithering. The commonality is long range ordering of short range features. Thats what the SD process struggles with.
  - GaggiX 4y ago
    You should read https://arxiv.org/abs/2301.10972 https://arxiv.org/abs/2301.10972
    - IIAOPSW 4y ago
      Thanks for that. I'm reading it now. Just off the abstract "when increasing the image size, the optimal noise scheduling shifts towards a noisier one (due to increased redundancy in pixels)" That's highly related to what I'm saying. Its a statement that may be true in the vast majority of cases, and thus would certainly be validated as true-on-average by whatever benchmark, but there's certain types of patterns that aren't redundant among nearby pixels despite being macroscopically visible. A similar thing happens with compression artifacts. For pictures of normal environments and objects jpeg works great. Save a 1 pixel wide line as a jpeg and it is surrounded by artifacts. jpeg compression is also largely built on the assumption that high frequency components in large images won't contain anything important to the image content. In both cases, the result is a visible digital artifact with a "tripy" patterning effect.
    - IIAOPSW 4y ago
      here's an article that happens to echo a lot of what I've said, albeit with a different manifested example. https://www.crosslabs.org/blog/diffusion-with-offset-noise https://www.crosslabs.org/blog/diffusion-with-offset-noise

GaggiX 4y ago

>The same process would never cause the red card to conmpletely go away The forward diffusion process completely destroy all information, in fact when the model is trained we simply sample directly from the gaussian distribution instead of actually applying the forward diffusion process on a sample, so the fact that you lose information in the diffusion process is not really a problem because with the diffusion process you lose all information but the model clearly can still generate images.

IIAOPSW 4y ago
I meant over a single step of the diffusion. Obviously if you keep adding noise it eventually all becomes noise. My point is the noise process is biased towards erasing some types of details faster than others. This works well when the length scale of features are proportional to the relevance and noticability, as is often the case (smaller size details usually matter less). But when the fine grain level actually has some sort of geometric patterning (such as the white lacing on the playing cards), stable diffusion can't even see it after the first steps of diffusion. It takes a lot more steps for the blob of red card overall to be eliminated than for the fine white lines. Its exactly the same sort of problem as in the dithering example. An original image of 2-pixel-thin stripes is very recognizable to us as a "thin striped object". Trying to color it in with random dithering will render it unrecognizable. However a much wider 10-pixel-stripe pattern can be safely colored with random dithering. The commonality is long range ordering of short range features. Thats what the SD process struggles with.
- GaggiX 4y ago
  You should read https://arxiv.org/abs/2301.10972 https://arxiv.org/abs/2301.10972
  - IIAOPSW 4y ago
    Thanks for that. I'm reading it now. Just off the abstract "when increasing the image size, the optimal noise scheduling shifts towards a noisier one (due to increased redundancy in pixels)" That's highly related to what I'm saying. Its a statement that may be true in the vast majority of cases, and thus would certainly be validated as true-on-average by whatever benchmark, but there's certain types of patterns that aren't redundant among nearby pixels despite being macroscopically visible. A similar thing happens with compression artifacts. For pictures of normal environments and objects jpeg works great. Save a 1 pixel wide line as a jpeg and it is surrounded by artifacts. jpeg compression is also largely built on the assumption that high frequency components in large images won't contain anything important to the image content. In both cases, the result is a visible digital artifact with a "tripy" patterning effect.
  - IIAOPSW 4y ago
    here's an article that happens to echo a lot of what I've said, albeit with a different manifested example. https://www.crosslabs.org/blog/diffusion-with-offset-noise https://www.crosslabs.org/blog/diffusion-with-offset-noise