3 ms·
I meant over a single step of the diffusion. Obviously if you keep adding noise it eventually all becomes noise. My point is the noise process is biased towards
by IIAOPSW 4y ago
I meant over a single step of the diffusion. Obviously if you keep adding noise it eventually all becomes noise. My point is the noise process is biased towards erasing some types of details faster than others. This works well when the length scale of features are proportional to the relevance and noticability, as is often the case (smaller size details usually matter less). But when the fine grain level actually has some sort of geometric patterning (such as the white lacing on the playing cards), stable diffusion can't even see it after the first steps of diffusion. It takes a lot more steps for the blob of red card overall to be eliminated than for the fine white lines.
Its exactly the same sort of problem as in the dithering example. An original image of 2-pixel-thin stripes is very recognizable to us as a "thin striped object". Trying to color it in with random dithering will render it unrecognizable. However a much wider 10-pixel-stripe pattern can be safely colored with random dithering.
The commonality is long range ordering of short range features. Thats what the SD process struggles with.
- GaggiX 4y agoYou should read https://arxiv.org/abs/2301.10972 https://arxiv.org/abs/2301.10972
- IIAOPSW 4y agoThanks for that. I'm reading it now. Just off the abstract "when increasing the image size, the optimal noise scheduling shifts towards a noisier one (due to increased redundancy in pixels)" That's highly related to what I'm saying. Its a statement that may be true in the vast majority of cases, and thus would certainly be validated as true-on-average by whatever benchmark, but there's certain types of patterns that aren't redundant among nearby pixels despite being macroscopically visible. A similar thing happens with compression artifacts. For pictures of normal environments and objects jpeg works great. Save a 1 pixel wide line as a jpeg and it is surrounded by artifacts. jpeg compression is also largely built on the assumption that high frequency components in large images won't contain anything important to the image content. In both cases, the result is a visible digital artifact with a "tripy" patterning effect.
- IIAOPSW 4y agohere's an article that happens to echo a lot of what I've said, albeit with a different manifested example. https://www.crosslabs.org/blog/diffusion-with-offset-noise https://www.crosslabs.org/blog/diffusion-with-offset-noise