The spectral properties of the noise does matter a bit (see the stable diffusion offset noise issue for example) but > These variations in the coarseness of no

The spectral properties of the noise does matter a bit (see the stable diffusion offset noise issue for example) but > These variations in the coarseness of noise pattern will each obscure patterns of the same coarseness. is not accurate. During the forward noise process the high frequency information is obscured first, and the coarse information last. During the reverse process, the neural net learns to denoise the coarse visual elements first, and adds high frequency details at the end. With the original clip guided diffusion notebook you can skip the last 10% of the denoising steps to get a smoother image. This is also the root cause of the noise offset problem (the coarse information is not completely obscured by the forward process)