4 ms·

A question for curiosity. Why can't it train on 256x256 pixels yet generate any size image? So if it was trained on multiple sizes of images, could you also gen

A question for curiosity. Why can't it train on 256x256 pixels yet generate any size image? So if it was trained on multiple sizes of images, could you also generate of a larger size without upscaling?

wincy 3y ago
As someone who doesn’t know the “why” but uses Stable diffusion a lot and has an intuitive feel for the “what” of what happens, it’s like trying to use a low res pattern for your wallpaper on Windows. It’ll either just repeat over and over so you’ll end up with weird multi headed people with heads on top of their heads, or you just upscale which hallucinates details in a totally different way that doesn’t really add new interesting details. With automatic1111 you can get around this by upscaling then inpainting the spots you want more detail and specifying a specific prompt for that particular area.
kmeisthax 3y ago
Well, it's complicated. The model can't just work on arbitrary image sizes because the model was trained with a fixed number of input and output neurons. For example, 512x512 is Stable Diffusion's "native size." However, there are tricks to work around this. Diffusion models work by predicting image noise, which is then subtracted from the image iteratively until you get a result that matches the prompt. Stable Diffusion specifically has the following architectural features: - A Variational Autoencoder (VAE) layer that encodes the 512x512 input into a 128x128 latent space[0] - Three cross-attention blocks that take the encoded text prompt and input latent-space image, and output a downscaled image to the next layer - A simpler downscaling block that just has a linear and convolutional layer - Skip connections between the last four downscaling blocks and corresponding upscaling blocks that do the opposite, in the opposite order (e.g. simple upscale, then three cross-attention blocks). - The aforementioned opposite blocks (upscale + cross-attn upscale) - VAE decoder that goes from latent space back to a 512x512 output At the end of this process you get what the combined model thinks is noise in the image according to the prompt you gave it. You then subtract the noise and repeat for a certain number of iterations until done. So obviously, if you wanted a smaller image, you could crop the input and output at each iteration so that the model can only draw in the 'center'. Larger images are a bit trickier, you have to feed the image through in halves and then merge the noise predictions together before subtracting. This of course has limitations: since the model is looking at only half the image, there's nothing to steer the overall process, so it will draw things that look locally coherent but make no sense globally[1]. I suspect - as in, I'm totally guessing here - that we might be able to fix that by also running the diffusion process on a downscaled version of the image and then scaling the noise prediction back up to average with the other outputs. As far as I'm aware no SD frontends do this. But if that worked you could build up a resolution pyramid of models at different sizes taking fragments of the image and working together to denoise the image. If you were training from scratch you could even add scale and position information to the condition vector so the model can learn what image features should exist at what sizes. [0] Think of this like if every pixel of the latent-space image was, instead of RGB, four different channels worth of information about the distribution of pixels in the color-space image. This compresses the image so that the U-Net part of the model can be architecturally simpler - in fact, lots of machine learning research is finding new ways to compress data into a smaller amount of input neurons. [1] Moreso than diffusion models normally do
dplavery92 3y ago
Presumably a transformer model or similar that uses positional encodings for the tokens could do that, but the U-Net decoder here uses a fixed-shape output and learns relationships between tokens (and sizes of image features) based on the positions of those tokens in a fixed-size vector. You could still apply this process convolutionally and slide the entire network around to generate an image that is an arbitrary multiple of the token size, but image content in one area of the image will only be "aware" of image content at a fixed-size neighborhood (e.g. 256x256).
- GaggiX 3y ago
  This is not how it works, there is no sliding window, the model has no restrictions on the W and H dimensions, only on the C dim, so the UNet can actually be used directly on images with different aspect ratio and resolution; the attention layers pay attention to the entire image.
  - dplavery92 3y ago
    From Sections 3 and 4 of the VQGAN paper[1] upon this work is built: "To generate images in the megapixel regime, we ... have to work patch-wise and crop images to restrict the length of [the quantized encoding vector] s to a maximally feasible size during training. To sample images, we then use the transformer in a sliding-window manner as illustrated in Fig.3." ... "The sliding window approach introduced in Sec.3.2 enables image synthesis beyond a resolution of 256×256pixels." From the Paella paper[2]: "Our proposal builds on the two-stage paradigm introduced by Esser et al. and consists of a Vector-quantized Generative Adversarial Network (VQGAN) for projecting the high dimensional images into a lower-dimensional latent space... [w]e use a pretrained VQGAN with an f=4 compression and a base resolution of 256×256×3, mapping the image to a latent resolution of 64×64indices." After training, in describing their token predictor architecture: "Our architecture consists of a U-Net-style encoder-decoder structure based on residual blocks,employing convolutional[sic] and attention in both, the encoder and decoder pathways." U-Net, of course, is a convolutional neural network architecture. [3]. The "down" and "up" encoder/decoder blocks in the Paella code are batch-normed CNN layers. [4] [1] https://arxiv.org/pdf/2012.09841.pdf https://arxiv.org/pdf/2012.09841.pdf [2] https://arxiv.org/pdf/2211.07292.pdf https://arxiv.org/pdf/2211.07292.pdf [3] https://arxiv.org/abs/1505.04597 https://arxiv.org/abs/1505.04597 [4] https://github.com/dome272/Paella/blob/main/src/modules.py#L148 https://github.com/dome272/Paella/blob/main/src/modules.py#L...
dome271 3y ago
Yea you can. It's the same as with any other CNN based model that is not forced to have a specific shape (like transformers if they use specific positional embeddings). You can also look at the blog post where different resolution images are generated https://laion.ai/blog/paella/ https://laion.ai/blog/paella/
- dome271 3y ago
  I also explain more in the video I made about Paella (https://youtu.be/zdE1I6kYKYc https://youtu.be/zdE1I6kYKYc). Maybe that clarifies things more.
- dplavery92 3y ago
  Transformers are not forced to use a specific input (or output) shape; the original ViT paper demonstrates interpolating positional embeddings to inference with arbitrary image shapes.

wincy 3y ago

As someone who doesn’t know the “why” but uses Stable diffusion a lot and has an intuitive feel for the “what” of what happens, it’s like trying to use a low res pattern for your wallpaper on Windows. It’ll either just repeat over and over so you’ll end up with weird multi headed people with heads on top of their heads, or you just upscale which hallucinates details in a totally different way that doesn’t really add new interesting details. With automatic1111 you can get around this by upscaling then inpainting the spots you want more detail and specifying a specific prompt for that particular area.

kmeisthax 3y ago

Well, it's complicated. The model can't just work on arbitrary image sizes because the model was trained with a fixed number of input and output neurons. For example, 512x512 is Stable Diffusion's "native size." However, there are tricks to work around this. Diffusion models work by predicting image noise, which is then subtracted from the image iteratively until you get a result that matches the prompt. Stable Diffusion specifically has the following architectural features: - A Variational Autoencoder (VAE) layer that encodes the 512x512 input into a 128x128 latent space[0] - Three cross-attention blocks that take the encoded text prompt and input latent-space image, and output a downscaled image to the next layer - A simpler downscaling block that just has a linear and convolutional layer - Skip connections between the last four downscaling blocks and corresponding upscaling blocks that do the opposite, in the opposite order (e.g. simple upscale, then three cross-attention blocks). - The aforementioned opposite blocks (upscale + cross-attn upscale) - VAE decoder that goes from latent space back to a 512x512 output At the end of this process you get what the combined model thinks is noise in the image according to the prompt you gave it. You then subtract the noise and repeat for a certain number of iterations until done. So obviously, if you wanted a smaller image, you could crop the input and output at each iteration so that the model can only draw in the 'center'. Larger images are a bit trickier, you have to feed the image through in halves and then merge the noise predictions together before subtracting. This of course has limitations: since the model is looking at only half the image, there's nothing to steer the overall process, so it will draw things that look locally coherent but make no sense globally[1]. I suspect - as in, I'm totally guessing here - that we might be able to fix that by also running the diffusion process on a downscaled version of the image and then scaling the noise prediction back up to average with the other outputs. As far as I'm aware no SD frontends do this. But if that worked you could build up a resolution pyramid of models at different sizes taking fragments of the image and working together to denoise the image. If you were training from scratch you could even add scale and position information to the condition vector so the model can learn what image features should exist at what sizes. [0] Think of this like if every pixel of the latent-space image was, instead of RGB, four different channels worth of information about the distribution of pixels in the color-space image. This compresses the image so that the U-Net part of the model can be architecturally simpler - in fact, lots of machine learning research is finding new ways to compress data into a smaller amount of input neurons. [1] Moreso than diffusion models normally do

dplavery92 3y ago

Presumably a transformer model or similar that uses positional encodings for the tokens could do that, but the U-Net decoder here uses a fixed-shape output and learns relationships between tokens (and sizes of image features) based on the positions of those tokens in a fixed-size vector. You could still apply this process convolutionally and slide the entire network around to generate an image that is an arbitrary multiple of the token size, but image content in one area of the image will only be "aware" of image content at a fixed-size neighborhood (e.g. 256x256).

GaggiX 3y ago
This is not how it works, there is no sliding window, the model has no restrictions on the W and H dimensions, only on the C dim, so the UNet can actually be used directly on images with different aspect ratio and resolution; the attention layers pay attention to the entire image.
- dplavery92 3y ago
  From Sections 3 and 4 of the VQGAN paper[1] upon this work is built: "To generate images in the megapixel regime, we ... have to work patch-wise and crop images to restrict the length of [the quantized encoding vector] s to a maximally feasible size during training. To sample images, we then use the transformer in a sliding-window manner as illustrated in Fig.3." ... "The sliding window approach introduced in Sec.3.2 enables image synthesis beyond a resolution of 256×256pixels." From the Paella paper[2]: "Our proposal builds on the two-stage paradigm introduced by Esser et al. and consists of a Vector-quantized Generative Adversarial Network (VQGAN) for projecting the high dimensional images into a lower-dimensional latent space... [w]e use a pretrained VQGAN with an f=4 compression and a base resolution of 256×256×3, mapping the image to a latent resolution of 64×64indices." After training, in describing their token predictor architecture: "Our architecture consists of a U-Net-style encoder-decoder structure based on residual blocks,employing convolutional[sic] and attention in both, the encoder and decoder pathways." U-Net, of course, is a convolutional neural network architecture. [3]. The "down" and "up" encoder/decoder blocks in the Paella code are batch-normed CNN layers. [4] [1] https://arxiv.org/pdf/2012.09841.pdf https://arxiv.org/pdf/2012.09841.pdf [2] https://arxiv.org/pdf/2211.07292.pdf https://arxiv.org/pdf/2211.07292.pdf [3] https://arxiv.org/abs/1505.04597 https://arxiv.org/abs/1505.04597 [4] https://github.com/dome272/Paella/blob/main/src/modules.py#L148 https://github.com/dome272/Paella/blob/main/src/modules.py#L...

dome271 3y ago

Yea you can. It's the same as with any other CNN based model that is not forced to have a specific shape (like transformers if they use specific positional embeddings). You can also look at the blog post where different resolution images are generated https://laion.ai/blog/paella/ https://laion.ai/blog/paella/

dome271 3y ago
I also explain more in the video I made about Paella (https://youtu.be/zdE1I6kYKYc https://youtu.be/zdE1I6kYKYc). Maybe that clarifies things more.
dplavery92 3y ago
Transformers are not forced to use a specific input (or output) shape; the original ViT paper demonstrates interpolating positional embeddings to inference with arbitrary image shapes.