7 ms·

Stable Diffusion XL technical report [pdf]

201 points by GaggiX 3y ago

in_the_bay 3y ago
Does it say anywhere what hardware they used for training?
- rhogar 3y ago
  The report does not detail hardware -- though it states that SDXL has 2.6B parameters in its UNet component, compared to SD 1.4/1.5 with 860M and SD 2.0/2.1 with 865M. So SDXL has roughly 3x more UNet parameters. In January, MosaicML claimed a model comparable to Stable Diffusion V2 could be trained with 79,000 A100-hours in 13 days. Some sort of inference can be made from this information, would be interested to hear someone with more insight here provide more perspective.
  - darqwolff 3y ago
    wouldnt that mean more vram is required to load the model? they are claiming it will still work on 8 gb cards.
    - Filligree 3y ago
      Stable Diffusion 1/2 were made to run on cards with as little as 3GB of memory. Using the same techniques, yes, this will fit in 8.
    - brucethemoose2 3y ago
      I am guessing 8 bit quantization will be a thing for SDXL. It should be easy(TM) with bitsandbytes, or ML compiler frameworks.
      - GaggiX 3y ago
        bitsandbytes is only used during training with these models tho (the 8-bit Adamw) quantizing the weights and the activations to a range of 256 values when the model needs to output a range 256 values creates noticeable artifacts as they are not going to map 1-to-1.
        liuliu 3y ago
        Draw Things recently released a 8-bit quantized SD model that has comparable output as the FP16. It does use k-means based LUT and separate weights into blocks to minimize quantization errors.
        GaggiX 3y ago
        I was going to search on the internet about it, but then I realized you are the author (and there is nothing online I think). I imagine that the activations are left in FP16 and the weights are converted in FP16 during inference, right? Btw very cool
        liuliu 3y ago
        Yes, computes are carried out in FP16 (so there is no compute efficiency gains, might be latency reductions due to memory-bandwidth saving). These savings are not realized yet because no custom kernels introduced yet.
p1esk 3y ago
“Images generated with our code use the invisible-watermark library to embed an invisible watermark into the model output. We also provide a script to easily detect that watermark.” Interesting - if it can be detected then it can be removed, right?
- menzoic 3y ago
  It's open source. It can be removed directly from the code.
  - donkeyd 3y ago
    Also, most people use other tools like web-ui to run these models, which don't seem to apply any watermarks.
  - p1esk 3y ago
    Sure, I just don’t understand why they’re doing it. In theory they could also train the model to always add a watermark.
    - aenvoker 3y ago
      Stability, Midjourney, Dall-e, etc don’t want their products to be used for misinformation. They really do want AI generated images to be trivially identifiable. There are papers around describing how to bake a watermark into a model. Don’t know if anyone is doing that yet.
      - Der_Einzige 3y ago
        And many will start working on techniques to unbake it from the model. A cat and mouse game, like all other attempts to track.
        wongarsu 3y ago
        Sure, but that can be said about any security measure. If we agree that passing AI generated art as human-made or as a photograph is bad, then making that harder to do is worthwhile.
naillo 3y ago
I love stability AI
chaos_emergent 3y ago
how refreshing it is to see technical details in a technical report /s
Jack000 3y ago
Not seeing anything about the dataset, are they still using LAION? There's no mention of LAION in the paper and the results look quite different from 1.5 so I'm guessing no. > the model may encounter challenges when synthesizing intricate structures, such as human hands I think there's two main reasons for poor hands/text - Humans care about certain areas of the image more than others, giving high saliency to faces, hands, body shape etc and lower saliency to backgrounds and textures. Due to the way the unet is trained it cares about all areas of the image equally. This means model capacity per area is uniform, leading to capacity problems for objects with a large number of configurations that humans care more about. - The sampling procedure implicitly assumes a uniform amount of variance over the entire image. Text glyphs basically never change, which means we should basically have infinite CFG in the parts of the image that contain text. I'm not sure if there's any point in working on this though, since both can be fixed by simply making a bigger model.
- sorenjan 3y ago
  Are any of these txt2img models being partially trained on synthetic datasets? Automatically rendering tens of thousands of images with different textures, backgrounds, camera poses, etc should be trivial with a handful of human models, or text using different fonts.
- mkaic 3y ago
  From hanging out around the LAION Discord server a bunch over the past few months, I've gathered that they're still using LAION-5B in some capacity, but they've done a bunch of filtering on it to remove low-quality samples. I believe Emad tweeted something to this effect at some point, too, but I can't find the tweet right now.
- Zetobal 3y ago
  The problem with laion was never the picture quality it's the tag quality and a lot of unusable data points. They traded some compute for a horde of users to tag them for them though. https://dbzer0.com/blog/a-collaboration-begins-between-stable-horde-and-laion/ https://dbzer0.com/blog/a-collaboration-begins-between-stabl...
brucethemoose2 3y ago
> At inference time, a user can then set the desired apparent resolution of the image via this size-conditioning. This is really cool. So is the difference between the first and second stage. The output of the first stage looks good enough for generating "drafts" when can then be picked by the user before being finished by the second stage. The improved cropping data augmentation is also a big deal. Bad framing is a constant issue with SD 1.5.
cypress66 3y ago
Are the weights out?
- GaggiX 3y ago
  Research weights but they should release the weights for everyone this month.
mkaic 3y ago
Some cool stuff from the paper: • Earlier SD versions would often generate images where the head or feet of the subject was cropped out of frame. This was because random cropping was applied to its training data as data augmentation, so it learned to make images that looked randomly cropped — not ideal! To fix this issue, they still used random cropping during training, but also gave the crop coordinates to the model so that it would know it was training on a cropped image. Then, they set those crop coordinates to 0 at test-time, and the model keeps the subject centered! They also did a similar thing with the pixel dimensions of the image, so that the model can learn to operate at different "DPI" ranges. • They're using a two-stage model instead of a single monolithic model. They have one model trained to get the image "most of the way there" and second model to take the output of the first and refine it, fixing textures and small details. Sort of mixture-of-experts-y. It makes sense that different "skillsets" would be required for the different stages of the denoising process, so it's reasonable to train separate models for each of the stages. Raises the question of whether unrolling the process further might yield more improvments — maybe a 3- or 4-stage model next? • Maybe I missed it, but I don't see in the paper whether the images they're showing come from the raw model or the RLHF-tuned variant. SDXL has been available for DreamStudio users to play with since April, and Emad indicated that the reason for this was to collect tons of human preference data. He also said that when the full SDXL 1.0 release happens later this month, both RLHF'd and non-RLHF'd variants of the weights will be available for download. I look forward to seeing detailed comparisons between the two. • They removed the lowest-resolution level of the U-Net — the 8x downsample block — which makes sense to me. I don't think there's really that much benefit from wasting flops on a tiny 4x4 or 8x8 latent tbh. Also thought it was interesting that they got rid of the cross-attention on the highest-resolution level of the U-Net.
- jack_riminton 3y ago
  The two stage process using different skill sets reminds me of the old painting masters. Often the master would come up with the overall composition, which requires a lot of creativity and judgement of forms, apprentice painters would then come in and paint clothes, trees or whatever and then the master would finish off the critical details such as eyes, mouth and hands etc. It makes sense to have different criteria of what good is for each stage
ilaksh 3y ago
You can access this with the Stability.ai API but it was really slow and unstable when I tried it (the day it came out). It was 20 seconds per image. Supposedly the Clipdrop API was also going to support it. Clipdrop has a web page for SDXL 0.9 but nothing in the API docs about it. Anyone know what the story is with the Clipdrop API and SDXL 0.9?

201 points by GaggiX 3y ago

in_the_bay 3y ago

Does it say anywhere what hardware they used for training?

rhogar 3y ago
The report does not detail hardware -- though it states that SDXL has 2.6B parameters in its UNet component, compared to SD 1.4/1.5 with 860M and SD 2.0/2.1 with 865M. So SDXL has roughly 3x more UNet parameters. In January, MosaicML claimed a model comparable to Stable Diffusion V2 could be trained with 79,000 A100-hours in 13 days. Some sort of inference can be made from this information, would be interested to hear someone with more insight here provide more perspective.
- darqwolff 3y ago
  wouldnt that mean more vram is required to load the model? they are claiming it will still work on 8 gb cards.
  - Filligree 3y ago
    Stable Diffusion 1/2 were made to run on cards with as little as 3GB of memory. Using the same techniques, yes, this will fit in 8.
  - brucethemoose2 3y ago
    I am guessing 8 bit quantization will be a thing for SDXL. It should be easy(TM) with bitsandbytes, or ML compiler frameworks.
    - GaggiX 3y ago
      bitsandbytes is only used during training with these models tho (the 8-bit Adamw) quantizing the weights and the activations to a range of 256 values when the model needs to output a range 256 values creates noticeable artifacts as they are not going to map 1-to-1.
      - liuliu 3y ago
        Draw Things recently released a 8-bit quantized SD model that has comparable output as the FP16. It does use k-means based LUT and separate weights into blocks to minimize quantization errors.
        GaggiX 3y ago
        I was going to search on the internet about it, but then I realized you are the author (and there is nothing online I think). I imagine that the activations are left in FP16 and the weights are converted in FP16 during inference, right? Btw very cool
        liuliu 3y ago
        Yes, computes are carried out in FP16 (so there is no compute efficiency gains, might be latency reductions due to memory-bandwidth saving). These savings are not realized yet because no custom kernels introduced yet.

p1esk 3y ago

“Images generated with our code use the invisible-watermark library to embed an invisible watermark into the model output. We also provide a script to easily detect that watermark.” Interesting - if it can be detected then it can be removed, right?

menzoic 3y ago
It's open source. It can be removed directly from the code.
- donkeyd 3y ago
  Also, most people use other tools like web-ui to run these models, which don't seem to apply any watermarks.
- p1esk 3y ago
  Sure, I just don’t understand why they’re doing it. In theory they could also train the model to always add a watermark.
  - aenvoker 3y ago
    Stability, Midjourney, Dall-e, etc don’t want their products to be used for misinformation. They really do want AI generated images to be trivially identifiable. There are papers around describing how to bake a watermark into a model. Don’t know if anyone is doing that yet.
    - Der_Einzige 3y ago
      And many will start working on techniques to unbake it from the model. A cat and mouse game, like all other attempts to track.
      - wongarsu 3y ago
        Sure, but that can be said about any security measure. If we agree that passing AI generated art as human-made or as a photograph is bad, then making that harder to do is worthwhile.

naillo 3y ago

I love stability AI

chaos_emergent 3y ago

how refreshing it is to see technical details in a technical report /s

Jack000 3y ago

Not seeing anything about the dataset, are they still using LAION? There's no mention of LAION in the paper and the results look quite different from 1.5 so I'm guessing no. > the model may encounter challenges when synthesizing intricate structures, such as human hands I think there's two main reasons for poor hands/text - Humans care about certain areas of the image more than others, giving high saliency to faces, hands, body shape etc and lower saliency to backgrounds and textures. Due to the way the unet is trained it cares about all areas of the image equally. This means model capacity per area is uniform, leading to capacity problems for objects with a large number of configurations that humans care more about. - The sampling procedure implicitly assumes a uniform amount of variance over the entire image. Text glyphs basically never change, which means we should basically have infinite CFG in the parts of the image that contain text. I'm not sure if there's any point in working on this though, since both can be fixed by simply making a bigger model.

sorenjan 3y ago
Are any of these txt2img models being partially trained on synthetic datasets? Automatically rendering tens of thousands of images with different textures, backgrounds, camera poses, etc should be trivial with a handful of human models, or text using different fonts.
mkaic 3y ago
From hanging out around the LAION Discord server a bunch over the past few months, I've gathered that they're still using LAION-5B in some capacity, but they've done a bunch of filtering on it to remove low-quality samples. I believe Emad tweeted something to this effect at some point, too, but I can't find the tweet right now.
Zetobal 3y ago
The problem with laion was never the picture quality it's the tag quality and a lot of unusable data points. They traded some compute for a horde of users to tag them for them though. https://dbzer0.com/blog/a-collaboration-begins-between-stable-horde-and-laion/ https://dbzer0.com/blog/a-collaboration-begins-between-stabl...

brucethemoose2 3y ago

> At inference time, a user can then set the desired apparent resolution of the image via this size-conditioning. This is really cool. So is the difference between the first and second stage. The output of the first stage looks good enough for generating "drafts" when can then be picked by the user before being finished by the second stage. The improved cropping data augmentation is also a big deal. Bad framing is a constant issue with SD 1.5.

cypress66 3y ago

Are the weights out?

GaggiX 3y ago
Research weights but they should release the weights for everyone this month.

mkaic 3y ago

Some cool stuff from the paper: • Earlier SD versions would often generate images where the head or feet of the subject was cropped out of frame. This was because random cropping was applied to its training data as data augmentation, so it learned to make images that looked randomly cropped — not ideal! To fix this issue, they still used random cropping during training, but also gave the crop coordinates to the model so that it would know it was training on a cropped image. Then, they set those crop coordinates to 0 at test-time, and the model keeps the subject centered! They also did a similar thing with the pixel dimensions of the image, so that the model can learn to operate at different "DPI" ranges. • They're using a two-stage model instead of a single monolithic model. They have one model trained to get the image "most of the way there" and second model to take the output of the first and refine it, fixing textures and small details. Sort of mixture-of-experts-y. It makes sense that different "skillsets" would be required for the different stages of the denoising process, so it's reasonable to train separate models for each of the stages. Raises the question of whether unrolling the process further might yield more improvments — maybe a 3- or 4-stage model next? • Maybe I missed it, but I don't see in the paper whether the images they're showing come from the raw model or the RLHF-tuned variant. SDXL has been available for DreamStudio users to play with since April, and Emad indicated that the reason for this was to collect tons of human preference data. He also said that when the full SDXL 1.0 release happens later this month, both RLHF'd and non-RLHF'd variants of the weights will be available for download. I look forward to seeing detailed comparisons between the two. • They removed the lowest-resolution level of the U-Net — the 8x downsample block — which makes sense to me. I don't think there's really that much benefit from wasting flops on a tiny 4x4 or 8x8 latent tbh. Also thought it was interesting that they got rid of the cross-attention on the highest-resolution level of the U-Net.

jack_riminton 3y ago
The two stage process using different skill sets reminds me of the old painting masters. Often the master would come up with the overall composition, which requires a lot of creativity and judgement of forms, apprentice painters would then come in and paint clothes, trees or whatever and then the master would finish off the critical details such as eyes, mouth and hands etc. It makes sense to have different criteria of what good is for each stage

ilaksh 3y ago

You can access this with the Stability.ai API but it was really slow and unstable when I tried it (the day it came out). It was 20 seconds per image. Supposedly the Clipdrop API was also going to support it. Clipdrop has a web page for SDXL 0.9 but nothing in the API docs about it. Anyone know what the story is with the Clipdrop API and SDXL 0.9?