6 ms·

I know 29 minutes is long but theoretically you can have all the images you ever want in a small 6gb package and run inference on (nearly) everything. That's fu

I know 29 minutes is long but theoretically you can have all the images you ever want in a small 6gb package and run inference on (nearly) everything. That's fucking amazing.

godelski 3y ago
But honest question, if this is your goal, why not use a GAN instead? You should still be able to produce high quality images but at a much faster rate (I'd guess around 10 minutes?). Sure, you'll have a bit lower diversity and maybe not SOTA quality image generation, but neither is this thing. Or you could reduce quality. This reddit user seems to be doing fast inference on a pi[0] using stylegan, but that's before mobile stylegan came out which uses <1GB for inference. (It is a distilled StyleGAN2 model. We could distill more recent models) Just seems like different models, different contexts. Certainly you'd want diffusion on the computer you're doing photoshop on, but random images? Different context. [0] https://www.reddit.com/r/raspberry_pi/comments/hf7lbh/i_made_a_pibased_standalone_generative_art_player/ https://www.reddit.com/r/raspberry_pi/comments/hf7lbh/i_made...
- GaggiX 3y ago
  The quality is not really close, also StyleGAN2 is not conditioned on text.
  - godelski 3y ago
    Sorry, I left too much knowledge for inference. I presumed it was safe to let users infer that a 2018 (StyleGAN) or 2019 (StyleGAN2) model was not going to compete with performance of a 2023 model, regardless of architecture. There has in fact been improvements in GANs in the last 5 years. Text conditioning is not unique to diffusion and is rather a subnetwork for conditioning (actually StyleGAN's whole innovation was creating a subnetwork for conditioning synthesis) There are definitely modern GANs that are T2I and computationally cheaper than a latent diffusion model for a comparable benchmark score (but recognizing that our metrics are limited and only rough qualifications of image quality, but that the meaningfulness of the metrics decreases with increased realism so that's not really a hindrance for our specific use case and as mentioned before decreased diversity.) > The quality is not really close, also StyleGAN is not conditioned on text. Some examples of T2I GANS with comparable quality (not even something I claimed for context of the request...): - GigaGAN (2023). Base generator @ 652.5M params and upsampler @ 359.1M params. While 512x512 generation has similar model size to SD 1.5 the inference speed is 16x faster. The gap widens for text conditioned super-resolution 128->1024 https://arxiv.org/abs/2303.05511 https://arxiv.org/abs/2303.05511 - StyleGAN-T (2023) (notably by Sauer who recently joined Stability and their first paper hit the front page a few weeks ago. Also includes the main SG authors from Nvidia) Figure 1 speaks for itself, noting that this is on T2I. It's also worth noting Sauer's previous work (StyleGAN-XL (2022)) did text conditioned experiments. https://arxiv.org/abs/2301.09515 https://arxiv.org/abs/2301.09515 - LAFITE (2022) comes in at a tiny 75M params for 256x256 generation and has quality comparable to the 12B param (autoregressive) DALL-E while being 1,600x faster in inference https://arxiv.org/abs/2111.13792 https://arxiv.org/abs/2111.13792 There are plenty more too. I'm not even suggesting proven architecture and train under comparable settings to that which the popular diffusion models have been trained to (which would be a fair one-to-one comparison), but what has already been done and demonstrated because the context is an engineering project not research. Certainly all the above works, including may diffusion methods, would vastly were they given the same treatment as Stable Diffusion but that's not the context here. Stop buying into hype. There isn't one model to rule them all, there are models that are better in differing contexts.
    - GaggiX 3y ago
      >Stop buying into hype. There isn't one model to rule them all, there are models that are better in differing contexts. I didn't say any of that, there is simply no open source GAN model that can compete with the open source diffusion models we have today, and the fact that these models can be distilled down to 1/2/4 steps makes GANs less attractive.
      - godelski 3y ago
        > I didn't say any of that I'm sorry, then I don't know what you're saying. Because what I read was 2 claims. 1) GAN quality is less than diffusion. 2) GANs can't do T2I. I think I adequately showed that both these assumptions were wrong. I'm not sure what else your comment meant as that contains all of its words... > there is simply no open source GAN model that can compete with the open source diffusion models we have today Again, I provided citations. Do you want the github links? Here's GigaGAN's: https://github.com/mingukkang/GigaGAN https://github.com/mingukkang/GigaGAN (checkpoints under the evaluation folder). What do you mean by compete? Because the metric scores are quite comparable. I think that's a reasonable interpretation of the word "competitive" but I'm a generative researcher so we might be using the terms to mean different things. (I even like diffusion more fwiw, but I'm particularly more interested in tractable density models) Do you mean "the open community has rallied around Stable Diffusion and sunk in more time to tuning this model and producing textual inversions and LoRAs which far out surpass that of any other model, even including other diffusion models and is a phenomena nearly exclusive to Stable Diffusion and has been mostly accomplished in the last year"? Because if so, yeah, I agree. But that's not a really good argument for saying Diffusion is better than GANs. It's a completely different argument. That's just to say __Stable Diffusion__ (not the class of diffusion models, or even more specifically latent diffusion models (the two are different)) has better community support. That's a reasonable argument, but a different one. Because it really just shows what you can do to __any__ model. These techniques and efforts are not architecture dependent nor are they even mode dependent. We see similar community effort around OpenLLaMa and GPT but less so around Bard/Gemini or Claude. So what? That's not really relevant to the conversation nor to the specific context that we're discussing which is very likely not going to include many of those TIs or LoRAs (or if it would you'd probably be training a custom one, so the point is again moot since LoRAs are not unique to diffusion models). I'm really happy a lot of people have entered the community and are effectively doing research, but it'd be quite naive to say that such a thing isn't possible around other architectures. Evangelism is quite useful, but not if it turns into religious beliefs. Hell, we can argue that Linux is better than Windows because Linux has a bigger hacker community but I'm not sure that's a great or even meaningful argument because it's void of context in what better actually means. The better OS is clearly situationally dependent and makes a lot of the OS holy wars silly. > and the fact that these models can be distilled Literally any model can be distilled. I'm not understanding your argument. Are you just arguing that such efforts have already taken place? Sure, I'll agree to that. But it's worth noting that the distilled diffusion models are still quite large and much larger than some of the aforementioned works. > 1/2/4 steps makes GANs less attractive. I even addressed the one step process. Which, single step diffusion generation represents a significant decrease in quality, so this seems to run counter to your prior argument. This is why I'm a bit confused. If we're going with a low step diffusion model then the case for the GAN becomes clearer because even your non-distilled GAN's inference is still higher and it's quality is definitely superior. So really, I am confused. I'm not sure what you're arguing. We can get into the weeds and discuss diversity, recall, memorization, density estimation, and all manner of things but these are quite open questions and frankly understudied. Plus we'll have to be extra nuanced because the metrics are proxies and incorporate different biases that different architectures are going to suffer from, making it difficult to compare in a more fair sense. But that's okay. Let's also be clear: GAN != StyleGAN{1,2,2-ada,3,XL,T,*} and diffusion != StableDiffusion. There are other GANs and other diffusions and even other image synthesis models, the vast majority of them being open source. Again, I'll assert, there are no universal models that are best for all situations. There are only models that are best at specific situations. You can translate this to "AGI doesn't exist but we have narrow AI" if that is clearer. But my claim is a bit broader still because if that AGI took a warehouse to run it still wouldn't be contextually appropriate.

I know 29 minutes is long but theoretically you can have all the images you ever want in a small 6gb package and run inference on (nearly) everything. That's fucking amazing.

godelski 3y ago

But honest question, if this is your goal, why not use a GAN instead? You should still be able to produce high quality images but at a much faster rate (I'd guess around 10 minutes?). Sure, you'll have a bit lower diversity and maybe not SOTA quality image generation, but neither is this thing. Or you could reduce quality. This reddit user seems to be doing fast inference on a pi[0] using stylegan, but that's before mobile stylegan came out which uses <1GB for inference. (It is a distilled StyleGAN2 model. We could distill more recent models) Just seems like different models, different contexts. Certainly you'd want diffusion on the computer you're doing photoshop on, but random images? Different context. [0] https://www.reddit.com/r/raspberry_pi/comments/hf7lbh/i_made_a_pibased_standalone_generative_art_player/ https://www.reddit.com/r/raspberry_pi/comments/hf7lbh/i_made...

GaggiX 3y ago
The quality is not really close, also StyleGAN2 is not conditioned on text.
- godelski 3y ago
  Sorry, I left too much knowledge for inference. I presumed it was safe to let users infer that a 2018 (StyleGAN) or 2019 (StyleGAN2) model was not going to compete with performance of a 2023 model, regardless of architecture. There has in fact been improvements in GANs in the last 5 years. Text conditioning is not unique to diffusion and is rather a subnetwork for conditioning (actually StyleGAN's whole innovation was creating a subnetwork for conditioning synthesis) There are definitely modern GANs that are T2I and computationally cheaper than a latent diffusion model for a comparable benchmark score (but recognizing that our metrics are limited and only rough qualifications of image quality, but that the meaningfulness of the metrics decreases with increased realism so that's not really a hindrance for our specific use case and as mentioned before decreased diversity.) > The quality is not really close, also StyleGAN is not conditioned on text. Some examples of T2I GANS with comparable quality (not even something I claimed for context of the request...): - GigaGAN (2023). Base generator @ 652.5M params and upsampler @ 359.1M params. While 512x512 generation has similar model size to SD 1.5 the inference speed is 16x faster. The gap widens for text conditioned super-resolution 128->1024 https://arxiv.org/abs/2303.05511 https://arxiv.org/abs/2303.05511 - StyleGAN-T (2023) (notably by Sauer who recently joined Stability and their first paper hit the front page a few weeks ago. Also includes the main SG authors from Nvidia) Figure 1 speaks for itself, noting that this is on T2I. It's also worth noting Sauer's previous work (StyleGAN-XL (2022)) did text conditioned experiments. https://arxiv.org/abs/2301.09515 https://arxiv.org/abs/2301.09515 - LAFITE (2022) comes in at a tiny 75M params for 256x256 generation and has quality comparable to the 12B param (autoregressive) DALL-E while being 1,600x faster in inference https://arxiv.org/abs/2111.13792 https://arxiv.org/abs/2111.13792 There are plenty more too. I'm not even suggesting proven architecture and train under comparable settings to that which the popular diffusion models have been trained to (which would be a fair one-to-one comparison), but what has already been done and demonstrated because the context is an engineering project not research. Certainly all the above works, including may diffusion methods, would vastly were they given the same treatment as Stable Diffusion but that's not the context here. Stop buying into hype. There isn't one model to rule them all, there are models that are better in differing contexts.
  - GaggiX 3y ago
    >Stop buying into hype. There isn't one model to rule them all, there are models that are better in differing contexts. I didn't say any of that, there is simply no open source GAN model that can compete with the open source diffusion models we have today, and the fact that these models can be distilled down to 1/2/4 steps makes GANs less attractive.
    - godelski 3y ago
      > I didn't say any of that I'm sorry, then I don't know what you're saying. Because what I read was 2 claims. 1) GAN quality is less than diffusion. 2) GANs can't do T2I. I think I adequately showed that both these assumptions were wrong. I'm not sure what else your comment meant as that contains all of its words... > there is simply no open source GAN model that can compete with the open source diffusion models we have today Again, I provided citations. Do you want the github links? Here's GigaGAN's: https://github.com/mingukkang/GigaGAN https://github.com/mingukkang/GigaGAN (checkpoints under the evaluation folder). What do you mean by compete? Because the metric scores are quite comparable. I think that's a reasonable interpretation of the word "competitive" but I'm a generative researcher so we might be using the terms to mean different things. (I even like diffusion more fwiw, but I'm particularly more interested in tractable density models) Do you mean "the open community has rallied around Stable Diffusion and sunk in more time to tuning this model and producing textual inversions and LoRAs which far out surpass that of any other model, even including other diffusion models and is a phenomena nearly exclusive to Stable Diffusion and has been mostly accomplished in the last year"? Because if so, yeah, I agree. But that's not a really good argument for saying Diffusion is better than GANs. It's a completely different argument. That's just to say __Stable Diffusion__ (not the class of diffusion models, or even more specifically latent diffusion models (the two are different)) has better community support. That's a reasonable argument, but a different one. Because it really just shows what you can do to __any__ model. These techniques and efforts are not architecture dependent nor are they even mode dependent. We see similar community effort around OpenLLaMa and GPT but less so around Bard/Gemini or Claude. So what? That's not really relevant to the conversation nor to the specific context that we're discussing which is very likely not going to include many of those TIs or LoRAs (or if it would you'd probably be training a custom one, so the point is again moot since LoRAs are not unique to diffusion models). I'm really happy a lot of people have entered the community and are effectively doing research, but it'd be quite naive to say that such a thing isn't possible around other architectures. Evangelism is quite useful, but not if it turns into religious beliefs. Hell, we can argue that Linux is better than Windows because Linux has a bigger hacker community but I'm not sure that's a great or even meaningful argument because it's void of context in what better actually means. The better OS is clearly situationally dependent and makes a lot of the OS holy wars silly. > and the fact that these models can be distilled Literally any model can be distilled. I'm not understanding your argument. Are you just arguing that such efforts have already taken place? Sure, I'll agree to that. But it's worth noting that the distilled diffusion models are still quite large and much larger than some of the aforementioned works. > 1/2/4 steps makes GANs less attractive. I even addressed the one step process. Which, single step diffusion generation represents a significant decrease in quality, so this seems to run counter to your prior argument. This is why I'm a bit confused. If we're going with a low step diffusion model then the case for the GAN becomes clearer because even your non-distilled GAN's inference is still higher and it's quality is definitely superior. So really, I am confused. I'm not sure what you're arguing. We can get into the weeds and discuss diversity, recall, memorization, density estimation, and all manner of things but these are quite open questions and frankly understudied. Plus we'll have to be extra nuanced because the metrics are proxies and incorporate different biases that different architectures are going to suffer from, making it difficult to compare in a more fair sense. But that's okay. Let's also be clear: GAN != StyleGAN{1,2,2-ada,3,XL,T,*} and diffusion != StableDiffusion. There are other GANs and other diffusions and even other image synthesis models, the vast majority of them being open source. Again, I'll assert, there are no universal models that are best for all situations. There are only models that are best at specific situations. You can translate this to "AGI doesn't exist but we have narrow AI" if that is clearer. But my claim is a bit broader still because if that AGI took a warehouse to run it still wouldn't be contextually appropriate.