4 ms·

You’re overstating the simplicity of a scaling a GAN well. GigaGAN is the best quality out of those and requires 7 loss functions and is incredibly complicated

by nbardy 3y ago

You’re overstating the simplicity of a scaling a GAN well. GigaGAN is the best quality out of those and requires 7 loss functions and is incredibly complicated. Sure GANs can scale, but Diffusion models are drastically easier to scale.

gwern 3y ago
No, I'm not. BigGAN did fine on scaling up to JFT-300M with basically no changes beyond model size and a simple architecture. This is also what we were observing, even with a buggy BigGAN implementation. GigaGAN is the best quality, but that's mostly because it's also the biggest; as Table 1 shows most of the gains come from various kinds of additional scaling. (And this is moving the goalposts from the usual assertion that "GANs can't scale" to "they're harder to scale"; note the self-fulfilling nature of such assertions. Considering how there is next to no GAN scaling research, these results are remarkable and show how much low-hanging fruit there is.) Diffusion models are only 'drastically easier to scale' because researchers have spent the past 3 years researching pretty much nothing but diffusion models, discovering all sorts of subtle issues with them and how to make them scale, which is why it took them so long to become SOTA, and why massive architectural sweeps like https://arxiv.org/abs/2206.00364#nvidia https://arxiv.org/abs/2206.00364#nvidia were necessary to discover what makes them 'easier to scale'. If this level of brute force and moon math is 'easy', lord save us from any architecture which is 'hard'!
- GaggiX 3y ago
  Researchers have spent several years try to even create a GAN that can fit well a distribution made of aligned faces (resulted into StyleGAN1/2); with a simple unet with e-objective and cosine schedule you can fit much complex distributions, still using one loss: L1/L2. Reading your comments make me feel like that you believe that just every researchers (even extremely smart dude like Karras) just switch to diffusion models because they are idiots, they should have instead focus on GANs and today we will have GANs that are as powerful or more than the diffusion models we have today and also work one step; this is just a weird delusion. Diffusion models are just simply much easier to train (just a L1/L2 loss in most cases), write (for example your buggy BigGAN implementation), they usually work out-of-box on different resolutions and aspect ratios, you can just finetune them if you want to create an inpainting model; and for what is right now you just need much less compute to reach a good image coherency or maybe just reaching a coherence that as not been achieved by GAN models; like I would be curious even on a small scale experiment what a GAN (with ~55M parameters) would be able to perform after a 1-day/2-day GPU time of training on Icon645 dataset, because my diffusion model I can assure is much better than I could have imagine while being trivial to implement (I just implemented a Unet as I remember one, nothing rigorous and of course no architecture sweep).
  - sillysaurusx 3y ago
    There are so many holes in this, but to pick just one: > for example your buggy BigGAN implementation The bug was that the batchnorm gamma was initialized to zero instead of one, so all the model params were being multiplied by zero instead of one. Literally any model that uses batchnorm is susceptible to this bug, and it certainly has nothing to do with GANs. I think it’s too easy to spout dogma, and calling gwern delusional is something I’ve learned from experience will prove foolish as t approaches infinity.
    - GaggiX 3y ago
      It has to do with GANs as the added complexity makes the model more difficult to debug, the other day a friend of mine had a problem with the normalization layers of his diffusion model, it was fixed almost immediately, if something goes wrong you have less to worry about. Also no I don't take "he is gwern" as an argument ahah
      - sillysaurusx 3y ago
        Bookmarking this for when time proves you wrong after a couple years. I’ll drop by to say hello.
- ShamelessC 3y ago
  > Diffusion models are only 'drastically easier to scale' because researchers have spent the past 3 years researching pretty much nothing but diffusion models This is what tends to happen when you find a superior method. GAN’s are fine, they have plenty of promise for tasks requiring rapid inference. But diffusion models beat GANs on robustness and image quality every time.

by nbardy 3y ago

gwern 3y ago

No, I'm not. BigGAN did fine on scaling up to JFT-300M with basically no changes beyond model size and a simple architecture. This is also what we were observing, even with a buggy BigGAN implementation. GigaGAN is the best quality, but that's mostly because it's also the biggest; as Table 1 shows most of the gains come from various kinds of additional scaling. (And this is moving the goalposts from the usual assertion that "GANs can't scale" to "they're harder to scale"; note the self-fulfilling nature of such assertions. Considering how there is next to no GAN scaling research, these results are remarkable and show how much low-hanging fruit there is.) Diffusion models are only 'drastically easier to scale' because researchers have spent the past 3 years researching pretty much nothing but diffusion models, discovering all sorts of subtle issues with them and how to make them scale, which is why it took them so long to become SOTA, and why massive architectural sweeps like https://arxiv.org/abs/2206.00364#nvidia https://arxiv.org/abs/2206.00364#nvidia were necessary to discover what makes them 'easier to scale'. If this level of brute force and moon math is 'easy', lord save us from any architecture which is 'hard'!

GaggiX 3y ago
Researchers have spent several years try to even create a GAN that can fit well a distribution made of aligned faces (resulted into StyleGAN1/2); with a simple unet with e-objective and cosine schedule you can fit much complex distributions, still using one loss: L1/L2. Reading your comments make me feel like that you believe that just every researchers (even extremely smart dude like Karras) just switch to diffusion models because they are idiots, they should have instead focus on GANs and today we will have GANs that are as powerful or more than the diffusion models we have today and also work one step; this is just a weird delusion. Diffusion models are just simply much easier to train (just a L1/L2 loss in most cases), write (for example your buggy BigGAN implementation), they usually work out-of-box on different resolutions and aspect ratios, you can just finetune them if you want to create an inpainting model; and for what is right now you just need much less compute to reach a good image coherency or maybe just reaching a coherence that as not been achieved by GAN models; like I would be curious even on a small scale experiment what a GAN (with ~55M parameters) would be able to perform after a 1-day/2-day GPU time of training on Icon645 dataset, because my diffusion model I can assure is much better than I could have imagine while being trivial to implement (I just implemented a Unet as I remember one, nothing rigorous and of course no architecture sweep).
- sillysaurusx 3y ago
  There are so many holes in this, but to pick just one: > for example your buggy BigGAN implementation The bug was that the batchnorm gamma was initialized to zero instead of one, so all the model params were being multiplied by zero instead of one. Literally any model that uses batchnorm is susceptible to this bug, and it certainly has nothing to do with GANs. I think it’s too easy to spout dogma, and calling gwern delusional is something I’ve learned from experience will prove foolish as t approaches infinity.
  - GaggiX 3y ago
    It has to do with GANs as the added complexity makes the model more difficult to debug, the other day a friend of mine had a problem with the normalization layers of his diffusion model, it was fixed almost immediately, if something goes wrong you have less to worry about. Also no I don't take "he is gwern" as an argument ahah
    - sillysaurusx 3y ago
      Bookmarking this for when time proves you wrong after a couple years. I’ll drop by to say hello.
ShamelessC 3y ago
> Diffusion models are only 'drastically easier to scale' because researchers have spent the past 3 years researching pretty much nothing but diffusion models This is what tends to happen when you find a superior method. GAN’s are fine, they have plenty of promise for tasks requiring rapid inference. But diffusion models beat GANs on robustness and image quality every time.