7 ms·
Diffusion Models Beat GANs on Image Synthesis
- ansk 5y agoWhy broadly claim an entire class of generative model outperforms another? They simply show that their particular implementation of a diffusion model attains better performance (for some metric) than the current best implementation of a GAN. Almost inevitably (and likely in the very near future) someone will invest the time and tuning to squeeze out some more performance from a GAN/VAE/flow, and this title will be outdated.
- eru 5y agoWhile I tend to agree with you, aren't title like these understood to be short hand for what you described?
- forgingahead 5y agoDiffusion Models Can Sometimes Beat GANs on Image Synthesis
- ansk 5y agoFor an actual example of this more cautious framing, see another paper (also by OpenAI, also comparing two classes of generative models), https://arxiv.org/abs/2011.10650 https://arxiv.org/abs/2011.10650.
- bigyikes 5y agoI read your comment, thought “to get clicks, duh”, and then I clicked through the link. This is a research paper. I would like to echo your question: why? Does academia incentivize clickbait?
- Fomite 5y agoWith the rise of Altmetrics? Arguably yes. IIRC, the last research I saw on this, which was awhile back, argued that clever titles get read more and cited less. https://www.researchtrends.com/issue24-september-2011/heading-for-success-or-how-not-to-title-your-paper/ https://www.researchtrends.com/issue24-september-2011/headin...
- eru 5y agoWell, not sure it's any worse than cite-bait?
- bearzoo 5y agoEh I feel like may be worth cutting some slack here.. It sounds like you’d like the title to be prefaced with “one time that”. I don’t think anyone reading this says, hey gans are done for. I doubt the authors think that and I (clearly) don’t get the impression that’s what they mean. Even the first sentence of the abstract: > We show that diffusion models can achieve image sample quality superior to the current state-of-the-art generative models Has the word current in front of state of the art generative models.
- world_peace42 5y agoRed Sox beat Yankees in the world series =\= Red Sox will always beat the Yankees in every world series game. And I’m guessing most people reading this title know that. I’m sure there are people new to the field or new to CS research in general who might not know how to interpret this, but the only thing most people take out of this title is “hmm, maybe we should consider and study diffusion models more seriously”.
- deleted 5y ago[deleted]
- ansk 5y agoThe preeminence of either GANs or diffusion models is an ongoing debate, with evidence trickling in paper by paper. To claim diffusion models beat GANs is like claiming the Red Sox beat the Yankees because they happen to be ahead in the third inning.
- catillac 5y agoIt’s like saying the Sox are ahead of the Yankees and it’s the third inning with no predetermined number of innings. That nuance is what makes the title fair.
- marksmith2996 5y agoThe Red Sox and Yankees can't play each other in the World Series fyi.
- jostmey 5y agoWhy I agree with your sentiment, GANs always seemed finickily and unreliable. I am willing to indulge new approaches to find something more robust
- peteretep 5y agoAre you going on the title? If so, the verb "beat" here admits both a singular past action and a present continuous action.
- sillysaurusx 5y agoIt's fine to claim it. It's shorthand for "We achieve smaller FID values (most of the time) than any other generative model." You're correct about what you're saying, though. It's also worse: FID is a measurement biased towards imagenet-style images, and this model was also trained on imagenet, so it's quite well-suited for achieving small FID values. Whereas for generative anime, it's still unclear what the best model architecture is.
- colordrops 5y agoI don't know enough about this to understand why their model is better, but the imperfections in their images are much more unsettling that other algorithms.
- brink 5y agoYeah, some of those images are nightmare material.
- tobr 5y agoI wonder if it would be possible to train them to specifically avoid generating images that look scary to people. Eg prefer blur or other artifacts over distorted bodies and faces.
- ec109685 5y agoThe last set of images has some really scary generated humans via the GAN they were comparing against in the male w/ fish examples.
- mateo1 5y agoMaybe that's a lot to ask, but can someone explain to me or guide me to material to gain a better understanding of principles behind diffusion probabilistic models? My knowledge of statistics is just too basic to even read these papers.
- ericjang 5y agoI can try to provide a high level intuition. For actual experts, please forgive my errors of simplification. Drawing samples from "simple" distributions like normal distributions is computationally easy. You can do it in microseconds. Sampling an arbitrary 1-D distribution is a bit harder - you have to invert its cumulative density function to recover probability under an "equivalent" uniform random variable, and potentially use a rejection sampling approach to sample 1-D values under this distribution. Sampling a high-D distribution (such as an image) is even harder - you need to learn a mapping from this high-D image back to "tractable" densities. This imposes some pretty harsh "optimization bottlenecks" when trying to contort the manifold of images to normal distributions. The whole point of this exercise is that the transformation respects a valid probability distribution, so you can start from the normal distribution and apply this mapping to get a valid sample from the image distribution. This in practice is pretty hard, and the quality of samples seems to be lower than other forms of deep generative models which use less parameters. Now instead of learning such a complex, hard-to-optimize transformation to valid densities, what if we instead learn a function E(x) that outputs a scalar "energy". The energy is low for "realistic" images, and high for unrealistic images. Kind of like inverse probability, except its not normalized - the energy value tells you nothing about likelihood unless you know the energy for all other images possible. This tends to actually be easier than learning densities, because the functional form of this energy function is unconstrained. Furthermore, not knowing likelihoods doesn't stop you from getting "realistic" image, as all you need to do is descend the gradient x -= d/dx E(x), which takes you to an image with "lower" energy (i.e. more realistic). Under certain procedures (e.g. adding some noise to the gradient), this can be thought of as actually equivalent to sampling from a valid probability distribution, even though you can't compute its likelihood analytically. The diffusion probabilistic model you refer to can be thought of as such a model - the more steps you take (i.e. the more compute you spend), the better the quality of the model. GANs can be thought of as a one-pass neural network amortization of the end result of the diffusion process, but unlike MCMC methods, they cannot be "iteratively refined" with additional compute. Their sample quality is limited to whatever the generator spits out, even if you had additional compute.
- sillysaurusx 5y agoThe biggest downside with diffusion models is that GANs can be rendered in much less than half a second (sometimes 10fps or higher) on one core of a standard device you probably have. (Many of you have an nVidia GPU, and any ol’ nVidia GPU will render stylegan quickly.) With diffusion models, you need to do >25 forward passes to achieve a result. It’s kind of like an O(1) algorithm vs O(N): stylegan has one pass, diffusion models have N. And N is currently 25 or more, which means it tends to be 25x slower than stylegan at a minimum. (In our experience it was often many seconds to a full minute before we saw results, but we didn’t try very hard to make it fast, and this paper shows advances in speed since then.) The flipside: this paper has the most beautiful photorealistic complex samples I’ve ever seen. I don’t really care if they’re cherry picked; it’s hard to pick cherries on a rotten tree. I know. The last thing I want to say is that I am disappointed the model is, yet again, not released. The hoarding of ML models needs to stop. These research models aren’t commercially useful, but they have incredible benefits for people outside of ML. I myself got into ML thanks to being able to play with GPT-2 and Sfylegan. I would leap at the chance to play with AlphaZero or OpenAI’s old Dota 2 models; no reason to keep those locked up. I also don’t care about the excuses for not releasing: none of them hold water. In my experience, no model has ever had a society-threatening impact, and it’s usually a euphemism for “we’re worried there’s a small chance we might look bad.” Releasing your model is as simple as scp’ing to your server; just do it and quit worrying so much. Love the work. Diffusion models are a really interesting way of thinking about generative approaches in ML.
- gfaure 5y agoAlthough the weights aren't available, I wanted to note that the model source itself is actually hosted at https://github.com/openai/guided-diffusion https://github.com/openai/guided-diffusion.
- sillysaurusx 5y agoIndeed, the weights aren't available, and researchers seem to console themselves by saying "at least we're publishing code." Well, thank you. Code is nice. Y'know what else is nice? Being able to reproduce your claimed results without investing six months of mental effort and 500 GPU hours on a V100. (Sorry, I'll step off the rant now. I just miss the old days, the ye olde long-long ago of "one and a half years." It seemed like every other day there was another neat model release that happened to be world-changing as a matter of course: StyleGAN, BigGAN, GPT-2. Before that there was YOLO and a ton of other neat stuff. Nowadays it's "Oh golly gee, I'm not so sure we should release this panda generator, it might have ethical implications down the line if people start generating pandas in sexy poses." But hopefully it's just a phase or something.)
- grouphugs 5y agoaaaaaaand the nazis weaponized it like fucking morons
- yarg 5y agoI've disliked GANs since the Nvidia man with his head melting back into the bedroom made me feel like I was having a stroke. GANs blindspots have the potential to generate some confusing optical illusions.
- kaoD 5y agoWhat Nvidia man are you referring to?
- yarg 5y agoSorry, I missed this - unfortunately I did not save the picture. Repeatedly clicking refresh on https://thispersondoesnotexist.com/ https://thispersondoesnotexist.com/ I eventually reached an image where the top of a generated man's head melted into the room behind it. Melted, not faded. The room and the man's head were not in anyway separate and superimposed, it was a perfect fluid transition. Looking at it, it genuinely felt like my brain was malfunctioning. I don't know the extent to which it was due to under-training or some more fundamental limitation of GANs; the feeling that I had was that the generator had found an exploitable hole in the discernor's comprehension.
- phonebucket 5y agoThis new approach to generative modelling looks very intriguing. In a similar ilk, there's this ICLR paper from this year using stochastic differential equations for generative modelling: https://arxiv.org/abs/2011.13456 https://arxiv.org/abs/2011.13456
- londons_explore 5y agoThere are samples with the "Download PDF" button. There are no samples if you click the https link to GitHub...
- mhagiwara 5y agoIf you are interested in learning the intuition & theory behind diffusion and score-based models, I highly recommend this talk by Stefano Ermon from Stanford: https://www.youtube.com/watch?v=8TcNXi3A5DI https://www.youtube.com/watch?v=8TcNXi3A5DI