4 ms·
Good post, I always thought diffusion originated from score matching, today I realized diffusion came before score matching theory, so when OpenAI trained on 25
by sashank_1509 2y ago
Good post, I always thought diffusion originated from score matching, today I realized diffusion came before score matching theory, so when OpenAI trained on 250 million images, they didn’t even have great theory explaining why they were modeling the underlying distribution. Gutsy move
- reasonableklout 2y agoThe original Dickstein 2015 paper [1] formulated diffusion as maximizing (a lower bound of) log-likelihood of generating the distribution, so there was some theory. But my understanding is that the breakthrough was empirical results from Ho [2] and Nichol [3] showing diffusion could produce not only high-quality samples but better than GANs in some cases. [1] https://arxiv.org/abs/1503.03585 https://arxiv.org/abs/1503.03585 [2] https://arxiv.org/abs/2006.11239 https://arxiv.org/abs/2006.11239 [3] https://arxiv.org/abs/2105.05233 https://arxiv.org/abs/2105.05233