3 ms·

The geometry of diffusion guidance

68 points by alexmolas 3y ago

GaggiX 3y ago
It has to be said that the use of CFG is not free, the difference between training and inference change the sample distribution. This difference makes it easy for a trained eye to distinguish a SD/MJ image from a real one just from a thumbnail, compare to the text generated by ChatGPT that is pretty much indistinguishable given that there is no discrepancy between training and inference. SD had another flaw that would make the sample distribution different than the one it was trained one but it's easily fixable with new models: https://arxiv.org/abs/2305.08891 https://arxiv.org/abs/2305.08891, aka the model was never trained on pure noise and there was always a little bit of signal so the model did learn to always use the mean of x_T.
gdubs 3y ago
It's funny because while it may be hard for us humans to visualize extra spatial dimensions, many can easily reckon with the concept of a spreadsheet. If you think deeply about diffusion art models, you can start to grok extra dimensions in a different sense – things like, composition, tonality, subject matter. My brain can't really comprehend the tesseract-like nature of higher-dimensionality in a spatial sense – but I can get a glimmer of what it means to explore that space along different vectors. How over in this corner of the 'cube' is an image of a dog rendered in the style of Rembrandt. And how if I travel along a particular axis, it's still a dog in the style of Rembrandt, but now it's a funny dog in the style of Rembrandt. And if I turn and travel down a different axis still, it's a funny dog in the style of Rembrandt, at night. In this way, I find diffusion models a really interesting way to think about spaces of hyper dimensionality. It's a bit 'Flat World' – but, it's really helped me visualize how something can be transformed or explored in these 'extra' dimensions.
- incrudible 3y ago
  It is a nice way to think about it, but generally the dimensions are not individually meaningful/interpretable like that. They are entangled.

68 points by alexmolas 3y ago

It has to be said that the use of CFG is not free, the difference between training and inference change the sample distribution. This difference makes it easy for a trained eye to distinguish a SD/MJ image from a real one just from a thumbnail, compare to the text generated by ChatGPT that is pretty much indistinguishable given that there is no discrepancy between training and inference. SD had another flaw that would make the sample distribution different than the one it was trained one but it's easily fixable with new models: https://arxiv.org/abs/2305.08891 https://arxiv.org/abs/2305.08891, aka the model was never trained on pure noise and there was always a little bit of signal so the model did learn to always use the mean of x_T.

gdubs 3y ago

It's funny because while it may be hard for us humans to visualize extra spatial dimensions, many can easily reckon with the concept of a spreadsheet. If you think deeply about diffusion art models, you can start to grok extra dimensions in a different sense – things like, composition, tonality, subject matter. My brain can't really comprehend the tesseract-like nature of higher-dimensionality in a spatial sense – but I can get a glimmer of what it means to explore that space along different vectors. How over in this corner of the 'cube' is an image of a dog rendered in the style of Rembrandt. And how if I travel along a particular axis, it's still a dog in the style of Rembrandt, but now it's a funny dog in the style of Rembrandt. And if I turn and travel down a different axis still, it's a funny dog in the style of Rembrandt, at night. In this way, I find diffusion models a really interesting way to think about spaces of hyper dimensionality. It's a bit 'Flat World' – but, it's really helped me visualize how something can be transformed or explored in these 'extra' dimensions.

incrudible 3y ago
It is a nice way to think about it, but generally the dimensions are not individually meaningful/interpretable like that. They are entangled.