10 ms·

I'll just never understand how any of this works. I know it is trained on millions of existing images, but when you say "... a bowl ..." in your prompt, how doe

I'll just never understand how any of this works. I know it is trained on millions of existing images, but when you say "... a bowl ..." in your prompt, how does it decide what the bowl should look like? Does it pick one of the bowls it's seen at random? It doesn't ever quite draw the same bowl twice, does it? Is it somehow "imagining" a bowl, the way a human would, and some all new image of a bowl pops into its "head"?

simonw 5y ago
The trick is to start with random "gaussian noise" - something like https://opendatascience.com/wp-content/uploads/2017/03/noise3-560x300.jpg https://opendatascience.com/wp-content/uploads/2017/03/noise... - and then iteratively modify that image until it starts to look like the concept you want it to look like. I find the concept of a GAN - a Generative Adversarial Network - useful. My high-level attempt at explaining how those work is that you create two machine learning models, one that tries to create fake images and one that tries to see if an image is fake or not. The first one says "here's an image", the second one says "that's a fake", the first one learns from that and tries again, then keep going until an image scores highly on the test. The networks are adversarial because they are trying to outwit each other. (I'm sure a ML researcher could provide a better explanation than I can, but that's the way I think about it.)
- goodside 5y ago
  I don't believe Dall-E 2 incorporates GANs at all, but I haven't read the paper in detail. GANs were the best text-to-image models maybe a year ago but lately diffusion techniques are taking over.
  - simonw 5y ago
    Thanks for the keyword hint - this explanation looks good for diffusion models: https://ai.googleblog.com/2021/07/high-fidelity-image-generation-using.html https://ai.googleblog.com/2021/07/high-fidelity-image-genera...
- hackernewds 5y ago
  that is a very interesting and clear explanation, assuming it is correct (it is not my field but I would love to know if it isn't!)
SleekEagle 5y ago
If you have a bit of background in math, I would encourage you to read the CLIP paper: https://arxiv.org/abs/2103.00020 https://arxiv.org/abs/2103.00020 Ultimately, the link between words and their representations comes from the CLIP training. The model generates encodings (vectors) for both an image and its corresponding caption, and then the parameters of these encoders (the functions that generate the vectors) are tuned in order to minimize the angle between the textual and visual encodings that represent the same concept. The core of your question is why minimizing the angle between like vectors is equivalent to learning what the "Platonic ideal" of a given object (in your example, a bowl) is, whether appearing as a textual representation or a visual one. This question is subtle and difficult to answer (if it's even a well-formulated question), but I'd say that the easiest interpretation is that the vector space is composed of a basis of vectors that each represent a distinct feature (which the model learns).
astrange 5y ago
It's not trained on labeled data so it doesn't know bowls are a specific concept necessarily. It's all statistical similarity in the same way Google Image Search works. (from the original CLIP paper, it seems to think an apple and the word "apple" written on a piece of paper are the same thing) The model in step 3 produces an image encoding (something like a sketch of the output) from a text encoding (something like what you typed), and the unCLIP model in step 2 produces images from that encoding. How much variation you get inside a specific input word varies a lot and is spread across those models.
- runnerup 5y ago
  > It’s not trained on labeled data This could be misleading to some people. The original inputs to the CLIP encoders are pairs of images and text which are known to match. Both the text encoder and the image encoder are then trained to minimize differences in output from eachother when given corresponding (“labeled” / “ground truth”) image/text pairs.
  - astrange 5y ago
    Oh that’s true, but I think they meant scraped collections like Wikipedia images with captions - so that is matching text, but it’s not labels written in a consistent vocabulary by annotator teams. This is good and bad, since it makes it more robust.

simonw 5y ago

The trick is to start with random "gaussian noise" - something like https://opendatascience.com/wp-content/uploads/2017/03/noise3-560x300.jpg https://opendatascience.com/wp-content/uploads/2017/03/noise... - and then iteratively modify that image until it starts to look like the concept you want it to look like. I find the concept of a GAN - a Generative Adversarial Network - useful. My high-level attempt at explaining how those work is that you create two machine learning models, one that tries to create fake images and one that tries to see if an image is fake or not. The first one says "here's an image", the second one says "that's a fake", the first one learns from that and tries again, then keep going until an image scores highly on the test. The networks are adversarial because they are trying to outwit each other. (I'm sure a ML researcher could provide a better explanation than I can, but that's the way I think about it.)

goodside 5y ago
I don't believe Dall-E 2 incorporates GANs at all, but I haven't read the paper in detail. GANs were the best text-to-image models maybe a year ago but lately diffusion techniques are taking over.
- simonw 5y ago
  Thanks for the keyword hint - this explanation looks good for diffusion models: https://ai.googleblog.com/2021/07/high-fidelity-image-generation-using.html https://ai.googleblog.com/2021/07/high-fidelity-image-genera...
hackernewds 5y ago
that is a very interesting and clear explanation, assuming it is correct (it is not my field but I would love to know if it isn't!)

SleekEagle 5y ago

If you have a bit of background in math, I would encourage you to read the CLIP paper: https://arxiv.org/abs/2103.00020 https://arxiv.org/abs/2103.00020 Ultimately, the link between words and their representations comes from the CLIP training. The model generates encodings (vectors) for both an image and its corresponding caption, and then the parameters of these encoders (the functions that generate the vectors) are tuned in order to minimize the angle between the textual and visual encodings that represent the same concept. The core of your question is why minimizing the angle between like vectors is equivalent to learning what the "Platonic ideal" of a given object (in your example, a bowl) is, whether appearing as a textual representation or a visual one. This question is subtle and difficult to answer (if it's even a well-formulated question), but I'd say that the easiest interpretation is that the vector space is composed of a basis of vectors that each represent a distinct feature (which the model learns).

astrange 5y ago

It's not trained on labeled data so it doesn't know bowls are a specific concept necessarily. It's all statistical similarity in the same way Google Image Search works. (from the original CLIP paper, it seems to think an apple and the word "apple" written on a piece of paper are the same thing) The model in step 3 produces an image encoding (something like a sketch of the output) from a text encoding (something like what you typed), and the unCLIP model in step 2 produces images from that encoding. How much variation you get inside a specific input word varies a lot and is spread across those models.

runnerup 5y ago
> It’s not trained on labeled data This could be misleading to some people. The original inputs to the CLIP encoders are pairs of images and text which are known to match. Both the text encoder and the image encoder are then trained to minimize differences in output from eachother when given corresponding (“labeled” / “ground truth”) image/text pairs.
- astrange 5y ago
  Oh that’s true, but I think they meant scraped collections like Wikipedia images with captions - so that is matching text, but it’s not labels written in a consistent vocabulary by annotator teams. This is good and bad, since it makes it more robust.