3 ms·
It's not trained on labeled data so it doesn't know bowls are a specific concept necessarily. It's all statistical similarity in the same way Google Image Searc
by astrange 5y ago
It's not trained on labeled data so it doesn't know bowls are a specific concept necessarily. It's all statistical similarity in the same way Google Image Search works. (from the original CLIP paper, it seems to think an apple and the word "apple" written on a piece of paper are the same thing)
The model in step 3 produces an image encoding (something like a sketch of the output) from a text encoding (something like what you typed), and the unCLIP model in step 2 produces images from that encoding. How much variation you get inside a specific input word varies a lot and is spread across those models.
- runnerup 5y ago> It’s not trained on labeled data This could be misleading to some people. The original inputs to the CLIP encoders are pairs of images and text which are known to match. Both the text encoder and the image encoder are then trained to minimize differences in output from eachother when given corresponding (“labeled” / “ground truth”) image/text pairs.
- astrange 5y agoOh that’s true, but I think they meant scraped collections like Wikipedia images with captions - so that is matching text, but it’s not labels written in a consistent vocabulary by annotator teams. This is good and bad, since it makes it more robust.