3 ms·
> It’s not trained on labeled data This could be misleading to some people. The original inputs to the CLIP encoders are pairs of images and text which are kno
by runnerup 5y ago
> It’s not trained on labeled data
This could be misleading to some people. The original inputs to the CLIP encoders are pairs of images and text which are known to match.
Both the text encoder and the image encoder are then trained to minimize differences in output from eachother when given corresponding (“labeled” / “ground truth”) image/text pairs.
- astrange 5y agoOh that’s true, but I think they meant scraped collections like Wikipedia images with captions - so that is matching text, but it’s not labels written in a consistent vocabulary by annotator teams. This is good and bad, since it makes it more robust.