7 ms·
Can someone explain to me how saying “we utilize CLIP” and “we are zero-shot” aren’t contradictory? You’d be hard pressed to come up with a class of things fro
by corysama 5y ago
Can someone explain to me how saying “we utilize CLIP” and “we are zero-shot” aren’t contradictory? You’d be hard pressed to come up with a class of things from reality or popular fiction that CLIP did not see a few shot of.
- yeldarb 5y agoZero-shot means it can generalize to things it's never seen before. You don't need to train CLIP on any examples of "The stacks from Ready Player One drawn by MC Escher" to have it do a pretty good job of being able to judge which outputs of a GAN more closely resemble that concept[1]. Saying this isn't "zero-shot" because CLIP has been trained on a corpus (the Internet) that had some info about the setting of the movie Ready Player One and the artist MC Escher isn't a particularly useful interpretation. By the same reasoning, humans aren't capable of "zero-shot" classification either because they have prior knowledge of real-world concepts. The model does object tracking well whether you feed it fish, or playing cards, or sperm because the generalized feature vector it extracts represent all the possible properties. It "knows" (and uses) properties like "facing right", "profile view", "blue sheen", "foggy", etc. to make its determination of which instances are which. [1] https://twitter.com/braddwyer/status/1430576141572657162 https://twitter.com/braddwyer/status/1430576141572657162
- gwern 5y agoCLIP has seen zero segmentations or bounding-boxed labeled examples of any object classes. Pure image-level contrastive learning against (extremely) noisy natural-language text captions.