3 ms·
If you have a bit of background in math, I would encourage you to read the CLIP paper: https://arxiv.org/abs/2103.00020 https://arxiv.org/abs/2103.00020 Ultima
by SleekEagle 5y ago
If you have a bit of background in math, I would encourage you to read the CLIP paper: https://arxiv.org/abs/2103.00020 https://arxiv.org/abs/2103.00020
Ultimately, the link between words and their representations comes from the CLIP training. The model generates encodings (vectors) for both an image and its corresponding caption, and then the parameters of these encoders (the functions that generate the vectors) are tuned in order to minimize the angle between the textual and visual encodings that represent the same concept.
The core of your question is why minimizing the angle between like vectors is equivalent to learning what the "Platonic ideal" of a given object (in your example, a bowl) is, whether appearing as a textual representation or a visual one. This question is subtle and difficult to answer (if it's even a well-formulated question), but I'd say that the easiest interpretation is that the vector space is composed of a basis of vectors that each represent a distinct feature (which the model learns).