4 ms·
Definitely possible. OpenAI's CLIP model already embeds images and text into the same embedding space. I don't know exactly how this particular model works but
by thomashop 4y ago
Definitely possible. OpenAI's CLIP model already embeds images and text into the same embedding space.
I don't know exactly how this particular model works but it is creating cross modal relationships otherwise it would not have the capacity to be good at so many tasks.
- ravi-delia 4y agoHow confident are we that it doesn't just have basically 600 smaller models and a classifier telling it which to use? Seems like it's a very small model (by comparison), which is certainly a mark in it's favor.
- sinenomine 4y agoYou can optimize pictures straight through it, and the pictures represent the combinatorial nature of the prompt pretty well. This contradicts the "flat array of classifiers" model.
- Der_Einzige 4y agoYou might find looking into the "lottery ticket hypothesis" fascinating.
- minimaxir 4y agoCLIP has a distinct Vision Transformer and distinct Text Transformer model that are then matmul'd to create the aligned embedding space. Gato apparently just uses a single model.
- sailingparrot 4y ago> OpenAI's CLIP model already embeds images and text into the same embedding space. Not really. Embeddings from images occupy a different region of the space than embeddings from text. A picture of cat and the text "cat" do not resolve to the same embedding. This is why DALL-E has a model learning to translate CLIP text embeddings into CLIP image embeddings before decoding the embedding.