4 ms·

I wonder if the outputted text-vectors will be similar to e.g the outputted image vector if they have the same "content". E.g will the vector for the text "A do

by opdahl 4y ago

I wonder if the outputted text-vectors will be similar to e.g the outputted image vector if they have the same "content". E.g will the vector for the text "A dog on a green field" be similar in the vector space to an image showing a dog on a green field? If so it opens up a lot of opportunities in the information retrieval / recommender system space. Edit: After reading the paper I see that they train an individual model for each modality, which means that they are not comparable. It would be interesting to see though if they could combine sound, text and images into the same training in the future.

tnzk 4y ago
It would be really nice if it was like that, and I guess they also hoped that they would get such a result "unexpectedly." I wonder if there was a technique to regulate the training so their higher layer will be similar despite the input are fundamentally different.
ttul 4y ago
This is basically how stable diffusion works.
- pineapple_sauce 4y ago
  Not quite. Stable Diffusion is CLIP guided, but I wouldn't say it's "basically" how it works. It's one of the core Also, just fyi, Google's alternative to Stable Diffusion/DALLE-2, Imagen is not CLIP guided. Thus being CLIP guided is not necessary for the problem.
cubacaban 4y ago
Mapping corresponding text and image into the same vector space is exactly how CLIP and other contrastive learning setups work: Have text and computer vision networks embed input data, and teach the networks to embed related inputs closely and unrelated ones far away. You can train the models on data scraped from the internet (e.g. images and corresponding captions) in a self-supervised way. Like you say it has huge applications for search and is also used by the generative models (to guide a generated image towards your textual input). BTW it seems that the data2vec is at its core actually just one model, only the input and output parts are different depending on the modality (text, image sound etc.) I wouldn't expect the learned representations to be similar for similar content across modalities. The point of data2vec is to use just one model with a very general self-supervised learning setup for tasks, with the different tasks hopefully benefitting from each other during training.

by opdahl 4y ago

tnzk 4y ago

It would be really nice if it was like that, and I guess they also hoped that they would get such a result "unexpectedly." I wonder if there was a technique to regulate the training so their higher layer will be similar despite the input are fundamentally different.

ttul 4y ago

This is basically how stable diffusion works.

pineapple_sauce 4y ago
Not quite. Stable Diffusion is CLIP guided, but I wouldn't say it's "basically" how it works. It's one of the core Also, just fyi, Google's alternative to Stable Diffusion/DALLE-2, Imagen is not CLIP guided. Thus being CLIP guided is not necessary for the problem.

cubacaban 4y ago

Mapping corresponding text and image into the same vector space is exactly how CLIP and other contrastive learning setups work: Have text and computer vision networks embed input data, and teach the networks to embed related inputs closely and unrelated ones far away. You can train the models on data scraped from the internet (e.g. images and corresponding captions) in a self-supervised way. Like you say it has huge applications for search and is also used by the generative models (to guide a generated image towards your textual input). BTW it seems that the data2vec is at its core actually just one model, only the input and output parts are different depending on the modality (text, image sound etc.) I wouldn't expect the learned representations to be similar for similar content across modalities. The point of data2vec is to use just one model with a very general self-supervised learning setup for tasks, with the different tasks hopefully benefitting from each other during training.