4 ms·
Not only does it perform better as a generalist, it also groups embeddings for the the multiple modalities of the same concept close together (keeps bird sounds
by m00x 4y ago
Not only does it perform better as a generalist, it also groups embeddings for the the multiple modalities of the same concept close together (keeps bird sounds close to bird image in its internal representation, etc). This way it can learn in different dimensions, just like humans do.
You could ask it to sing like a bird in text, and it could respond in image + sound, or just sound. This allows for better general understanding of things. Deepmind did the same with Gato: https://www.deepmind.com/publications/a-generalist-agent https://www.deepmind.com/publications/a-generalist-agent
- notsoprocoder 4y agoI can’t see where in the paper Data2Vec is reported to have contextual embeddings that work across modalities. Can you reference the section in the paper?
- m00x 4y agoDifferent models for now, but it's essentially doing this through the teacher model. If you go down to the numbers, it's very similar. They're saying they want to do this later. Gato does this in the same embeddings IIRC, but I skimmed the paper so I could be wrong.
- notsoprocoder 4y agoThanks! I should probably read the gato paper. If only there were enough hours in the day.