3 ms·

I do not want to be too much of a downer, but is there something keeping us from just using the traditional verbiage we've used in the community for years and c

I do not want to be too much of a downer, but is there something keeping us from just using the traditional verbiage we've used in the community for years and call it "projecting into a shared latent space" (or something bland but descriptive that a researcher like me could quickly latch onto, like 'separate key and query encoders')? I understand that proprietary names are necessary to sell ideas, architectures, etc, but projecting things into the same latent space is an....old concept, that, like the old Ecclesiastes verse, come up in new and unique ways/applications. Though there is nothing new under the sun, indeed. Please forgive this stodgy young person. Frippery and grumpiness are a deep skill of mine, and I apply it to my own DL research as well. I do not want to discourage the author from writing more pieces, explaining concepts is I believe a great trend to have in a community. Thank you and curious for anyone's thoughts. <3 :) :')))) <3

jamesblonde 3y ago
You are correct that we have two modalities projecting into a shared latent space. However, that doesn't convey anything about how to jointly train the embedding models, does it? That's what the two tower embedding model shows - how to jointly train the models using distance functions like cosine distance. On an aside, others, as esteemed as Yann Lecun, are talking about learning in a shared latent space between 2 modalities, but use different names - joint embeddings, because it's a different method. https://arxiv.org/abs/2307.12698 https://arxiv.org/abs/2307.12698
- imjonse 3y ago
  My understanding is the LeCun uses the term joint embedding as an alternate name for siamese networks. It does not necessarily imply multimodality and he has used it mostly in the context of SSL for images.
isaacfung 3y ago
Two tower isn't a new term afaik. It's often used in the context of recommendation systems where the two modalities are users/query and items. https://github.com/creyesp/Awesome-recsys https://github.com/creyesp/Awesome-recsys
- tysam_and 3y ago
  Gotcha, thanks. I guess it could be useful as shorthand for specifying 'joint latent space of multiple modalities', since 'two tower' is shorter and more established (though I guess it had to get established at some point, lol). Much appreciated on the clarification, something I did not know/had not heard about! Recommendation systems seem to be pretty disjoint from a lot of ML talk/news/etc generally, at least in the sphere of the world that I'm in! <3 :'))))
imjonse 3y ago
I haven't heard the term before either, but it's mentioned at least by Google too so not entirely unheard of in the multimodal training context. Even so I also think this could have been conveyed without using that term. Two towers and even twin towers as seen in some places sounds exceedingly marketingy. But as others have pointed out, this may have been a popular term in recommendation systems before the rise of deep learning multimodality. https://blog.research.google/2022/06/limoe-learning-multiple-modalities-with.html https://blog.research.google/2022/06/limoe-learning-multiple...
Palmik 3y ago
"Two-tower" or "dual-encoder" is an established terminology, many years old.

jamesblonde 3y ago

You are correct that we have two modalities projecting into a shared latent space. However, that doesn't convey anything about how to jointly train the embedding models, does it? That's what the two tower embedding model shows - how to jointly train the models using distance functions like cosine distance. On an aside, others, as esteemed as Yann Lecun, are talking about learning in a shared latent space between 2 modalities, but use different names - joint embeddings, because it's a different method. https://arxiv.org/abs/2307.12698 https://arxiv.org/abs/2307.12698

imjonse 3y ago
My understanding is the LeCun uses the term joint embedding as an alternate name for siamese networks. It does not necessarily imply multimodality and he has used it mostly in the context of SSL for images.

isaacfung 3y ago

Two tower isn't a new term afaik. It's often used in the context of recommendation systems where the two modalities are users/query and items. https://github.com/creyesp/Awesome-recsys https://github.com/creyesp/Awesome-recsys

tysam_and 3y ago
Gotcha, thanks. I guess it could be useful as shorthand for specifying 'joint latent space of multiple modalities', since 'two tower' is shorter and more established (though I guess it had to get established at some point, lol). Much appreciated on the clarification, something I did not know/had not heard about! Recommendation systems seem to be pretty disjoint from a lot of ML talk/news/etc generally, at least in the sphere of the world that I'm in! <3 :'))))

imjonse 3y ago

I haven't heard the term before either, but it's mentioned at least by Google too so not entirely unheard of in the multimodal training context. Even so I also think this could have been conveyed without using that term. Two towers and even twin towers as seen in some places sounds exceedingly marketingy. But as others have pointed out, this may have been a popular term in recommendation systems before the rise of deep learning multimodality. https://blog.research.google/2022/06/limoe-learning-multiple-modalities-with.html https://blog.research.google/2022/06/limoe-learning-multiple...

Palmik 3y ago

"Two-tower" or "dual-encoder" is an established terminology, many years old.