4 ms·

Two-Tower Embedding Model

77 points by jamesblonde 3y ago

joewferrara 3y ago
This is a very cool concept. The example given of Bytedance combining the text embedder ALBERT with the a transformer image embedder, to make an embedder that can do image and text at the same time to get an interaction score is fascinating. I had not heard of being able to combine unrelated embedders before, and I want to know more examples. A quick google search found this in depth blog article of how their using two-tower embeddings at Uber, which was written recently (this past July) https://www.uber.com/blog/innovative-recommendation-applications-using-two-tower-embeddings/ https://www.uber.com/blog/innovative-recommendation-applicat...
- jamesblonde 3y ago
  The two modalities that will be combined first will be the most profitable modalities :) - Uber combine products with user-query-history-context. Images and text are two other modalities that are getting traction. I am looking at combining financial transactions and suspicious accounts. You need ground truth data that combines the two modalities - that's the starting point.
- pkage 3y ago
  Combining embeddings is the backbone of multimodal LLMs, such as InstructBLIP[0] or LLaVA[1]. Those architectures take the output tokens from a (frozen) vision transformer and train a very small projection layer between the output token space of the ViT and the input space of the LLM. [0] https://arxiv.org/abs/2305.06500 https://arxiv.org/abs/2305.06500 [1] https://llava-vl.github.io/ https://llava-vl.github.io/
  - 3abiton 3y ago
    I wonder when Dalle3 is released if OpenAi will release any technical documents about the integration with gpt4. Their approach might be similar.
    - isaacfung 3y ago
      OpenAI used this approach for CLIP before blip and llava. CLIP is used to encode the text prompt in stable diffusion. Not sure about Dalle.
tysam_and 3y ago
I do not want to be too much of a downer, but is there something keeping us from just using the traditional verbiage we've used in the community for years and call it "projecting into a shared latent space" (or something bland but descriptive that a researcher like me could quickly latch onto, like 'separate key and query encoders')? I understand that proprietary names are necessary to sell ideas, architectures, etc, but projecting things into the same latent space is an....old concept, that, like the old Ecclesiastes verse, come up in new and unique ways/applications. Though there is nothing new under the sun, indeed. Please forgive this stodgy young person. Frippery and grumpiness are a deep skill of mine, and I apply it to my own DL research as well. I do not want to discourage the author from writing more pieces, explaining concepts is I believe a great trend to have in a community. Thank you and curious for anyone's thoughts. <3 :) :')))) <3
- jamesblonde 3y ago
  You are correct that we have two modalities projecting into a shared latent space. However, that doesn't convey anything about how to jointly train the embedding models, does it? That's what the two tower embedding model shows - how to jointly train the models using distance functions like cosine distance. On an aside, others, as esteemed as Yann Lecun, are talking about learning in a shared latent space between 2 modalities, but use different names - joint embeddings, because it's a different method. https://arxiv.org/abs/2307.12698 https://arxiv.org/abs/2307.12698
  - imjonse 3y ago
    My understanding is the LeCun uses the term joint embedding as an alternate name for siamese networks. It does not necessarily imply multimodality and he has used it mostly in the context of SSL for images.
- isaacfung 3y ago
  Two tower isn't a new term afaik. It's often used in the context of recommendation systems where the two modalities are users/query and items. https://github.com/creyesp/Awesome-recsys https://github.com/creyesp/Awesome-recsys
choeger 3y ago
Soo, if and when this "AI" is applied to suggestions, will it stop to suggest washing machines to me after I just bought one the other day in the same shop? Or the same article I just read this morning? As long as this very simply case is not covered, I consider these algorithms cow manure.
taylorius 3y ago
Please don't take this the wrong way, but I'm somewhat knowledgeable about AI and machine learning, and when I heard the phrase "two tower embedding" the first thing I thought of was the tragedy on 9/11. So I'm not sure how good a slogan it is, if that was your aim at all.
- world2vec 3y ago
  It could be a reference to Tolkien's "The Two Towers".
  - taylorius 3y ago
    Very true!
- 121789 3y ago
  Two-tower embedding are fairly common approaches in big tech firms, so you should adjust your perception of how knowledgeable you are about machine learning
  - taylorius 3y ago
    Fair enough, haha! I did say somewhat!
rdedev 3y ago
So i have played around with this architecture specifically for entity linking. Two BERT encoders: one for the query text and another for the candidates. Initially the two encoders were separate but trained together. When I tried using the same BERT model for both, the accuracy jumped by 4 percentage points. Was pretty suprised by this and i guess it got me thinking that maybe a simple cosine similarity loss function is not enough information for the model to shared latent space. Maybe we also need some weights to be the same between encoders. Granted in my use case above they are the same modality but if we are building a model with image and text encoders it might be helpful to try and tie the weights in the last layers of those two encoders
- refulgentis 3y ago
  Check into SBERT, sounds perfect for what you're trying for: same encoder, asymmetric search
johnsutor 3y ago
I much prefer the n-tower approach https://arxiv.org/pdf/2307.10802.pdf https://arxiv.org/pdf/2307.10802.pdf

77 points by jamesblonde 3y ago

This is a very cool concept. The example given of Bytedance combining the text embedder ALBERT with the a transformer image embedder, to make an embedder that can do image and text at the same time to get an interaction score is fascinating. I had not heard of being able to combine unrelated embedders before, and I want to know more examples. A quick google search found this in depth blog article of how their using two-tower embeddings at Uber, which was written recently (this past July) https://www.uber.com/blog/innovative-recommendation-applications-using-two-tower-embeddings/ https://www.uber.com/blog/innovative-recommendation-applicat...

jamesblonde 3y ago
The two modalities that will be combined first will be the most profitable modalities :) - Uber combine products with user-query-history-context. Images and text are two other modalities that are getting traction. I am looking at combining financial transactions and suspicious accounts. You need ground truth data that combines the two modalities - that's the starting point.
pkage 3y ago
Combining embeddings is the backbone of multimodal LLMs, such as InstructBLIP[0] or LLaVA[1]. Those architectures take the output tokens from a (frozen) vision transformer and train a very small projection layer between the output token space of the ViT and the input space of the LLM. [0] https://arxiv.org/abs/2305.06500 https://arxiv.org/abs/2305.06500 [1] https://llava-vl.github.io/ https://llava-vl.github.io/
- 3abiton 3y ago
  I wonder when Dalle3 is released if OpenAi will release any technical documents about the integration with gpt4. Their approach might be similar.
  - isaacfung 3y ago
    OpenAI used this approach for CLIP before blip and llava. CLIP is used to encode the text prompt in stable diffusion. Not sure about Dalle.

tysam_and 3y ago

I do not want to be too much of a downer, but is there something keeping us from just using the traditional verbiage we've used in the community for years and call it "projecting into a shared latent space" (or something bland but descriptive that a researcher like me could quickly latch onto, like 'separate key and query encoders')? I understand that proprietary names are necessary to sell ideas, architectures, etc, but projecting things into the same latent space is an....old concept, that, like the old Ecclesiastes verse, come up in new and unique ways/applications. Though there is nothing new under the sun, indeed. Please forgive this stodgy young person. Frippery and grumpiness are a deep skill of mine, and I apply it to my own DL research as well. I do not want to discourage the author from writing more pieces, explaining concepts is I believe a great trend to have in a community. Thank you and curious for anyone's thoughts. <3 :) :')))) <3

jamesblonde 3y ago
You are correct that we have two modalities projecting into a shared latent space. However, that doesn't convey anything about how to jointly train the embedding models, does it? That's what the two tower embedding model shows - how to jointly train the models using distance functions like cosine distance. On an aside, others, as esteemed as Yann Lecun, are talking about learning in a shared latent space between 2 modalities, but use different names - joint embeddings, because it's a different method. https://arxiv.org/abs/2307.12698 https://arxiv.org/abs/2307.12698
- imjonse 3y ago
  My understanding is the LeCun uses the term joint embedding as an alternate name for siamese networks. It does not necessarily imply multimodality and he has used it mostly in the context of SSL for images.
isaacfung 3y ago
Two tower isn't a new term afaik. It's often used in the context of recommendation systems where the two modalities are users/query and items. https://github.com/creyesp/Awesome-recsys https://github.com/creyesp/Awesome-recsys

choeger 3y ago

Soo, if and when this "AI" is applied to suggestions, will it stop to suggest washing machines to me after I just bought one the other day in the same shop? Or the same article I just read this morning? As long as this very simply case is not covered, I consider these algorithms cow manure.

taylorius 3y ago

Please don't take this the wrong way, but I'm somewhat knowledgeable about AI and machine learning, and when I heard the phrase "two tower embedding" the first thing I thought of was the tragedy on 9/11. So I'm not sure how good a slogan it is, if that was your aim at all.

world2vec 3y ago
It could be a reference to Tolkien's "The Two Towers".
- taylorius 3y ago
  Very true!
121789 3y ago
Two-tower embedding are fairly common approaches in big tech firms, so you should adjust your perception of how knowledgeable you are about machine learning
- taylorius 3y ago
  Fair enough, haha! I did say somewhat!

rdedev 3y ago

So i have played around with this architecture specifically for entity linking. Two BERT encoders: one for the query text and another for the candidates. Initially the two encoders were separate but trained together. When I tried using the same BERT model for both, the accuracy jumped by 4 percentage points. Was pretty suprised by this and i guess it got me thinking that maybe a simple cosine similarity loss function is not enough information for the model to shared latent space. Maybe we also need some weights to be the same between encoders. Granted in my use case above they are the same modality but if we are building a model with image and text encoders it might be helpful to try and tie the weights in the last layers of those two encoders

refulgentis 3y ago
Check into SBERT, sounds perfect for what you're trying for: same encoder, asymmetric search

johnsutor 3y ago

I much prefer the n-tower approach https://arxiv.org/pdf/2307.10802.pdf https://arxiv.org/pdf/2307.10802.pdf