5 ms·
This is a very cool concept. The example given of Bytedance combining the text embedder ALBERT with the a transformer image embedder, to make an embedder that c
by joewferrara 3y ago
This is a very cool concept. The example given of Bytedance combining the text embedder ALBERT with the a transformer image embedder, to make an embedder that can do image and text at the same time to get an interaction score is fascinating. I had not heard of being able to combine unrelated embedders before, and I want to know more examples. A quick google search found this in depth blog article of how their using two-tower embeddings at Uber, which was written recently (this past July)
https://www.uber.com/blog/innovative-recommendation-applications-using-two-tower-embeddings/ https://www.uber.com/blog/innovative-recommendation-applicat...
- jamesblonde 3y agoThe two modalities that will be combined first will be the most profitable modalities :) - Uber combine products with user-query-history-context. Images and text are two other modalities that are getting traction. I am looking at combining financial transactions and suspicious accounts. You need ground truth data that combines the two modalities - that's the starting point.
- pkage 3y agoCombining embeddings is the backbone of multimodal LLMs, such as InstructBLIP[0] or LLaVA[1]. Those architectures take the output tokens from a (frozen) vision transformer and train a very small projection layer between the output token space of the ViT and the input space of the LLM. [0] https://arxiv.org/abs/2305.06500 https://arxiv.org/abs/2305.06500 [1] https://llava-vl.github.io/ https://llava-vl.github.io/
- 3abiton 3y agoI wonder when Dalle3 is released if OpenAi will release any technical documents about the integration with gpt4. Their approach might be similar.
- isaacfung 3y agoOpenAI used this approach for CLIP before blip and llava. CLIP is used to encode the text prompt in stable diffusion. Not sure about Dalle.