3 ms·

I should have added that it depends on the model. MiniLM didn’t exhibit this behavior, but it unfortunately didn’t perform as well on recall or ranking as other

I should have added that it depends on the model. MiniLM didn’t exhibit this behavior, but it unfortunately didn’t perform as well on recall or ranking as other models that did. GTE comes to mind. You can try the demo widget on HuggingFace and see this: https://huggingface.co/thenlper/gte-large https://huggingface.co/thenlper/gte-large As an example, against “Chocolate chip cookies”, “Oreos” has a cosine similarity of .808, while “Bubonic plague” is at .709.

refulgentis 3y ago
Amazing lol: for what it's worth, it's hugely, hugely, shocking to me how little anyone outside sentence-transformers bothers noting whether their embedding model is for asymmetric vs. symmetric search. i.e. in all seriousness, afaik, 90% of people are using the "wrong" embeddings, and either handwave about it, don't care, or think it's being picky. So of course I want to reach to blame that here. I'm doing a cross-platform app pipeline that basically search API => get pages => embed => run against prompt, so unfortunately I don't have much more insight. MiniLM V3 is good enough, the nearest constraint at this point is more "will I download all 10 search results in time?" than "am I grabbing the absolutely best passages?" -- good on you for legitimately testing those things. edit: actually, I do have one more thought: sometimes I end up with embeddings for the last part of a web page that match ~anything. i.e. imagine i have a 1005 word web page and split it into 250 word chunks to embed. Then I'll have 4 "full" vectors, and one that represents just 5 words. That 5 word tends to match almost ~anything, i.e. I'll see ones from previous search queries match the current search query. Maybe you're seeing noise because its just 3-5 words? But then again, you were probably just illustrating without splatting in huge chunks of texts.
- anon373839 3y ago
  That’s a good point! Actually I’ve been using these in the asymmetric search setting, and the queries are very short phrases (not full questions like you might see in a RAG application).

refulgentis 3y ago

Amazing lol: for what it's worth, it's hugely, hugely, shocking to me how little anyone outside sentence-transformers bothers noting whether their embedding model is for asymmetric vs. symmetric search. i.e. in all seriousness, afaik, 90% of people are using the "wrong" embeddings, and either handwave about it, don't care, or think it's being picky. So of course I want to reach to blame that here. I'm doing a cross-platform app pipeline that basically search API => get pages => embed => run against prompt, so unfortunately I don't have much more insight. MiniLM V3 is good enough, the nearest constraint at this point is more "will I download all 10 search results in time?" than "am I grabbing the absolutely best passages?" -- good on you for legitimately testing those things. edit: actually, I do have one more thought: sometimes I end up with embeddings for the last part of a web page that match ~anything. i.e. imagine i have a 1005 word web page and split it into 250 word chunks to embed. Then I'll have 4 "full" vectors, and one that represents just 5 words. That 5 word tends to match almost ~anything, i.e. I'll see ones from previous search queries match the current search query. Maybe you're seeing noise because its just 3-5 words? But then again, you were probably just illustrating without splatting in huge chunks of texts.

anon373839 3y ago
That’s a good point! Actually I’ve been using these in the asymmetric search setting, and the queries are very short phrases (not full questions like you might see in a RAG application).