5 ms·

These are extremely useful embedding models, and some are small enough to use in the frontend (with transformers.js) for on-device semantic search. One issue

These are extremely useful embedding models, and some are small enough to use in the frontend (with transformers.js) for on-device semantic search. One issue I’ve run into is that they produce a very good ranking, but the actual cosine similarity scores appear to be meaningless. (E.g., “chocolate chip cookies” and “PLS6;YJBXSRF&/” could have a similarity of 0.8.) Consequently, I’ve had a hard time selecting sensible cutoff values to balance recall and precision. Has anyone found a good approach to this?

ilaksh 3y ago
maybe the normalize_embeddings flag on encode?
- Clueed 3y ago
  Not sure about the specific implementation here but the very definition of cosine similarity includes normalization. [0] [0] https://en.wikipedia.org/wiki/Cosine_similarity https://en.wikipedia.org/wiki/Cosine_similarity
eigenvalue 3y ago
Yes, check out my library for vector similarity that has various other measures which are more discriminative: https://github.com/Dicklesworthstone/fast_vector_similarity https://github.com/Dicklesworthstone/fast_vector_similarity pip install fast_vector_similarity
- visarga 3y ago
  great resource
- Havoc 3y ago
  That’s cool. Thanks!
refibrillator 3y ago
Even if you choose a more sophisticated similarity measure as suggested by another commenter, you’ll still need to set a threshold on that metric to perform binary classification. In my experience there are two paths forward, the one I recommend is to train an MLP classifier on the embeddings to produce a binary classification (ie similar or not). The advantage is that you no longer need to set a numeric threshold on a distance metric, however you will need labeled training data to define what is “similar” in the context of your use case. The other path is to calculate the statistics of pair-wise distances for every record in some unlabeled dataset you have, and use the resulting distribution to inform choice of threshold. This will at least give you an estimate of what % of records will be classified as similar for a given threshold value.
- whakim 3y ago
  I’ve had a lot of success training a classifier on top of these embeddings. An MLP works well, as does an SVM. If you don’t have labeled data, I’ve also found that various active learning techniques can produce extremely strong results while only requiring a small amount of human labeling.
  - abhgh 3y ago
    I second the point about learning a classifier over them - it is practically quite useful. But I'd caution against believing Active Learning (AL) would be unequivocally be useful; most positive performances seem to arise in very specific circumstances [1]. Esp. see Table 1: a Linear SVM with MPNet (one of the embeddings sbert supports) has indeed strong performance, but, in general, random sampling (over an AL strategy) performs quite well! [1] https://arxiv.org/pdf/2403.15744v1.pdf https://arxiv.org/pdf/2403.15744v1.pdf
    - whakim 3y ago
      Extremely interesting paper that runs counter to my experience. In my previous role I ran tens of thousands of classification pipelines similar to those described in the paper, using bert-based models and then running SVMs on top as classifiers. In almost all cases AL techniques such as Least Certainty and Query by Committee dramatically outperformed random sampling. Reading the paper a bit more closely, the two differences that stand out are the use of multiclass classifiers (we transformed multiclass classification problems into a number of binary classification problems in order to improve performance), and the relative simplicity of the datasets used in the paper. That being said, anyone who is doing AL should of course first consider model performance with a randomly-selected subsample of the data!
pietz 3y ago
I haven't noticed this behavior in such extreme ways as you. I've worked with many different embeddings using this library and the OpenAI API for data normalization, extraction and structuring tasks. On one hand it's super impressive what you can get out of these embeddings. On the other I'm with you that there seems to be a missing piece to make them work great. An additional MLP or Siamese Network works really well but it seems like there should be something easier and unsupervised given your set of vectorized samples. Thinking out loud, could PCA help with this by centering the data and removing dimensions that mostly model noise?
- anon373839 3y ago
  > I haven't noticed this behavior in such extreme ways as you. I should have clarified that it really depends on the model. I have had this problem to a greater extent with the GTE and BGE embeddings. But I use them anyway, because they’re so strong overall. PCA is an interesting idea, and worth looking at.
  - whakim 3y ago
    I've tried PCA (and perhaps more usefully, UMAP) and while can be great for stuff like unsupervised learning, I think the trick is to choose a classifier (as per my comment above) that can do a good job with high-dimensional data.
YossarianFrPrez 3y ago
Er, is it possible that you are using `scipy.spatial.distance.cosine` to compute the similarity? If so, note that this computes the cosine distance, and the cosine similarity is defined as 1-cosine distance. I tried out your example using the following code: from sentence_transformers import SentenceTransformer import scipy.spatial as ssp model = SentenceTransformer("all-mpnet-base-v2") A = model.encode(['chocolate chip cookies','PLS6;YJBXSRF&/']) CosineDistance = ssp.distance.cosine(A[0],A[1]) Where `CosineDistance == 0.953` This means the model is actually working quite well, were these to be similar to each other we'd expect CosineDistance to be much closer to 0. The other comments about such distances being useful for relative comparisons also apply: I've used SentenceTransformers quite successfully for nearest-neighbor searches.
refulgentis 3y ago
Something is off if you're seeing behavior like that, 0.1-0.15 with MiniLM L6 V3 are good for "has any relevancy whatsoever"
- anon373839 3y ago
  I should have added that it depends on the model. MiniLM didn’t exhibit this behavior, but it unfortunately didn’t perform as well on recall or ranking as other models that did. GTE comes to mind. You can try the demo widget on HuggingFace and see this: https://huggingface.co/thenlper/gte-large https://huggingface.co/thenlper/gte-large As an example, against “Chocolate chip cookies”, “Oreos” has a cosine similarity of .808, while “Bubonic plague” is at .709.
  - refulgentis 3y ago
    Amazing lol: for what it's worth, it's hugely, hugely, shocking to me how little anyone outside sentence-transformers bothers noting whether their embedding model is for asymmetric vs. symmetric search. i.e. in all seriousness, afaik, 90% of people are using the "wrong" embeddings, and either handwave about it, don't care, or think it's being picky. So of course I want to reach to blame that here. I'm doing a cross-platform app pipeline that basically search API => get pages => embed => run against prompt, so unfortunately I don't have much more insight. MiniLM V3 is good enough, the nearest constraint at this point is more "will I download all 10 search results in time?" than "am I grabbing the absolutely best passages?" -- good on you for legitimately testing those things. edit: actually, I do have one more thought: sometimes I end up with embeddings for the last part of a web page that match ~anything. i.e. imagine i have a 1005 word web page and split it into 250 word chunks to embed. Then I'll have 4 "full" vectors, and one that represents just 5 words. That 5 word tends to match almost ~anything, i.e. I'll see ones from previous search queries match the current search query. Maybe you're seeing noise because its just 3-5 words? But then again, you were probably just illustrating without splatting in huge chunks of texts.
    - anon373839 3y ago
      That’s a good point! Actually I’ve been using these in the asymmetric search setting, and the queries are very short phrases (not full questions like you might see in a RAG application).
fermisea 3y ago
Bootstrap a pvalue by creating a set of thousands of random words, calculate the metric for those and either explicitly keep these numbers to compute the rank, or fit some normal distribution and use this mean and std to estimate the probability that it's similar.

ilaksh 3y ago

maybe the normalize_embeddings flag on encode?

Clueed 3y ago
Not sure about the specific implementation here but the very definition of cosine similarity includes normalization. [0] [0] https://en.wikipedia.org/wiki/Cosine_similarity https://en.wikipedia.org/wiki/Cosine_similarity

eigenvalue 3y ago

Yes, check out my library for vector similarity that has various other measures which are more discriminative: https://github.com/Dicklesworthstone/fast_vector_similarity https://github.com/Dicklesworthstone/fast_vector_similarity pip install fast_vector_similarity

visarga 3y ago
great resource
Havoc 3y ago
That’s cool. Thanks!

refibrillator 3y ago

Even if you choose a more sophisticated similarity measure as suggested by another commenter, you’ll still need to set a threshold on that metric to perform binary classification. In my experience there are two paths forward, the one I recommend is to train an MLP classifier on the embeddings to produce a binary classification (ie similar or not). The advantage is that you no longer need to set a numeric threshold on a distance metric, however you will need labeled training data to define what is “similar” in the context of your use case. The other path is to calculate the statistics of pair-wise distances for every record in some unlabeled dataset you have, and use the resulting distribution to inform choice of threshold. This will at least give you an estimate of what % of records will be classified as similar for a given threshold value.

whakim 3y ago
I’ve had a lot of success training a classifier on top of these embeddings. An MLP works well, as does an SVM. If you don’t have labeled data, I’ve also found that various active learning techniques can produce extremely strong results while only requiring a small amount of human labeling.
- abhgh 3y ago
  I second the point about learning a classifier over them - it is practically quite useful. But I'd caution against believing Active Learning (AL) would be unequivocally be useful; most positive performances seem to arise in very specific circumstances [1]. Esp. see Table 1: a Linear SVM with MPNet (one of the embeddings sbert supports) has indeed strong performance, but, in general, random sampling (over an AL strategy) performs quite well! [1] https://arxiv.org/pdf/2403.15744v1.pdf https://arxiv.org/pdf/2403.15744v1.pdf
  - whakim 3y ago
    Extremely interesting paper that runs counter to my experience. In my previous role I ran tens of thousands of classification pipelines similar to those described in the paper, using bert-based models and then running SVMs on top as classifiers. In almost all cases AL techniques such as Least Certainty and Query by Committee dramatically outperformed random sampling. Reading the paper a bit more closely, the two differences that stand out are the use of multiclass classifiers (we transformed multiclass classification problems into a number of binary classification problems in order to improve performance), and the relative simplicity of the datasets used in the paper. That being said, anyone who is doing AL should of course first consider model performance with a randomly-selected subsample of the data!

pietz 3y ago

I haven't noticed this behavior in such extreme ways as you. I've worked with many different embeddings using this library and the OpenAI API for data normalization, extraction and structuring tasks. On one hand it's super impressive what you can get out of these embeddings. On the other I'm with you that there seems to be a missing piece to make them work great. An additional MLP or Siamese Network works really well but it seems like there should be something easier and unsupervised given your set of vectorized samples. Thinking out loud, could PCA help with this by centering the data and removing dimensions that mostly model noise?

anon373839 3y ago
> I haven't noticed this behavior in such extreme ways as you. I should have clarified that it really depends on the model. I have had this problem to a greater extent with the GTE and BGE embeddings. But I use them anyway, because they’re so strong overall. PCA is an interesting idea, and worth looking at.
- whakim 3y ago
  I've tried PCA (and perhaps more usefully, UMAP) and while can be great for stuff like unsupervised learning, I think the trick is to choose a classifier (as per my comment above) that can do a good job with high-dimensional data.

YossarianFrPrez 3y ago

Er, is it possible that you are using `scipy.spatial.distance.cosine` to compute the similarity? If so, note that this computes the cosine distance, and the cosine similarity is defined as 1-cosine distance. I tried out your example using the following code: from sentence_transformers import SentenceTransformer import scipy.spatial as ssp model = SentenceTransformer("all-mpnet-base-v2") A = model.encode(['chocolate chip cookies','PLS6;YJBXSRF&/']) CosineDistance = ssp.distance.cosine(A[0],A[1]) Where `CosineDistance == 0.953` This means the model is actually working quite well, were these to be similar to each other we'd expect CosineDistance to be much closer to 0. The other comments about such distances being useful for relative comparisons also apply: I've used SentenceTransformers quite successfully for nearest-neighbor searches.

refulgentis 3y ago

Something is off if you're seeing behavior like that, 0.1-0.15 with MiniLM L6 V3 are good for "has any relevancy whatsoever"

anon373839 3y ago
I should have added that it depends on the model. MiniLM didn’t exhibit this behavior, but it unfortunately didn’t perform as well on recall or ranking as other models that did. GTE comes to mind. You can try the demo widget on HuggingFace and see this: https://huggingface.co/thenlper/gte-large https://huggingface.co/thenlper/gte-large As an example, against “Chocolate chip cookies”, “Oreos” has a cosine similarity of .808, while “Bubonic plague” is at .709.
- refulgentis 3y ago
  Amazing lol: for what it's worth, it's hugely, hugely, shocking to me how little anyone outside sentence-transformers bothers noting whether their embedding model is for asymmetric vs. symmetric search. i.e. in all seriousness, afaik, 90% of people are using the "wrong" embeddings, and either handwave about it, don't care, or think it's being picky. So of course I want to reach to blame that here. I'm doing a cross-platform app pipeline that basically search API => get pages => embed => run against prompt, so unfortunately I don't have much more insight. MiniLM V3 is good enough, the nearest constraint at this point is more "will I download all 10 search results in time?" than "am I grabbing the absolutely best passages?" -- good on you for legitimately testing those things. edit: actually, I do have one more thought: sometimes I end up with embeddings for the last part of a web page that match ~anything. i.e. imagine i have a 1005 word web page and split it into 250 word chunks to embed. Then I'll have 4 "full" vectors, and one that represents just 5 words. That 5 word tends to match almost ~anything, i.e. I'll see ones from previous search queries match the current search query. Maybe you're seeing noise because its just 3-5 words? But then again, you were probably just illustrating without splatting in huge chunks of texts.
  - anon373839 3y ago
    That’s a good point! Actually I’ve been using these in the asymmetric search setting, and the queries are very short phrases (not full questions like you might see in a RAG application).

fermisea 3y ago

Bootstrap a pvalue by creating a set of thousands of random words, calculate the metric for those and either explicitly keep these numbers to compute the rank, or fit some normal distribution and use this mean and std to estimate the probability that it's similar.