3 ms·
>but because embeddings search orders by similarity score it will ALWAYS return results, really scraping the bottom of the barrel if it has to Why not have a s
by kgeist 2y ago
>but because embeddings search orders by similarity score it will ALWAYS return results, really scraping the bottom of the barrel if it has to
Why not have a similarity threshold? Say, if the distance is below 0.7, do not accept the search result.
- simonw 2y agoIt turns out picking that threshold is extremely difficult - I've tried! The value seems to differ for different searches, so picking eg 0.7 as a fixed value isn't actually as useful as you would expect.
- zmccormick7 2y agoAgreed that thresholds don't work when applied to the cosine similarity of embeddings. But I have found that the similarity score returned by high-quality rerankers, especially Cohere, are consistent and meaningful enough that using a threshold works well there.
- kgeist 2y agoI use similarity threshold (to remove absolutely irrelevant results) and then use a reranker to get Top N.
- jairuhme 2y agoI'll add to what the other commenter noted, but sometimes the difference between results get very granular (i.e. .65789 vs .65788) so deciding on where that threshold should be is little trickier.