5 ms·

The first unstated assumption is that similar vectors are relevant documents, and for many use cases that's just not true. Cosine similarity != relevance. So if

The first unstated assumption is that similar vectors are relevant documents, and for many use cases that's just not true. Cosine similarity != relevance. So if your pipeline pulls 2 or 4 or 12 document chunks into the LLM's context, and half or more of them aren't relevant, does this make the LLM's response more or less relevant? The second unstated assumption is that the vector index can accurately identify the top K vectors by cosine similarity, and that's not true either. If you retrieve the top K vectors according to the vector index (instead of computing all the pairwise similarities in advance), that set of 10 vectors will be missing documents that have a higher cosine similarity than that of the K'th vector retrieved. All of this means you'll need to retrieve a multiple of K vectors, figure out some way to re-rank them to exclude the irrelevant ones, and have your own ground truth to measure the index's precision and recall.

NhanH 3y ago
Could you please explain a bit on your 2nd paragraph. I couldn’t quite understand either the problem statement nor the reasoning itself.
Jimmc414 3y ago
Switching to Word2Vec embeddings led to a substantial improvement in my cosine similarity evaluations for text similarity, but granted I was looking for actual similarity, not relevance. I tried many different methods and had lots of mediocre results initially. code: https://github.com/jimmc414/document_intelligence/blob/main/text_similarity.py https://github.com/jimmc414/document_intelligence/blob/main/... https://github.com/jimmc414/document_intelligence https://github.com/jimmc414/document_intelligence
- bugglebeetle 3y ago
  As opposed to sentencebert or what?
  - Jimmc414 3y ago
    DistilBERT and RoBERTa
- Nowado 3y ago
  Interesting, do you happen to have some quantitative results on this/additional insights/etc? I've interpreted transformer vector similarity as 'likelihood to be followed by the same thing' which is close to word2vec's 'sum of likelihoods of all words to be replaced by the other set' (kinda), but also very different in some contexts.
  - BoorishBears 3y ago
    There's no simplified definition like that, vectors can even capture logical properties, it's all down to what the model was tuned for: https://www.sbert.net/examples/training/nli/README.html https://www.sbert.net/examples/training/nli/README.html
- sandGorgon 3y ago
  this is very interesting. you had better results here than the openai ada02 and other embeddings like bge ?
brigadier132 3y ago
The vectors are literally constructed so that cosine similarity is semantic similarity. > second unstated assumption is that the vector index can accurately identify the top K vectors by cosine similarity, and that's not true either Its not unstated, its called ANN for a reason
- spott 3y ago
  To be fair… semantic similarity isn’t the same as relevance either. They are related, and we frequently assume they are close enough that it doesn’t matter, but they are different.
  - brigadier132 3y ago
    I disagree, the embeddings are what are used by the llms themselves to produce relevant output and the output is relevant ergo the embeddings do produce relevant output via similarity search
    - viscanti 3y ago
      Not if you're using ANN. In some cases that will be very similar to exhaustive search but in other cases you'll get results that you don't want. You also need embeddings that distribute things mostly evenly across the embedding space (not all will).
    - spott 3y ago
      You probably aren’t using an LLM for your text embeddings for document retrieval (they don’t perform as well as specialist embedding models[0]), and even if they did, you have an embedding about a bare document, without any context of what you are trying to get out of it. If you were to add your context in and then get an embedding, you would get a different answer. As your query gets specific, irrelevant aspects of the embedding space can overwhelm the similarity function, leading to irrelevant answers that are still semantically similar. [0] https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard
saliagato 3y ago
Azure Cognitive Search takes care of all of this combining semantic search with other layers of traditional search methods
choppaface 3y ago
"Cosine similarity != relevance" In all ML search products, there's a tradeoff between precision and recall, and moreover there's almost never any "gold" data that ensures the "correctness" of surfaced results. I mean, Bing and Google have both invested millions of dollars in labeling web pages and even evaluating search results, but those labels can become useless as your set of documents change. Cosine similar is a useful compromise and yes a lot of authors take this for granted. At the end of the day, an LLM product probably won't be evaluated on accuracy but rather "lift" over an alternative. And the evaluation will be in units of user happiness. > All of this means you'll need to retrieve a multiple of K vectors, figure out some way to re-rank them to exclude the irrelevant ones, and have your own ground truth to measure the index's precision and recall. This is usually a Series E problem, not a Series A problem.

NhanH 3y ago

Could you please explain a bit on your 2nd paragraph. I couldn’t quite understand either the problem statement nor the reasoning itself.

Jimmc414 3y ago

Switching to Word2Vec embeddings led to a substantial improvement in my cosine similarity evaluations for text similarity, but granted I was looking for actual similarity, not relevance. I tried many different methods and had lots of mediocre results initially. code: https://github.com/jimmc414/document_intelligence/blob/main/text_similarity.py https://github.com/jimmc414/document_intelligence/blob/main/... https://github.com/jimmc414/document_intelligence https://github.com/jimmc414/document_intelligence

bugglebeetle 3y ago
As opposed to sentencebert or what?
- Jimmc414 3y ago
  DistilBERT and RoBERTa
Nowado 3y ago
Interesting, do you happen to have some quantitative results on this/additional insights/etc? I've interpreted transformer vector similarity as 'likelihood to be followed by the same thing' which is close to word2vec's 'sum of likelihoods of all words to be replaced by the other set' (kinda), but also very different in some contexts.
- BoorishBears 3y ago
  There's no simplified definition like that, vectors can even capture logical properties, it's all down to what the model was tuned for: https://www.sbert.net/examples/training/nli/README.html https://www.sbert.net/examples/training/nli/README.html
sandGorgon 3y ago
this is very interesting. you had better results here than the openai ada02 and other embeddings like bge ?

brigadier132 3y ago

The vectors are literally constructed so that cosine similarity is semantic similarity. > second unstated assumption is that the vector index can accurately identify the top K vectors by cosine similarity, and that's not true either Its not unstated, its called ANN for a reason

spott 3y ago
To be fair… semantic similarity isn’t the same as relevance either. They are related, and we frequently assume they are close enough that it doesn’t matter, but they are different.
- brigadier132 3y ago
  I disagree, the embeddings are what are used by the llms themselves to produce relevant output and the output is relevant ergo the embeddings do produce relevant output via similarity search
  - viscanti 3y ago
    Not if you're using ANN. In some cases that will be very similar to exhaustive search but in other cases you'll get results that you don't want. You also need embeddings that distribute things mostly evenly across the embedding space (not all will).
  - spott 3y ago
    You probably aren’t using an LLM for your text embeddings for document retrieval (they don’t perform as well as specialist embedding models[0]), and even if they did, you have an embedding about a bare document, without any context of what you are trying to get out of it. If you were to add your context in and then get an embedding, you would get a different answer. As your query gets specific, irrelevant aspects of the embedding space can overwhelm the similarity function, leading to irrelevant answers that are still semantically similar. [0] https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard

saliagato 3y ago

Azure Cognitive Search takes care of all of this combining semantic search with other layers of traditional search methods

choppaface 3y ago

"Cosine similarity != relevance" In all ML search products, there's a tradeoff between precision and recall, and moreover there's almost never any "gold" data that ensures the "correctness" of surfaced results. I mean, Bing and Google have both invested millions of dollars in labeling web pages and even evaluating search results, but those labels can become useless as your set of documents change. Cosine similar is a useful compromise and yes a lot of authors take this for granted. At the end of the day, an LLM product probably won't be evaluated on accuracy but rather "lift" over an alternative. And the evaluation will be in units of user happiness. > All of this means you'll need to retrieve a multiple of K vectors, figure out some way to re-rank them to exclude the irrelevant ones, and have your own ground truth to measure the index's precision and recall. This is usually a Series E problem, not a Series A problem.