3 ms·

To be fair… semantic similarity isn’t the same as relevance either. They are related, and we frequently assume they are close enough that it doesn’t matter, bu

by spott 3y ago

To be fair… semantic similarity isn’t the same as relevance either. They are related, and we frequently assume they are close enough that it doesn’t matter, but they are different.

brigadier132 3y ago
I disagree, the embeddings are what are used by the llms themselves to produce relevant output and the output is relevant ergo the embeddings do produce relevant output via similarity search
- viscanti 3y ago
  Not if you're using ANN. In some cases that will be very similar to exhaustive search but in other cases you'll get results that you don't want. You also need embeddings that distribute things mostly evenly across the embedding space (not all will).
- spott 3y ago
  You probably aren’t using an LLM for your text embeddings for document retrieval (they don’t perform as well as specialist embedding models[0]), and even if they did, you have an embedding about a bare document, without any context of what you are trying to get out of it. If you were to add your context in and then get an embedding, you would get a different answer. As your query gets specific, irrelevant aspects of the embedding space can overwhelm the similarity function, leading to irrelevant answers that are still semantically similar. [0] https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard
  - regularfry 3y ago
    The recent SILO-LM paper has a slightly different approach: rather than using input embeddings and prompting the LLM with documents, it searches the database according to the LLM's output embedding and uses KNN search to skew the output embedding vector before token generation. Done that way round, using LLM embeddings outperforms RAG, allegedly. They did it with a custom language model. I really want to give this a try with llama2 embeddings but haven't had the bandwidth yet (and llama2's embedding vectors are inconveniently huge, but that's a different problem).
    - spott 3y ago
      Interesting! I’ll have to look into that.
- viraptor 3y ago
  Consider the extreme case: when I ask a question about X, then a page with just the questions about X will get the highest similarity. But what I want in terms of relevance for the answer is a page with a little bit about X and lots of surrounding context that answers the question. By definition the extra context will likely lower the similarity.
petra 3y ago
That's interesting. Are there any good sources to learn more about that?

by spott 3y ago

To be fair… semantic similarity isn’t the same as relevance either. They are related, and we frequently assume they are close enough that it doesn’t matter, but they are different.

brigadier132 3y ago

I disagree, the embeddings are what are used by the llms themselves to produce relevant output and the output is relevant ergo the embeddings do produce relevant output via similarity search

viscanti 3y ago
Not if you're using ANN. In some cases that will be very similar to exhaustive search but in other cases you'll get results that you don't want. You also need embeddings that distribute things mostly evenly across the embedding space (not all will).
spott 3y ago
You probably aren’t using an LLM for your text embeddings for document retrieval (they don’t perform as well as specialist embedding models[0]), and even if they did, you have an embedding about a bare document, without any context of what you are trying to get out of it. If you were to add your context in and then get an embedding, you would get a different answer. As your query gets specific, irrelevant aspects of the embedding space can overwhelm the similarity function, leading to irrelevant answers that are still semantically similar. [0] https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard
- regularfry 3y ago
  The recent SILO-LM paper has a slightly different approach: rather than using input embeddings and prompting the LLM with documents, it searches the database according to the LLM's output embedding and uses KNN search to skew the output embedding vector before token generation. Done that way round, using LLM embeddings outperforms RAG, allegedly. They did it with a custom language model. I really want to give this a try with llama2 embeddings but haven't had the bandwidth yet (and llama2's embedding vectors are inconveniently huge, but that's a different problem).
  - spott 3y ago
    Interesting! I’ll have to look into that.
viraptor 3y ago
Consider the extreme case: when I ask a question about X, then a page with just the questions about X will get the highest similarity. But what I want in terms of relevance for the answer is a page with a little bit about X and lots of surrounding context that answers the question. By definition the extra context will likely lower the similarity.

petra 3y ago

That's interesting. Are there any good sources to learn more about that?