4 ms·
"Query-Document Mismatch: This model assumes that query embedding and the content embedding are similar in the embedding space, which is not always true based o
by simonw 3y ago
"Query-Document Mismatch: This model assumes that query embedding and the content embedding are similar in the embedding space, which is not always true based on the text you're trying to search over."
There are embeddings models that take this into account, which are pretty fascinating.
I've been exploring https://huggingface.co/intfloat/e5-large-v2 https://huggingface.co/intfloat/e5-large-v2 which lets you calculate two different types of embeddings in the same space. Example from their README:
passage: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day
query: how much protein should a female eat
You can then build your embedding database out of "passage: " embeddings, then run "query: " embeddings against it to try and find passages that can answer the question.
I've had pretty great initial results trying that out against paragraphs from my blog: https://til.simonwillison.net/llms/embed-paragraphs#user-content-answering-a-question https://til.simonwillison.net/llms/embed-paragraphs#user-con...
This won't help address other challenges mentioned in that post, like "what problems did we fix last week?" - but it's still a useful starting point.
- deleted 3y ago[deleted]
- bthomas 3y agoNeat! Do you happen to have the analogous similarity queries with a default embedding? Curious to see them side by side. (I know I can reproduce myself and I appreciate all the code you posted there - thought I'd ask first!)
- simonw 3y agoNo, I haven't been disciplined enough to have good examples for that yet. One of my goals right now is to put together a solid RAG system based on top of LLM and Datasette that makes it really easy to compare different embedding models, chunking strategies and prompts to figure out what works best - but that's still just an idea in my head at the moment.