4 ms·

Ive been playing with embeddings lately for document search based off queries/questions. this makes it seem like they work great, but it hasn't been very smooth

by tayo42 3y ago

Ive been playing with embeddings lately for document search based off queries/questions. this makes it seem like they work great, but it hasn't been very smooth for me. I kept running into people recommending against using them for long documents? Is openai better then the models sentenance-transformers uses? I found some recommendations to average together the embeddings of parts. I guess its cutting edge-ish still, a lot still feels like you can have a cool demo quickly, but something reliable and accurate is a lot of work?

simonw 3y ago
OpenAI's embedding model works up to 8,000 tokens. The sentence-transformers ones are mostly smaller than that, though there's a new model that just came out that can handle 8,000: https://huggingface.co/jinaai/jina-embeddings-v2-base-en https://huggingface.co/jinaai/jina-embeddings-v2-base-en People tend to "chunk" larger documents - the chunking strategy that's best is very dependent on what you are using the embeddings for. I've found it frustratingly hard to find really good guidance as to chunking strategies. I've had good results for Q&A chunking my blog content up into paragraph sized chunks, as described here: https://simonwillison.net/2023/Oct/23/embeddings/#answering-questions-with-retrieval-augmented-generation https://simonwillison.net/2023/Oct/23/embeddings/#answering-... - but I'm not ready to say that's a universally good practice.
- CityOfThrowaway 3y ago
  I'm also curious about how to think about chunking for Doc-to-Doc similarity. Say I have an input document that I want to use as my search query and a database of potential results documents. It's not clear to me that I should simply embed both the input document and each of the results documents. If the documents contain a variety of ideas, I'd be nervous they wind getting embedded into some generic space. But I don't have any good ideas as to what type of chunk-match-and-aggregate strategy might work well here. Would love ideas from folks that have done stuff like this!
  - deleted 3y ago
    [deleted]
  - Swizec 3y ago
    > If the documents contain a variety of ideas I've found chunking by sub-headings works really well. But here's a dirty secret I've learned as a writer: Your document can only ever contain 1 idea. That's the most that human readers can manage.
- llmllmllm 3y ago
  I have found success with chunking with 100 tokens, preceeded by the last 10 tokens of the previous chunk, and the first 10 tokens of the next chunk, 120 tokens total. I generate an embedding for each, then compare that to embedding(s) derived from the input query. How to generate embeddings from the input query well is where one's focus should be IMO. An example: "don't mention x" being turned into filtering out / de-emphasizing chunks that align with the embedding for x. I've been using these techniques along with pgvector and OpenAI's embeddings for https://flowch.ai https://flowch.ai and it works really well. A user uploads a document or uses the Chrome Extension on a webpage and FlowChai chunks up the content, generates embeddings, builds up a RAG context and then produces a report based on the user's prompt. I hope that helps show a real world example. You're welcome to play with FlowChai for free to see how it works in practice at the application level.
- tayo42 3y ago
  > I've found it frustratingly hard to find really good guidance as to chunking strategies. Atleast I'm not alone. Paragraph can make sense, I was worried about paragraphs that are related in concepts, they're not always standalone section of thoughts in real writing. Maybe if two paragraphs meet some similarity threshold you can merge them.
freediver 3y ago
If you are looking for lightweight, low- latency, fully local, end-to-end solution (chunking, embedding, storage and vector search), try vectordb [1] Just spent a day updating it with latest benchmarks for text embedding models. [1] https://github.com/kagisearch/vectordb https://github.com/kagisearch/vectordb
- tayo42 3y ago
  It looks like it does sliding window on tokens? That works well, you don't slice it in the middle of sentences?
jn2clark 3y ago
Try this https://github.com/marqo-ai/marqo https://github.com/marqo-ai/marqo which handles all the chunking for you (and is configurable). Also handles chunking of images in an analogous way. This enables highlighting in longer docs and also for images in a single retrieval step.

by tayo42 3y ago

simonw 3y ago

OpenAI's embedding model works up to 8,000 tokens. The sentence-transformers ones are mostly smaller than that, though there's a new model that just came out that can handle 8,000: https://huggingface.co/jinaai/jina-embeddings-v2-base-en https://huggingface.co/jinaai/jina-embeddings-v2-base-en People tend to "chunk" larger documents - the chunking strategy that's best is very dependent on what you are using the embeddings for. I've found it frustratingly hard to find really good guidance as to chunking strategies. I've had good results for Q&A chunking my blog content up into paragraph sized chunks, as described here: https://simonwillison.net/2023/Oct/23/embeddings/#answering-questions-with-retrieval-augmented-generation https://simonwillison.net/2023/Oct/23/embeddings/#answering-... - but I'm not ready to say that's a universally good practice.

CityOfThrowaway 3y ago
I'm also curious about how to think about chunking for Doc-to-Doc similarity. Say I have an input document that I want to use as my search query and a database of potential results documents. It's not clear to me that I should simply embed both the input document and each of the results documents. If the documents contain a variety of ideas, I'd be nervous they wind getting embedded into some generic space. But I don't have any good ideas as to what type of chunk-match-and-aggregate strategy might work well here. Would love ideas from folks that have done stuff like this!
- deleted 3y ago
  [deleted]
- Swizec 3y ago
  > If the documents contain a variety of ideas I've found chunking by sub-headings works really well. But here's a dirty secret I've learned as a writer: Your document can only ever contain 1 idea. That's the most that human readers can manage.
llmllmllm 3y ago
I have found success with chunking with 100 tokens, preceeded by the last 10 tokens of the previous chunk, and the first 10 tokens of the next chunk, 120 tokens total. I generate an embedding for each, then compare that to embedding(s) derived from the input query. How to generate embeddings from the input query well is where one's focus should be IMO. An example: "don't mention x" being turned into filtering out / de-emphasizing chunks that align with the embedding for x. I've been using these techniques along with pgvector and OpenAI's embeddings for https://flowch.ai https://flowch.ai and it works really well. A user uploads a document or uses the Chrome Extension on a webpage and FlowChai chunks up the content, generates embeddings, builds up a RAG context and then produces a report based on the user's prompt. I hope that helps show a real world example. You're welcome to play with FlowChai for free to see how it works in practice at the application level.
tayo42 3y ago
> I've found it frustratingly hard to find really good guidance as to chunking strategies. Atleast I'm not alone. Paragraph can make sense, I was worried about paragraphs that are related in concepts, they're not always standalone section of thoughts in real writing. Maybe if two paragraphs meet some similarity threshold you can merge them.

freediver 3y ago

If you are looking for lightweight, low- latency, fully local, end-to-end solution (chunking, embedding, storage and vector search), try vectordb [1] Just spent a day updating it with latest benchmarks for text embedding models. [1] https://github.com/kagisearch/vectordb https://github.com/kagisearch/vectordb

tayo42 3y ago
It looks like it does sliding window on tokens? That works well, you don't slice it in the middle of sentences?

jn2clark 3y ago

Try this https://github.com/marqo-ai/marqo https://github.com/marqo-ai/marqo which handles all the chunking for you (and is configurable). Also handles chunking of images in an analogous way. This enables highlighting in longer docs and also for images in a single retrieval step.