8 ms·

Breaking up is hard to do: Chunking in RAG applications

143 points by meysamazad 2y ago

seabass 2y ago
When they recommend using smaller chunks, how small in general are we talking? One sentence? One paragraph? 100 tokens or 1000? While I understand that the context of the data is important, it’s hard for me to ground vague statements like that without a concrete realistic example. I’m curious what chunk sizes people have found the most success with for various tasks in the wild.
- homarp 2y ago
  think this way: you are going to return the chunk to your user as 'proof'. the size therefore depends of your content style. E.g. for a HN discussion, I would go 'paragraph' of each comment. in a contract, each clause. in a non-fiction book, maybe each section... you can also decide to do some kind of reverse adaptive tree: you chunk at sentences level, then compare, if 'close enough', you merge them into a bigger chunk
  - seabass 2y ago
    Suppose you don't know the shape of the content ahead of time. This would be the case for most apps that allow users to upload their own sources. It sounds like (at the expense of more computation and time) the reverse adaptive tree approach you described would be ideal for those scenarios.
    - homarp 2y ago
      that's why the article talks about using machine learning to decide of best strategy: to deal with unknown content 'shape'
- leobg 2y ago
  Why not overlapping sizes? (1), (1,2), (1,2,3) Sometimes the match is in a single sentence. Sometimes in the full paragraph. Sometimes across two paragraphs. If, in your top n results, some of these items overlap, you use the greater unit. And you slide these windows. Also, I wouldn’t necessarily use a “sentence” as the lower bound, since that can be something like “Yes.”
- CuriouslyC 2y ago
  When using vector search, you want chunks in the ~1k tokens range. If you're using full text search then chunks should probably be smaller, say a few paragraphs at most. If you use trigrams, you want chunks to be short, maybe even sentence level.
- laborcontract 2y ago
  I’ll share my strategy. I usually keep chunks at the max of 0.5% of a document or 270 tokens. Multiple that by three and those are the size of the sliding windows that are then worried.
bob1029 2y ago
Has anyone tried skipping the embedding path in favor of combining a proper FTS engine with an LLM yet? A tool like Lucene seems far more competent at the task of "find most relevant text fragment" compared to what is realized in a typical vector search application today. I'd also argue that you get more inspectability and control this way. You could even manage the preferred size of the fragments on a per-document basis using an entirely separate heuristic at indexing time. The semantic capabilities of vector search seem nice, but could you not achieve a similar outcome by using the LLM to project synonymous OR clauses into the FTS query based upon static background material or prior search iteration(s)?
- lmeyerov 2y ago
  Generally you do both and rank the combined results This is also why generally you don't get a vectordb and instead just add a vector index to the DB you are already using Re:semantic vs synonym, that's really domain dependent. As the number of hits go up, and queries get more interesting, the more vectors get interesting. At the same time, vectors are heavy, so there's also the question of pushing the semantic aspect to the ranker vs the index, but I don't see that discussed much (search & storage vs compute & latency)
  - bob1029 2y ago
    > There's also the question of pushing the semantic aspect to the ranker vs the index Could it make sense to perform dynamic vector lookup over the FTS result set best fragments? This could save a lot of money if you have a massive corpus to index because you'd only be paying to embed things that are being searched for at runtime. Focusing on just the best fragments could also improve the SnR going into the final vector search phase, especially if the fragment length is managed appropriately for each kind of document. If we are dealing with a method from a codebase, then we might prefer to have an unlimited fragment length. For a 20 megabyte PDF, it could be closer to the size of a tweet.
- woodson 2y ago
  Query expansion has been done since forever, long before even word2vec and other semantic embedding models, e.g. using WordNet (not a DNN, despite its name), see https://lucene.apache.org/core/3_3_0/api/contrib-wordnet/org/apache/lucene/wordnet/package-summary.html https://lucene.apache.org/core/3_3_0/api/contrib-wordnet/org....
diptanu 2y ago
We use grid search to figure out what's the best chunking strategy to use. Create a bunch of different strategies such as recursive chunking, semantic, etc, and parameterize them and see which one works best. The "best" chunking strategy depends on the nature of the documents and the questions being asked.
stoicjumbotron 2y ago
What would be the chunking strategy for q&a pairs? Right now I'm embedding the complete question and answer but the query results are not good as the response contains data not related to the question at all.
- harpastum 2y ago
  It depends on the specifics of your format, but we’ve had success embedding the questions and answers separately. If either match, you return the complete question and answer text. Make sure to deduplicate before returning, in case both match.
  - stoicjumbotron 2y ago
    I'll try embedding it separately as well and try to figure out from there. Thanks for the suggestion
- laborcontract 2y ago
  Join q and a when vectorizing them. Questions alone are too short to carry a lot of semantic richness. When you get a query, you then run two semantic search queries: one using the original question and one using a HYDE version of the question. Take those results and run it through cohere’s rerank.
  - stoicjumbotron 2y ago
    Yes, right now I'm vectorizing them as a pair. I'm then running a query in Pinecone by embedding the exact question but still the result does not have the actual q&a pair. I'm not familiar with HYDE version. I'll check it out. Thanks for the suggestion
constantinum 2y ago
with ever increasing context size — Claude 3 Sonnet, 204,800 token context size, 500 page document — sometimes not opting for chunking but at the same time optimizing for costs and latency might be a better solution, strategies like: Summary extraction and single pass extraction works well[1] [1] - https://docs.unstract.com/editions/cloud_edition#summary-extraction-and-single-pass-extraction-reduce-cost-by-up-to-6x-per-document-extraction https://docs.unstract.com/editions/cloud_edition#summary-ext...
- J_Shelby_J 2y ago
  In theory. In practice, context size degrades performance.
J_Shelby_J 2y ago
The problem with most chunking schemes I’ve seen is they’re naive. They don’t care about content; only token count. That’s fine, but what would be best is to chunk by topic. I’m currently trying to implement chunking by topic using an LLM. It’s much slower, but I hope it will be a huge win in retrieval accuracy. First step is to extract topics from the document by asking the LLM to identify all topics, and then split by sentences and feed each sentence to the LLM to identify the topic. I’m hoping the result will be the original text, split by topics. From there, they can be further chunked if needed. Of course, it could be done by just one shot asking the LLM to summarize each topic in the document, but the more the LLM is relied on to write, the more distortion is introduced. Retaining the original text is the goal and the LLM should just be used for decision making. Here is the crate I’m working out of. The chunking hasn’t been pushed, but you can see the decision making workflows. https://github.com/ShelbyJenkins/llm_client https://github.com/ShelbyJenkins/llm_client
- Swizec 2y ago
  > They don’t care about content; only token count. That’s fine, but what would be best is to chunk by topic. I’ve had a lot of success with chunking documents by subtitle. Works especially well for web published documents because each section tends to be fairly short.
- serjester 2y ago
  You’re basically reinventing embeddings. Using an LLM for this is overkill since capturing the meaning of a sentence can be done with a model 1000x smaller. I have a chunking library that does something similar to this, and there’s actually quite a few different libraries that have implemented some variant of “semantic chunking”. [1] https://github.com/Filimoa/open-parse https://github.com/Filimoa/open-parse
  - nostrebored 2y ago
    Excellent, now show me a single one working in production with various docs. I’ve talked through this problem with dozens of startups working on RAG, and everyone has the same problem. LLMs are arguably not reinventing embeddings if you’re using them to infer the structure of a broad document. Understanding the “external structure” of the document, the internal structure of each segment, and the semantic structure of each chunk is important.
serjester 2y ago
As embedding models become more performant and context windows increase, “ideal chunking” becomes less relevant. Cost isn’t as important to us, so we use small chunks and then just pull in the page before and after. If you do this on 20+ matches (since you’re decomposing the query multiple times), you’re very likely finding the content. Queries can get more expensive but you’re getting a corpus of “great answers” to test against as you refine your approach. Model costs are also plummeting which makes brute forcing it more and more viable.
- emporas 2y ago
  Bigger context windows don't work as well as advertised. The "hay in a haystack" problem is not solved yet. Also bigger context windows mean a lot more time waiting for an answer. Given the quadratic nature of context windows, we are stuck using transformers in smaller chunks. Other architectures like Mamba may solve that, but even then, increases in context window accuracy are not 1000x.
philbo 2y ago
Just as important as chunking strategy, imo, is recombination. After you've picked the highest-scoring chunks from the search matter, how do you glue them back together? Simple concatenation is the obvious way, but I've had better results from using the chunks as anchors for expansion. Use something like Wilson score interval to decide how many high-scoring chunks to pick, then grab the surrounding text for each chunk up to some semantic boundary like period or newline (or arbitrary limit if no boundary is detected).
antman 2y ago
tldr: Use adaptive chunking methods, where ML decides if consecutive texts are relevant. Does not mention any actual adaptive chunking method name
chiccomagnus 2y ago
Well-written article, missing key considerations: - Titles matter, a lot: if you add the title of the section at the start of each chunk you will get 10x better embeddings and so more accurate results. - The size doesn't matter: It depends on the combination of the layout and semantics of the content. - Avoid garbage in / out: increased context windows don't mean you can put trash inside them. The more good you are at putting relevant information the more precise answers you get. Especially for enterprise-grade solutions, this is so important. There are good emerging API solutions that implement semantic + layout-based chunking, which in my opinion is the best chunking strategy for PDF / Office files (the widest use case scenario for enterprises).

143 points by meysamazad 2y ago

When they recommend using smaller chunks, how small in general are we talking? One sentence? One paragraph? 100 tokens or 1000? While I understand that the context of the data is important, it’s hard for me to ground vague statements like that without a concrete realistic example. I’m curious what chunk sizes people have found the most success with for various tasks in the wild.

homarp 2y ago
think this way: you are going to return the chunk to your user as 'proof'. the size therefore depends of your content style. E.g. for a HN discussion, I would go 'paragraph' of each comment. in a contract, each clause. in a non-fiction book, maybe each section... you can also decide to do some kind of reverse adaptive tree: you chunk at sentences level, then compare, if 'close enough', you merge them into a bigger chunk
- seabass 2y ago
  Suppose you don't know the shape of the content ahead of time. This would be the case for most apps that allow users to upload their own sources. It sounds like (at the expense of more computation and time) the reverse adaptive tree approach you described would be ideal for those scenarios.
  - homarp 2y ago
    that's why the article talks about using machine learning to decide of best strategy: to deal with unknown content 'shape'
leobg 2y ago
Why not overlapping sizes? (1), (1,2), (1,2,3) Sometimes the match is in a single sentence. Sometimes in the full paragraph. Sometimes across two paragraphs. If, in your top n results, some of these items overlap, you use the greater unit. And you slide these windows. Also, I wouldn’t necessarily use a “sentence” as the lower bound, since that can be something like “Yes.”
CuriouslyC 2y ago
When using vector search, you want chunks in the ~1k tokens range. If you're using full text search then chunks should probably be smaller, say a few paragraphs at most. If you use trigrams, you want chunks to be short, maybe even sentence level.
laborcontract 2y ago
I’ll share my strategy. I usually keep chunks at the max of 0.5% of a document or 270 tokens. Multiple that by three and those are the size of the sliding windows that are then worried.

bob1029 2y ago

Has anyone tried skipping the embedding path in favor of combining a proper FTS engine with an LLM yet? A tool like Lucene seems far more competent at the task of "find most relevant text fragment" compared to what is realized in a typical vector search application today. I'd also argue that you get more inspectability and control this way. You could even manage the preferred size of the fragments on a per-document basis using an entirely separate heuristic at indexing time. The semantic capabilities of vector search seem nice, but could you not achieve a similar outcome by using the LLM to project synonymous OR clauses into the FTS query based upon static background material or prior search iteration(s)?

lmeyerov 2y ago
Generally you do both and rank the combined results This is also why generally you don't get a vectordb and instead just add a vector index to the DB you are already using Re:semantic vs synonym, that's really domain dependent. As the number of hits go up, and queries get more interesting, the more vectors get interesting. At the same time, vectors are heavy, so there's also the question of pushing the semantic aspect to the ranker vs the index, but I don't see that discussed much (search & storage vs compute & latency)
- bob1029 2y ago
  > There's also the question of pushing the semantic aspect to the ranker vs the index Could it make sense to perform dynamic vector lookup over the FTS result set best fragments? This could save a lot of money if you have a massive corpus to index because you'd only be paying to embed things that are being searched for at runtime. Focusing on just the best fragments could also improve the SnR going into the final vector search phase, especially if the fragment length is managed appropriately for each kind of document. If we are dealing with a method from a codebase, then we might prefer to have an unlimited fragment length. For a 20 megabyte PDF, it could be closer to the size of a tweet.
woodson 2y ago
Query expansion has been done since forever, long before even word2vec and other semantic embedding models, e.g. using WordNet (not a DNN, despite its name), see https://lucene.apache.org/core/3_3_0/api/contrib-wordnet/org/apache/lucene/wordnet/package-summary.html https://lucene.apache.org/core/3_3_0/api/contrib-wordnet/org....

diptanu 2y ago

We use grid search to figure out what's the best chunking strategy to use. Create a bunch of different strategies such as recursive chunking, semantic, etc, and parameterize them and see which one works best. The "best" chunking strategy depends on the nature of the documents and the questions being asked.

stoicjumbotron 2y ago

What would be the chunking strategy for q&a pairs? Right now I'm embedding the complete question and answer but the query results are not good as the response contains data not related to the question at all.

harpastum 2y ago
It depends on the specifics of your format, but we’ve had success embedding the questions and answers separately. If either match, you return the complete question and answer text. Make sure to deduplicate before returning, in case both match.
- stoicjumbotron 2y ago
  I'll try embedding it separately as well and try to figure out from there. Thanks for the suggestion
laborcontract 2y ago
Join q and a when vectorizing them. Questions alone are too short to carry a lot of semantic richness. When you get a query, you then run two semantic search queries: one using the original question and one using a HYDE version of the question. Take those results and run it through cohere’s rerank.
- stoicjumbotron 2y ago
  Yes, right now I'm vectorizing them as a pair. I'm then running a query in Pinecone by embedding the exact question but still the result does not have the actual q&a pair. I'm not familiar with HYDE version. I'll check it out. Thanks for the suggestion

constantinum 2y ago

with ever increasing context size — Claude 3 Sonnet, 204,800 token context size, 500 page document — sometimes not opting for chunking but at the same time optimizing for costs and latency might be a better solution, strategies like: Summary extraction and single pass extraction works well[1] [1] - https://docs.unstract.com/editions/cloud_edition#summary-extraction-and-single-pass-extraction-reduce-cost-by-up-to-6x-per-document-extraction https://docs.unstract.com/editions/cloud_edition#summary-ext...

J_Shelby_J 2y ago
In theory. In practice, context size degrades performance.

J_Shelby_J 2y ago

The problem with most chunking schemes I’ve seen is they’re naive. They don’t care about content; only token count. That’s fine, but what would be best is to chunk by topic. I’m currently trying to implement chunking by topic using an LLM. It’s much slower, but I hope it will be a huge win in retrieval accuracy. First step is to extract topics from the document by asking the LLM to identify all topics, and then split by sentences and feed each sentence to the LLM to identify the topic. I’m hoping the result will be the original text, split by topics. From there, they can be further chunked if needed. Of course, it could be done by just one shot asking the LLM to summarize each topic in the document, but the more the LLM is relied on to write, the more distortion is introduced. Retaining the original text is the goal and the LLM should just be used for decision making. Here is the crate I’m working out of. The chunking hasn’t been pushed, but you can see the decision making workflows. https://github.com/ShelbyJenkins/llm_client https://github.com/ShelbyJenkins/llm_client

Swizec 2y ago
> They don’t care about content; only token count. That’s fine, but what would be best is to chunk by topic. I’ve had a lot of success with chunking documents by subtitle. Works especially well for web published documents because each section tends to be fairly short.
serjester 2y ago
You’re basically reinventing embeddings. Using an LLM for this is overkill since capturing the meaning of a sentence can be done with a model 1000x smaller. I have a chunking library that does something similar to this, and there’s actually quite a few different libraries that have implemented some variant of “semantic chunking”. [1] https://github.com/Filimoa/open-parse https://github.com/Filimoa/open-parse
- nostrebored 2y ago
  Excellent, now show me a single one working in production with various docs. I’ve talked through this problem with dozens of startups working on RAG, and everyone has the same problem. LLMs are arguably not reinventing embeddings if you’re using them to infer the structure of a broad document. Understanding the “external structure” of the document, the internal structure of each segment, and the semantic structure of each chunk is important.

serjester 2y ago

As embedding models become more performant and context windows increase, “ideal chunking” becomes less relevant. Cost isn’t as important to us, so we use small chunks and then just pull in the page before and after. If you do this on 20+ matches (since you’re decomposing the query multiple times), you’re very likely finding the content. Queries can get more expensive but you’re getting a corpus of “great answers” to test against as you refine your approach. Model costs are also plummeting which makes brute forcing it more and more viable.

emporas 2y ago
Bigger context windows don't work as well as advertised. The "hay in a haystack" problem is not solved yet. Also bigger context windows mean a lot more time waiting for an answer. Given the quadratic nature of context windows, we are stuck using transformers in smaller chunks. Other architectures like Mamba may solve that, but even then, increases in context window accuracy are not 1000x.

philbo 2y ago

Just as important as chunking strategy, imo, is recombination. After you've picked the highest-scoring chunks from the search matter, how do you glue them back together? Simple concatenation is the obvious way, but I've had better results from using the chunks as anchors for expansion. Use something like Wilson score interval to decide how many high-scoring chunks to pick, then grab the surrounding text for each chunk up to some semantic boundary like period or newline (or arbitrary limit if no boundary is detected).

antman 2y ago

tldr: Use adaptive chunking methods, where ML decides if consecutive texts are relevant. Does not mention any actual adaptive chunking method name

chiccomagnus 2y ago

Well-written article, missing key considerations: - Titles matter, a lot: if you add the title of the section at the start of each chunk you will get 10x better embeddings and so more accurate results. - The size doesn't matter: It depends on the combination of the layout and semantics of the content. - Avoid garbage in / out: increased context windows don't mean you can put trash inside them. The more good you are at putting relevant information the more precise answers you get. Especially for enterprise-grade solutions, this is so important. There are good emerging API solutions that implement semantic + layout-based chunking, which in my opinion is the best chunking strategy for PDF / Office files (the widest use case scenario for enterprises).