3 ms·

When they recommend using smaller chunks, how small in general are we talking? One sentence? One paragraph? 100 tokens or 1000? While I understand that the cont

When they recommend using smaller chunks, how small in general are we talking? One sentence? One paragraph? 100 tokens or 1000? While I understand that the context of the data is important, it’s hard for me to ground vague statements like that without a concrete realistic example. I’m curious what chunk sizes people have found the most success with for various tasks in the wild.

homarp 2y ago
think this way: you are going to return the chunk to your user as 'proof'. the size therefore depends of your content style. E.g. for a HN discussion, I would go 'paragraph' of each comment. in a contract, each clause. in a non-fiction book, maybe each section... you can also decide to do some kind of reverse adaptive tree: you chunk at sentences level, then compare, if 'close enough', you merge them into a bigger chunk
- seabass 2y ago
  Suppose you don't know the shape of the content ahead of time. This would be the case for most apps that allow users to upload their own sources. It sounds like (at the expense of more computation and time) the reverse adaptive tree approach you described would be ideal for those scenarios.
  - homarp 2y ago
    that's why the article talks about using machine learning to decide of best strategy: to deal with unknown content 'shape'
leobg 2y ago
Why not overlapping sizes? (1), (1,2), (1,2,3) Sometimes the match is in a single sentence. Sometimes in the full paragraph. Sometimes across two paragraphs. If, in your top n results, some of these items overlap, you use the greater unit. And you slide these windows. Also, I wouldn’t necessarily use a “sentence” as the lower bound, since that can be something like “Yes.”
CuriouslyC 2y ago
When using vector search, you want chunks in the ~1k tokens range. If you're using full text search then chunks should probably be smaller, say a few paragraphs at most. If you use trigrams, you want chunks to be short, maybe even sentence level.
laborcontract 2y ago
I’ll share my strategy. I usually keep chunks at the max of 0.5% of a document or 270 tokens. Multiple that by three and those are the size of the sliding windows that are then worried.

homarp 2y ago

think this way: you are going to return the chunk to your user as 'proof'. the size therefore depends of your content style. E.g. for a HN discussion, I would go 'paragraph' of each comment. in a contract, each clause. in a non-fiction book, maybe each section... you can also decide to do some kind of reverse adaptive tree: you chunk at sentences level, then compare, if 'close enough', you merge them into a bigger chunk

seabass 2y ago
Suppose you don't know the shape of the content ahead of time. This would be the case for most apps that allow users to upload their own sources. It sounds like (at the expense of more computation and time) the reverse adaptive tree approach you described would be ideal for those scenarios.
- homarp 2y ago
  that's why the article talks about using machine learning to decide of best strategy: to deal with unknown content 'shape'

leobg 2y ago

Why not overlapping sizes? (1), (1,2), (1,2,3) Sometimes the match is in a single sentence. Sometimes in the full paragraph. Sometimes across two paragraphs. If, in your top n results, some of these items overlap, you use the greater unit. And you slide these windows. Also, I wouldn’t necessarily use a “sentence” as the lower bound, since that can be something like “Yes.”

CuriouslyC 2y ago

When using vector search, you want chunks in the ~1k tokens range. If you're using full text search then chunks should probably be smaller, say a few paragraphs at most. If you use trigrams, you want chunks to be short, maybe even sentence level.

laborcontract 2y ago

I’ll share my strategy. I usually keep chunks at the max of 0.5% of a document or 270 tokens. Multiple that by three and those are the size of the sliding windows that are then worried.