3 ms·
think this way: you are going to return the chunk to your user as 'proof'. the size therefore depends of your content style. E.g. for a HN discussion, I would
by homarp 2y ago
think this way:
you are going to return the chunk to your user as 'proof'.
the size therefore depends of your content style.
E.g. for a HN discussion, I would go 'paragraph' of each comment.
in a contract, each clause.
in a non-fiction book, maybe each section...
you can also decide to do some kind of reverse adaptive tree: you chunk at sentences level, then compare, if 'close enough', you merge them into a bigger chunk
- seabass 2y agoSuppose you don't know the shape of the content ahead of time. This would be the case for most apps that allow users to upload their own sources. It sounds like (at the expense of more computation and time) the reverse adaptive tree approach you described would be ideal for those scenarios.
- homarp 2y agothat's why the article talks about using machine learning to decide of best strategy: to deal with unknown content 'shape'