4 ms·
Just as important as chunking strategy, imo, is recombination. After you've picked the highest-scoring chunks from the search matter, how do you glue them back
by philbo 2y ago
Just as important as chunking strategy, imo, is recombination. After you've picked the highest-scoring chunks from the search matter, how do you glue them back together?
Simple concatenation is the obvious way, but I've had better results from using the chunks as anchors for expansion. Use something like Wilson score interval to decide how many high-scoring chunks to pick, then grab the surrounding text for each chunk up to some semantic boundary like period or newline (or arbitrary limit if no boundary is detected).