2 ms·
I made a rudimentary semantic chunking in just a few lines of code. I just removed one sentence at a time from the left until there was a jump in the embedding
by andai 2y ago
I made a rudimentary semantic chunking in just a few lines of code.
I just removed one sentence at a time from the left until there was a jump in the embedding distance. Then repeated for the right side.