6 ms·
Experimenting with creating semantic chunks of large podcasts. Got the following chunks, https://gist.github.com/nutanc/a9e6321649be5ea9806b4450b0bd6552 https:/
by nutanc 2y ago
Experimenting with creating semantic chunks of large podcasts. Got the following chunks, https://gist.github.com/nutanc/a9e6321649be5ea9806b4450b0bd6552 https://gist.github.com/nutanc/a9e6321649be5ea9806b4450b0bd6...
Dwarkesh has 18 splits. https://www.dwarkeshpatel.com/i/151435243/timestamps https://www.dwarkeshpatel.com/i/151435243/timestamps
I got 171. So roughly 9 context discussions in one time stamp.
- maujim 2y agoWhat did you use to create the chunks?
- nutanc 2y agoIt's a new approach I am experimenting with. https://gpt3experiments.substack.com/p/a-new-chunking-approach-to-rag https://gpt3experiments.substack.com/p/a-new-chunking-approa...
- maujim 2y agoInteresting approach. Have you tried using existing clustering algorithms to determine the chunks instead of using distance or slope as a measure?