3 ms·
How does your chunker actually work? The gist only shows an API call. Or is that secret?
by pamelafox 2y ago
How does your chunker actually work? The gist only shows an API call. Or is that secret?
- nutanc 2y agoWill be sharing the code next week. The basic idea is to find the embeddings of sentences and then finding the distance in the latent space to see if there is too big a jump in context.
- magicalhippo 2y agoA single sentence can have a lot of different meanings. I wonder if it would be more fruitful to use rolling pairs and triplets? Ie was thinking about how you use two moving averages with different window sizes to detect trend shifts.
- nutanc 2y agoYes. We are using rolling sentences. Not just pairs. But the whole story. I have written a little about this in my previous post, shape of stories, https://gpt3experiments.substack.com/p/the-shape-of-stories-or-how-ai-sees https://gpt3experiments.substack.com/p/the-shape-of-stories-... So we can see how as we are appending sentences and getting the embedding we get the movement of the story in the latent space.