4 ms·
Yes. I split the text into sentence and append sentences to a chunk until the max context window is reached. The context window size is dynamic for each article
by gfourfour 2y ago
Yes. I split the text into sentence and append sentences to a chunk until the max context window is reached. The context window size is dynamic for each article so that each chunk is roughly the same size. Then I just do a mean pool of the chunks for each article.
- thomasfromcdnjs 2y agoThanks for the answer. Wikipedia has a lot of tables so I was wondering if content-aware sentence chunking would be good enough for Wikipedia. https://www.pinecone.io/learn/chunking-strategies/ https://www.pinecone.io/learn/chunking-strategies/
- gfourfour 2y agomwparserfromhell can parse the text content without including tables