2 ms·
That's the same problem I haven't figured out yet: the best strategies for chunking. I'm hoping good, well proven patterns emerge soon so I can integrate them i
by simonw 3y ago
That's the same problem I haven't figured out yet: the best strategies for chunking. I'm hoping good, well proven patterns emerge soon so I can integrate them into my various tools.
- alchemist1e9 3y agoI’ve been looking at this https://freeling-user-manual.readthedocs.io/en/v4.2/modules/splitter/ https://freeling-user-manual.readthedocs.io/en/v4.2/modules/... at the freeling library in general, also spaCy and NLTK. The chunking algorithms being used in the likes of LangChain are remarkably bad surprisingly. There is also https://github.com/Unstructured-IO/unstructured https://github.com/Unstructured-IO/unstructured But I don’t like it, can’t explain why yet. My intuition is that 1st step is clean sentences and paragraphs and titles/labels/headers. Then probably an LLM can handle outlining and table of contents generation using a stripped down list of objects in the text. BRIO/BERT summarization could also have a role of some type. Those are my ideas so far.