I’ve been looking at this https://freeling-user-manual.readthedocs.io/en/v4.2/modules/splitter/ https://freeling-user-manual.readthedocs.io/en/v4.2/modules/...

I’ve been looking at this https://freeling-user-manual.readthedocs.io/en/v4.2/modules/splitter/ https://freeling-user-manual.readthedocs.io/en/v4.2/modules/... at the freeling library in general, also spaCy and NLTK. The chunking algorithms being used in the likes of LangChain are remarkably bad surprisingly. There is also https://github.com/Unstructured-IO/unstructured https://github.com/Unstructured-IO/unstructured But I don’t like it, can’t explain why yet. My intuition is that 1st step is clean sentences and paragraphs and titles/labels/headers. Then probably an LLM can handle outlining and table of contents generation using a stripped down list of objects in the text. BRIO/BERT summarization could also have a role of some type. Those are my ideas so far.