3 ms·

The problem with most chunking schemes I’ve seen is they’re naive. They don’t care about content; only token count. That’s fine, but what would be best is to ch

The problem with most chunking schemes I’ve seen is they’re naive. They don’t care about content; only token count. That’s fine, but what would be best is to chunk by topic. I’m currently trying to implement chunking by topic using an LLM. It’s much slower, but I hope it will be a huge win in retrieval accuracy. First step is to extract topics from the document by asking the LLM to identify all topics, and then split by sentences and feed each sentence to the LLM to identify the topic. I’m hoping the result will be the original text, split by topics. From there, they can be further chunked if needed. Of course, it could be done by just one shot asking the LLM to summarize each topic in the document, but the more the LLM is relied on to write, the more distortion is introduced. Retaining the original text is the goal and the LLM should just be used for decision making. Here is the crate I’m working out of. The chunking hasn’t been pushed, but you can see the decision making workflows. https://github.com/ShelbyJenkins/llm_client https://github.com/ShelbyJenkins/llm_client

Swizec 2y ago
> They don’t care about content; only token count. That’s fine, but what would be best is to chunk by topic. I’ve had a lot of success with chunking documents by subtitle. Works especially well for web published documents because each section tends to be fairly short.
serjester 2y ago
You’re basically reinventing embeddings. Using an LLM for this is overkill since capturing the meaning of a sentence can be done with a model 1000x smaller. I have a chunking library that does something similar to this, and there’s actually quite a few different libraries that have implemented some variant of “semantic chunking”. [1] https://github.com/Filimoa/open-parse https://github.com/Filimoa/open-parse
- nostrebored 2y ago
  Excellent, now show me a single one working in production with various docs. I’ve talked through this problem with dozens of startups working on RAG, and everyone has the same problem. LLMs are arguably not reinventing embeddings if you’re using them to infer the structure of a broad document. Understanding the “external structure” of the document, the internal structure of each segment, and the semantic structure of each chunk is important.
  - serjester 2y ago
    See my other comment on this thread - larger and larger context windows make perfect chunking irrelevant. We’ve ingested millions of docs and have a product in production, using LLM’s for chunking would have 10Xed our ingestion costs for marginal performance improvement.
  - J_Shelby_J 2y ago
    Good to be validated. I guess I'll publish a blog post on the topic when I finalize what I've been working on.
emporas 2y ago
Why not use langchain-rust and make your own client? If you don't know about langchain, i think you are missing out. I took a look at other langchain implementions in js and python, in each one people have done some serious work. Langchain-rust also uses tree-sitter to chunk code, it works very well in some quick tests i tried. >The problem with most chunking schemes I’ve seen is they’re naive. They don’t care about content; only token count. I think controlling different inputs depending on context is used in agents. For the moment i haven't seen anything really impressive coming out of agents. Maybe Perplexity style web search, but nothing more.
- sophiabits 2y ago
  Not sure what the situation is like now, but we stopped using LangChain last year because the rate of change in the library was huge. Whenever we needed to upgrade for a new feature or bug fix we’d be 20~ versions behind and need to work through breaking changes. Eventually we decided that it was easier to just write everything ourselves. This is from the first half of 2023 or so; maybe things are more stable now, but looks like the Python implementation is still pre-v1.
dcsan 2y ago
isn’t this just semantic chunking? There’s been white papers and already implementations in Langchain and llama index you could look through.
powerapple 2y ago
I used to think the same, then I realize that topic is just a concept we draw from the content. It just reminded me the time we do feature engineering for many things, now we just feed everything into the algorithm and let it figure out how to handle things. Maybe the problem for us is not how to organize data in a way we think it is best, but for a format which is useful for the algorithm. A fixed window size fits perfectly with the embedding model would just work.

Swizec 2y ago

> They don’t care about content; only token count. That’s fine, but what would be best is to chunk by topic. I’ve had a lot of success with chunking documents by subtitle. Works especially well for web published documents because each section tends to be fairly short.

serjester 2y ago

You’re basically reinventing embeddings. Using an LLM for this is overkill since capturing the meaning of a sentence can be done with a model 1000x smaller. I have a chunking library that does something similar to this, and there’s actually quite a few different libraries that have implemented some variant of “semantic chunking”. [1] https://github.com/Filimoa/open-parse https://github.com/Filimoa/open-parse

nostrebored 2y ago
Excellent, now show me a single one working in production with various docs. I’ve talked through this problem with dozens of startups working on RAG, and everyone has the same problem. LLMs are arguably not reinventing embeddings if you’re using them to infer the structure of a broad document. Understanding the “external structure” of the document, the internal structure of each segment, and the semantic structure of each chunk is important.
- serjester 2y ago
  See my other comment on this thread - larger and larger context windows make perfect chunking irrelevant. We’ve ingested millions of docs and have a product in production, using LLM’s for chunking would have 10Xed our ingestion costs for marginal performance improvement.
- J_Shelby_J 2y ago
  Good to be validated. I guess I'll publish a blog post on the topic when I finalize what I've been working on.

emporas 2y ago

Why not use langchain-rust and make your own client? If you don't know about langchain, i think you are missing out. I took a look at other langchain implementions in js and python, in each one people have done some serious work. Langchain-rust also uses tree-sitter to chunk code, it works very well in some quick tests i tried. >The problem with most chunking schemes I’ve seen is they’re naive. They don’t care about content; only token count. I think controlling different inputs depending on context is used in agents. For the moment i haven't seen anything really impressive coming out of agents. Maybe Perplexity style web search, but nothing more.

sophiabits 2y ago
Not sure what the situation is like now, but we stopped using LangChain last year because the rate of change in the library was huge. Whenever we needed to upgrade for a new feature or bug fix we’d be 20~ versions behind and need to work through breaking changes. Eventually we decided that it was easier to just write everything ourselves. This is from the first half of 2023 or so; maybe things are more stable now, but looks like the Python implementation is still pre-v1.

dcsan 2y ago

isn’t this just semantic chunking? There’s been white papers and already implementations in Langchain and llama index you could look through.

powerapple 2y ago

I used to think the same, then I realize that topic is just a concept we draw from the content. It just reminded me the time we do feature engineering for many things, now we just feed everything into the algorithm and let it figure out how to handle things. Maybe the problem for us is not how to organize data in a way we think it is best, but for a format which is useful for the algorithm. A fixed window size fits perfectly with the embedding model would just work.