2 ms·

I've found that 512 characters, more or less, is a good length for embeddings. More than that, and a bunch of models will just ignore the text. Less than half t

I've found that 512 characters, more or less, is a good length for embeddings. More than that, and a bunch of models will just ignore the text. Less than half that, there isn't always enough context in the text to be able to group the chunks by keyterm or other labels. I do this type of additive context using the first few results from the embedding search. If it's tweets, it makes sense to use shorter chunks. I was dissatisfied with the results of chunks "broken off" from sentences, so I use nltk to find sentence boundaries and then approximate the lengths of chunks. I also built a custom splitter function for Jinja2 that will chunk by length, page offset, and also does overlaps, where bits of the beginning and ends of a chunk will run into the previous and next chunk. I have a pipeline for this particular type of use, running on an Open Source thing I built to handle indexing texts: https://mitta.ai https://mitta.ai. The RAG pipeline is under cookbooks.

alchemist1e9 3y ago
That makes a lot of sense that a fuzzy 512 character window that preserves sentences properly would be a good strategy. Do you also use nltk to extract keyterm then? I was looking at Qdrant labels and thinking about how vectors could be labeled with location metadata but also perhaps with named entity information from the chunk. I imagine it how I might flip though a book using the index and also table of contents and then perhaps skim the intro paragraphs etc. Eventually the RAG systems need to replicate a more sophisticated human like retrieval of relevant information.
- kordlessagain 3y ago
  I use a dict completion prompt to extract keyterms from various LLMs. Technically, this is transfer learning and I use it in a way that allows set operations to filter the vector space to speed things up. In a version I built of this, I referred to it as a "back of book index". One issue would be that the prompt for keyterm extraction might miss crucial keyterms, but a subsequent access of those might update them...

alchemist1e9 3y ago

That makes a lot of sense that a fuzzy 512 character window that preserves sentences properly would be a good strategy. Do you also use nltk to extract keyterm then? I was looking at Qdrant labels and thinking about how vectors could be labeled with location metadata but also perhaps with named entity information from the chunk. I imagine it how I might flip though a book using the index and also table of contents and then perhaps skim the intro paragraphs etc. Eventually the RAG systems need to replicate a more sophisticated human like retrieval of relevant information.

kordlessagain 3y ago
I use a dict completion prompt to extract keyterms from various LLMs. Technically, this is transfer learning and I use it in a way that allows set operations to filter the vector space to speed things up. In a version I built of this, I referred to it as a "back of book index". One issue would be that the prompt for keyterm extraction might miss crucial keyterms, but a subsequent access of those might update them...