3 ms·
That makes a lot of sense that a fuzzy 512 character window that preserves sentences properly would be a good strategy. Do you also use nltk to extract keyterm
by alchemist1e9 3y ago
That makes a lot of sense that a fuzzy 512 character window that preserves sentences properly would be a good strategy.
Do you also use nltk to extract keyterm then? I was looking at Qdrant labels and thinking about how vectors could be labeled with location metadata but also perhaps with named entity information from the chunk.
I imagine it how I might flip though a book using the index and also table of contents and then perhaps skim the intro paragraphs etc. Eventually the RAG systems need to replicate a more sophisticated human like retrieval of relevant information.
- kordlessagain 3y agoI use a dict completion prompt to extract keyterms from various LLMs. Technically, this is transfer learning and I use it in a way that allows set operations to filter the vector space to speed things up. In a version I built of this, I referred to it as a "back of book index". One issue would be that the prompt for keyterm extraction might miss crucial keyterms, but a subsequent access of those might update them...