4 ms·
This is where chunking comes into play. If you embed the entire document, then yes it’s likely to only be close in vector space to other documents containing li
by didgeoridoo 3y ago
This is where chunking comes into play. If you embed the entire document, then yes it’s likely to only be close in vector space to other documents containing lions, tigers, AND bears. But if you set up your chunking such that you’re dividing the document by sentence, paragraph, etc. you will get a finer-grained set of separate embeddings that can point to the document (or even different areas within it) whether you’re looking for lions, tigers, OR bears.
- taneq 3y agoSounds like a ‘mipmap’ sort of approach could be useful. Add the whole document, then the chapters, then paragraphs. How does the system know how much context is required, though? If I ask “Steve wears turtlenecks, what company does Steve work at?” then a paragraph about a Steve who likes turtlenecks and a paragraph about a Steve who works at MS would both be highly relevant but when combined they would result in the wrong answer.
- didgeoridoo 3y agoThe boring answer is to bash domain expertise and evals together until your data pipeline seems to be spitting out embeddings that give you acceptable results for whatever you’re trying to do. I do wonder if there is some additional cleverness that’s possible with fast, long-context LLMs, maybe a summarizing/preprocessing step that can slice up the document into a sensible hierarchy of topics that can be used as the embeddings source. Never seen this in practice though.