4 ms·
Hi, I created the diagrams using Figma. The retrieval process consists of three stages. The first stage retrieves small chunks from multiple documents to creat
by joeyxiong 3y ago
Hi, I created the diagrams using Figma.
The retrieval process consists of three stages. The first stage retrieves small chunks from multiple documents to create a document filter using their metadat. This filter is then applied in the second stage to extract relevant large chunks, essentially sections of documents, which further refines our search parameters. Finally, using both the document and large chunk filters, the third stage retrieves the most pertinent medium-sized chunks of information to be passed to the Language Model, ensuring a focused and relevant response to your query.
- gsuuon 3y agoSo to rephrase: stage 1 - it finds the top k most relevant small chunks, stage 2 - it searches the source documents of those small chunks for most relevant large chunks, stage 3 - it searches medium chunks in the source documents of relevant small chunks that are contained in the large chunks found in stage 2?
- joeyxiong 3y agoYeah, the sliding window chunking mechanism appears in stage 2.