3 ms·
Those diagrams are nice! What did you use to make them? The sliding window mechanic is interesting but I'm not seeing how the first, second and third retrievers
by gsuuon 3y ago
Those diagrams are nice! What did you use to make them? The sliding window mechanic is interesting but I'm not seeing how the first, second and third retrievers relate. Only the final medium chunks are used, but how are those arrived at?
- joeyxiong 3y agoHi, I created the diagrams using Figma. The retrieval process consists of three stages. The first stage retrieves small chunks from multiple documents to create a document filter using their metadat. This filter is then applied in the second stage to extract relevant large chunks, essentially sections of documents, which further refines our search parameters. Finally, using both the document and large chunk filters, the third stage retrieves the most pertinent medium-sized chunks of information to be passed to the Language Model, ensuring a focused and relevant response to your query.
- gsuuon 3y agoSo to rephrase: stage 1 - it finds the top k most relevant small chunks, stage 2 - it searches the source documents of those small chunks for most relevant large chunks, stage 3 - it searches medium chunks in the source documents of relevant small chunks that are contained in the large chunks found in stage 2?
- joeyxiong 3y agoYeah, the sliding window chunking mechanism appears in stage 2.