3 ms·
> The text needs to be fully analyzed with LLMs before embedding. If you happen to know what kinds of questions you will be asking about your RAG index, you sh
by visarga 3y ago
> The text needs to be fully analyzed with LLMs before embedding.
If you happen to know what kinds of questions you will be asking about your RAG index, you should pre-process the texts to add QA pairs. Otherwise you can prompt the LLM to do chain-of-thought inferences based on the source text and add them to the material.
- sroussey 3y agoI guess you log queries to see what is popular and then reprocess texts based on those?
- sroussey 3y agoAside from a feedback loop from usage, is there a way to guess? I guess you put put a whole doc into the I’ll and ask what questions it answers? And then use those question plus a piece of the text and do an embedding?