3 ms·

I don't have links unfortunately. What I meant was that at the time of indexing, you can add more information to any chunk. This[1] is a simple example by Anth

I don't have links unfortunately. What I meant was that at the time of indexing, you can add more information to any chunk. This[1] is a simple example by Anthropic where they add more relevant context. In our case, say you have two models, D1 and D2. At the time of creating a vector store, you can add which model is more suitable to a chunk, so that when you retrieve it, you use the same model for inference. This is custom built, very dependent on datasets, but would get you to the functionality described. I suggest this approach when there are linkages between various docs (eg: financial statements/earning calls etc.). [1]: https://www.anthropic.com/news/contextual-retrieval https://www.anthropic.com/news/contextual-retrieval

Pamar 2y ago
Thanks... I also have another lingering doubt about the ability of RAG to make sense of "history", i.e. how to make sure that a more recent document on a given topic has more "weight" than older documents on the same issue. I try to explain this a bit better here: https://pa-mar.net/Study/AiKiDo/VirtualBudoPass.html https://pa-mar.net/Study/AiKiDo/VirtualBudoPass.html
- ankit219 2y ago
  This is done at a reranking step. It's again custom. You have two variables - 1/ relevance (which most algos focus on) 2/ Date. Create a new score based on some combination of weights for relevance and date. Eg; Could be 50% of date. If the document has 70% relevance, but was published yesterday, it's overall score would be 85%. (A conceptual idea). This is similar to how you do weighted sorting anywhere.
  - Pamar 2y ago
    Thank you! Btw, I notice only now that the link that was supposed to explain my question better is completely wrong. https://news.ycombinator.com/item?id=42589014 https://news.ycombinator.com/item?id=42589014 (But you still provided much needed clarification).
    - ankit219 2y ago
      No worries. If you need any specific help with the use case, please feel free to reach out on my email. ankit at clioapp.ai

Pamar 2y ago

Thanks... I also have another lingering doubt about the ability of RAG to make sense of "history", i.e. how to make sure that a more recent document on a given topic has more "weight" than older documents on the same issue. I try to explain this a bit better here: https://pa-mar.net/Study/AiKiDo/VirtualBudoPass.html https://pa-mar.net/Study/AiKiDo/VirtualBudoPass.html

ankit219 2y ago
This is done at a reranking step. It's again custom. You have two variables - 1/ relevance (which most algos focus on) 2/ Date. Create a new score based on some combination of weights for relevance and date. Eg; Could be 50% of date. If the document has 70% relevance, but was published yesterday, it's overall score would be 85%. (A conceptual idea). This is similar to how you do weighted sorting anywhere.
- Pamar 2y ago
  Thank you! Btw, I notice only now that the link that was supposed to explain my question better is completely wrong. https://news.ycombinator.com/item?id=42589014 https://news.ycombinator.com/item?id=42589014 (But you still provided much needed clarification).
  - ankit219 2y ago
    No worries. If you need any specific help with the use case, please feel free to reach out on my email. ankit at clioapp.ai