3 ms·

Short answer: Yes, there are ways it can be done. Multiple. Needs to be custom built though, given no one has explored it deeply yet. One simple way is what Om

Short answer: Yes, there are ways it can be done. Multiple. Needs to be custom built though, given no one has explored it deeply yet. One simple way is what Omar Khattab (ColBert) mentioned about scoring function instead of a simple vector. Another is to use a classifier at the start directing queries to the right model. You will have to train the classifier though. (I mean a language model kind of does this implicitly, you are just taking more control by making it explicit.) Another is how you index your docs. Today, most RAG approaches do not encode enough information. If you have defined domains/models already, you can encode the same in metadata for your docs at the time of indexing, and you pick the model based on the metadata. These approaches would work pretty well, given a model as small as 100M size can regurgitate what is in your docs. And is faster compared to your larger models. Benefit wise, I don't see a lot of benefit except preserving privacy and gaining more control.

daxfohl 2y ago
I was originally thinking about it as like a bazel plugin for large codebases. Each module would have its own LLM context, and it might make it easier to put whole modules into the context, plus summaries of the dependencies. That could work better than a single huge context attempting to summarize the whole monorepo. The general idea is probably be better for the code use case too, since having the module's whole codebase in context likely allows for more precise edits. Whereas RAG is just search, not edit. That said, probably code assistants must somewhat do this already, though it must be more ad-hoc. Obviously they wouldn't be able to do any completions if they don't have detailed context of the adjacent code.
Pamar 2y ago
Another is how you index your docs. Today, most RAG approaches do not encode enough information.... Could you please provide some more info (or maybe links) about this, please?
- antman 2y ago
  I think he might be saying, have metadata in your vector retrieval that describe the domain of the retrieved chunk and use that as a decision on which model to use downstream. Sounds like very interesting improvement of RAG
- ankit219 2y ago
  I don't have links unfortunately. What I meant was that at the time of indexing, you can add more information to any chunk. This[1] is a simple example by Anthropic where they add more relevant context. In our case, say you have two models, D1 and D2. At the time of creating a vector store, you can add which model is more suitable to a chunk, so that when you retrieve it, you use the same model for inference. This is custom built, very dependent on datasets, but would get you to the functionality described. I suggest this approach when there are linkages between various docs (eg: financial statements/earning calls etc.). [1]: https://www.anthropic.com/news/contextual-retrieval https://www.anthropic.com/news/contextual-retrieval
  - Pamar 2y ago
    Thanks... I also have another lingering doubt about the ability of RAG to make sense of "history", i.e. how to make sure that a more recent document on a given topic has more "weight" than older documents on the same issue. I try to explain this a bit better here: https://pa-mar.net/Study/AiKiDo/VirtualBudoPass.html https://pa-mar.net/Study/AiKiDo/VirtualBudoPass.html
    - ankit219 2y ago
      This is done at a reranking step. It's again custom. You have two variables - 1/ relevance (which most algos focus on) 2/ Date. Create a new score based on some combination of weights for relevance and date. Eg; Could be 50% of date. If the document has 70% relevance, but was published yesterday, it's overall score would be 85%. (A conceptual idea). This is similar to how you do weighted sorting anywhere.
      - Pamar 2y ago
        Thank you! Btw, I notice only now that the link that was supposed to explain my question better is completely wrong. https://news.ycombinator.com/item?id=42589014 https://news.ycombinator.com/item?id=42589014 (But you still provided much needed clarification).
        2y ago

daxfohl 2y ago

I was originally thinking about it as like a bazel plugin for large codebases. Each module would have its own LLM context, and it might make it easier to put whole modules into the context, plus summaries of the dependencies. That could work better than a single huge context attempting to summarize the whole monorepo. The general idea is probably be better for the code use case too, since having the module's whole codebase in context likely allows for more precise edits. Whereas RAG is just search, not edit. That said, probably code assistants must somewhat do this already, though it must be more ad-hoc. Obviously they wouldn't be able to do any completions if they don't have detailed context of the adjacent code.

Pamar 2y ago

Another is how you index your docs. Today, most RAG approaches do not encode enough information.... Could you please provide some more info (or maybe links) about this, please?

antman 2y ago
I think he might be saying, have metadata in your vector retrieval that describe the domain of the retrieved chunk and use that as a decision on which model to use downstream. Sounds like very interesting improvement of RAG
ankit219 2y ago
I don't have links unfortunately. What I meant was that at the time of indexing, you can add more information to any chunk. This[1] is a simple example by Anthropic where they add more relevant context. In our case, say you have two models, D1 and D2. At the time of creating a vector store, you can add which model is more suitable to a chunk, so that when you retrieve it, you use the same model for inference. This is custom built, very dependent on datasets, but would get you to the functionality described. I suggest this approach when there are linkages between various docs (eg: financial statements/earning calls etc.). [1]: https://www.anthropic.com/news/contextual-retrieval https://www.anthropic.com/news/contextual-retrieval
- Pamar 2y ago
  Thanks... I also have another lingering doubt about the ability of RAG to make sense of "history", i.e. how to make sure that a more recent document on a given topic has more "weight" than older documents on the same issue. I try to explain this a bit better here: https://pa-mar.net/Study/AiKiDo/VirtualBudoPass.html https://pa-mar.net/Study/AiKiDo/VirtualBudoPass.html
  - ankit219 2y ago
    This is done at a reranking step. It's again custom. You have two variables - 1/ relevance (which most algos focus on) 2/ Date. Create a new score based on some combination of weights for relevance and date. Eg; Could be 50% of date. If the document has 70% relevance, but was published yesterday, it's overall score would be 85%. (A conceptual idea). This is similar to how you do weighted sorting anywhere.
    - Pamar 2y ago
      Thank you! Btw, I notice only now that the link that was supposed to explain my question better is completely wrong. https://news.ycombinator.com/item?id=42589014 https://news.ycombinator.com/item?id=42589014 (But you still provided much needed clarification).
      - 2y ago