4 ms·

I don’t understand what the memory is or does from the README. Can anyone explain how it works differently from vector database results in vanilla RAG applicati

by davedx 2y ago

I don’t understand what the memory is or does from the README. Can anyone explain how it works differently from vector database results in vanilla RAG applications?

jszymborski 2y ago
Ok, I think I get it now from scanning the paper and reading Eq. 1 and 2. Normally RAG just sends your query `q` to a information retrieval function which searches a database of documents using full-text search or vector search. Those documents are then passed to a generative model along with your query to give you your final answer. MemoRAG instead immediately passes `q` to a generative model to generate some uninformed response `y`. `y` is then passed to the information retrieval function. Then, just like vanilla RAG, `q` and the retrieved documents are sent to a generative model to give you your final answer. Not sure how this is any more "memory-based" than regular RAG, but it seems interesting. Def check out the pre-print, especially eq. 1 and 2. https://arxiv.org/abs/2409.05591 https://arxiv.org/abs/2409.05591 EDIT: The "memory" part comes from the first generative model being able to handle larger context, covered in Section 2.1
- isoprophlex 2y ago
  thanks for boiling it down to the most salient point... to me, their approach is just query rewriting, which is pretty standard when doing RAG.
  - jszymborski 2y ago
    There's a lot there about the generative model ("Memory Models") in the paper, so perhaps I've misrepresented it, but generally speaking yah I agree with you. It doesn't sound like a fundamental change to how we think about RAG, but it might be a nice formalization of an incremental improvement :)
  - opdahl 2y ago
    Agreed. In the RAG space there are a million Open Source projects on GitHub all calling it memory, recreating the same thing over and over again.
  - fraboniface 2y ago
    Not exactly, they use a small but long-context model that has the whole dataset in its context (or a large part of it) to generate the chunks as elements of the reply, before passing those to the final model. So the retrieval itself is different, there is no embeddeding model or vector db.
- danielbln 2y ago
  I wonder how this different from HyDE. https://docs.haystack.deepset.ai/docs/hypothetical-document-embeddings-hyde https://docs.haystack.deepset.ai/docs/hypothetical-document-...
  - jszymborski 2y ago
    It seems to be fundamentally the same deal except instead of passing `q` to GPT-4, they have some long-context "Memory Model" (whose details I've yet to fully understand). Also, MemoRAG uses a more conventional Retrieve/Generate pipeline downstream of the generated queries than "Contriever" (whose details I similarly haven't informed myself on). It would be interesting to see a performance comparison, it certainly seems the most relevant one (that or an ablation of their "memory model" with the LLMs upon which they are based).
    - spmurrayzzz 2y ago
      > they have some long-context "Memory Model" (whose details I've yet to fully understand) Section 2.2 of the paper[1] goes into this in more detail. They pretrain the draft model using the redpajama dataset, followed by a supervised fine-tuning step. The training objective "aims to maximize the generation probability of the next token given the KV cache of the previous memory tokens". This suggests that any model with long context and good retrieval performance could do the same job (and maybe better in the case of the SOTA frontier models). [1] https://arxiv.org/pdf/2409.05591 https://arxiv.org/pdf/2409.05591
- bbor 2y ago
  Not sure how this is any more "memory-based" than regular RAG, but it seems interesting. I can't remember where I read this joke, but as a self-proclaimed Cognitive Engineer I think about it every day: "An AI startup's financial evaluation is directly proportional to how many times they can cram 'mind' into their pitch deck!"
- mycall 2y ago
  I do memory-based RAGs using Semantic Kernel function calls which do specialized memory caches with useful calculations based on telemetry data. It is so simple to do and I love the LLM figures out how to call the SemanticFunctions on its own. ollama and langchain can do something simimlar.

by davedx 2y ago

I don’t understand what the memory is or does from the README. Can anyone explain how it works differently from vector database results in vanilla RAG applications?

jszymborski 2y ago

Ok, I think I get it now from scanning the paper and reading Eq. 1 and 2. Normally RAG just sends your query `q` to a information retrieval function which searches a database of documents using full-text search or vector search. Those documents are then passed to a generative model along with your query to give you your final answer. MemoRAG instead immediately passes `q` to a generative model to generate some uninformed response `y`. `y` is then passed to the information retrieval function. Then, just like vanilla RAG, `q` and the retrieved documents are sent to a generative model to give you your final answer. Not sure how this is any more "memory-based" than regular RAG, but it seems interesting. Def check out the pre-print, especially eq. 1 and 2. https://arxiv.org/abs/2409.05591 https://arxiv.org/abs/2409.05591 EDIT: The "memory" part comes from the first generative model being able to handle larger context, covered in Section 2.1

isoprophlex 2y ago
thanks for boiling it down to the most salient point... to me, their approach is just query rewriting, which is pretty standard when doing RAG.
- jszymborski 2y ago
  There's a lot there about the generative model ("Memory Models") in the paper, so perhaps I've misrepresented it, but generally speaking yah I agree with you. It doesn't sound like a fundamental change to how we think about RAG, but it might be a nice formalization of an incremental improvement :)
- opdahl 2y ago
  Agreed. In the RAG space there are a million Open Source projects on GitHub all calling it memory, recreating the same thing over and over again.
- fraboniface 2y ago
  Not exactly, they use a small but long-context model that has the whole dataset in its context (or a large part of it) to generate the chunks as elements of the reply, before passing those to the final model. So the retrieval itself is different, there is no embeddeding model or vector db.
danielbln 2y ago
I wonder how this different from HyDE. https://docs.haystack.deepset.ai/docs/hypothetical-document-embeddings-hyde https://docs.haystack.deepset.ai/docs/hypothetical-document-...
- jszymborski 2y ago
  It seems to be fundamentally the same deal except instead of passing `q` to GPT-4, they have some long-context "Memory Model" (whose details I've yet to fully understand). Also, MemoRAG uses a more conventional Retrieve/Generate pipeline downstream of the generated queries than "Contriever" (whose details I similarly haven't informed myself on). It would be interesting to see a performance comparison, it certainly seems the most relevant one (that or an ablation of their "memory model" with the LLMs upon which they are based).
  - spmurrayzzz 2y ago
    > they have some long-context "Memory Model" (whose details I've yet to fully understand) Section 2.2 of the paper[1] goes into this in more detail. They pretrain the draft model using the redpajama dataset, followed by a supervised fine-tuning step. The training objective "aims to maximize the generation probability of the next token given the KV cache of the previous memory tokens". This suggests that any model with long context and good retrieval performance could do the same job (and maybe better in the case of the SOTA frontier models). [1] https://arxiv.org/pdf/2409.05591 https://arxiv.org/pdf/2409.05591
bbor 2y ago
Not sure how this is any more "memory-based" than regular RAG, but it seems interesting. I can't remember where I read this joke, but as a self-proclaimed Cognitive Engineer I think about it every day: "An AI startup's financial evaluation is directly proportional to how many times they can cram 'mind' into their pitch deck!"
mycall 2y ago
I do memory-based RAGs using Semantic Kernel function calls which do specialized memory caches with useful calculations based on telemetry data. It is so simple to do and I love the LLM figures out how to call the SemanticFunctions on its own. ollama and langchain can do something simimlar.