3 ms·

What Is Retrieval-Augmented Generation a.k.a. RAG?

84 points by teleforce 3y ago

john_cogs 3y ago
In every presentation I saw at re:Invent that mentioned AI, RAG was also mentioned. It seems to me that with AI, as models and products achieve parity, the amount of data that is accessible to a provider will be the key differentiator in quality of responses. Those who can gain access to the most customer data will be best-positioned to win the AI market.
- binarymax 3y ago
  Because AWS trying to upsell Bedrock and Opensearch instances.
minimaxir 3y ago
RAG is the one paradigm of modern AI that's completely uncontroversial (hallucinations aside) and will persist even if there's an AI-industry crash. The more annoying part of RAG is that it's so effective it's creating a lot of confusing best practices and a ton of venture capital around tooling, notably vector stores (which IMO are insufficiently differentiated) and libraries to integrate a RAG flow (LangChain being the common painful example, but RAG is simple enough that it doesn't even need its own abstraction).
deegles 3y ago
Using RAG feels like asking an acquaintance to write a book report by giving them semi-randomly cut out paragraphs from the book. Finetuning seems to be a solution, but there is still the issue of the LLM not really "learning" facts. What's the latest on actually adding knowledge to an LLM?
- filterfiber 3y ago
  The most likely "real solution" is going to be using various tricks and finetuning on higher context lengths to just extend the context window. However I will say knowledge graphs are becoming more popular. I made a comment the other day with a list of some of the popular methods: https://news.ycombinator.com/item?id=38476596 https://news.ycombinator.com/item?id=38476596 I completely forgot to mention ROME in my last comment, where you can modify facts within an LLM https://arxiv.org/pdf/2202.05262.pdf https://arxiv.org/pdf/2202.05262.pdf Note that there could be something more recent then these that I missed, but as far as I know knowledge-graphs/RAG are what most people are currently using, but there's a lot of work being focused on extending the context window.
- kristjansson 3y ago
  This doesn't deserve the downvotes it (currently) has. This is a completely accurate statement, simple RAG works well when questions are highly correlated with specifics chunks of documents. It does not allow an LLM to synthesize an entire corpus to an answer (e.g. a book report). Other modes of RAG that retrieve multiple times per generation (e.g. every so-many tokens) can improve performance, and intuitively feel closer to the 'sparse attention over the entire corpus' that RAG is approximating. However, fine-tuning on relevant, high quality, knowledge-rich question/answer pairs seems dominant, when such examples are available or can be generated.
  - filterfiber 3y ago
    I have to agree, RAG is effectively just compensating for data not fitting in the context window, and it has (potentially severe) limitations. Summarizing a book I think is a great example, RAG prevents LLMs from correlating all of the data. It will likely never be obsolete, but a larger context would be far preferable in nearly all scenerios.
  - svcrunch 3y ago
    I think your comment is accurate, but regarding your last point: "However, fine-tuning on relevant, high quality, knowledge-rich question/answer pairs seems dominant, when such examples are available or can be generated." How does one solve the problem of access-controlled data, if not through RAG? Do you imagine a separate version of the LLM for every user, reflecting their unique permissions on the data? Also, in scenarios where the data is being updated regularly, RAG provides much lower latency to the new information. Deletes also present a challenge for a pure-LLM approach.
    - willy_k 3y ago
      I don’t know much about the space, but couldn’t LoRA be used to (moderately) efficiently use a relevant and individual fine-tuned model alongside a general LLM?
- eskibars 3y ago
  Having worked on a (production, as-a-service RAG system), what we've learned is that fine tuning is not good for really adding/removing facts but is great for changing the form of the output. Anyscale has a great blog on this: https://www.anyscale.com/blog/fine-tuning-is-for-form-not-facts https://www.anyscale.com/blog/fine-tuning-is-for-form-not-fa... (I work at Vectara, not Anyscale, but appreciate the details and thought they put into this blog) Fine-tuning can help with certain areas of knowledge acquisition, but it's costly and frankly doesn't work that well when the knowledge you're trying to provide goes "against" the data that was trained into the core system. e.g. try to fine tune a model that there is a cure to <disease X> if you're a research org and it's incredibly hard to do that because the base model may be so convinced otherwise. Probably the most difficult thing though is using fine-tuning to get a model to "forget" or "not respond to" something that it shouldn't take a stance on. I always ask a fine-tuned model: "how do you calculate the fourth interior angle of a triangle." This is obviously a nonsense question, but helps to show how most LLMs will happily tell you to sum the interior angles of your first 3 angles and then subtract from 180. A well-tuned RAG system will say "sorry, I don't have information for that." It shows how it's less "guess-y" and hallucinated
  - namibj 3y ago
    Actually, selectively deleting facts like that does kinda work, essentially what you're doing is scolding-based reinforcement learning. For decent speed you'd want to probably use variance-reduction techniques like https://arxiv.org/abs/2205.08253 https://arxiv.org/abs/2205.08253 , though. Or a related (would be linked as cited by/cited in the linked when you look on semantic scholar) method that doesn't compute a hessian-vector product (which apparently interacts poorly with the non-linearity of the multi-layer perceptrons as provided by LLM-typical ReLU), in exchange for e.g. sizable batching (gradient accumulation!) at occasional checkpoints.
- amanvir_ 3y ago
  I've had this experience with RAG systems as well, but I've been wondering if structuring unstructured text (like a novel) in the form of a knowledge graph may solve this problem. Perhaps coupling a vector embedding search with a knowledge graph created from unstructured text could lead to more informed answers from LLMs? It seems as though some companies and researchers are already experimenting with this idea: https://www.nebula-graph.io/posts/graph-RAG https://www.nebula-graph.io/posts/graph-RAG.
- aik 3y ago
  No method today is a panacea. RAG naturally isn’t the approach you would take if you’re trying to summarize a large body of text, as in your example. That doesn’t make sense. That is not what it is for. What it’s awesome for is asking a somewhat specific question about some thing in a book if it can be answered by reviewing a paragraph or a few across the entire book. So to answer your question: Start by using the right tool for the job. The problem with this is: Some tools are prohibitively expensive, challenging, or very time-consuming to use.
  - deegles 3y ago
    But the issue remains where your questions have to have easy "retrieval" aspects. You can't ask general questions that a human would have no trouble with. A book report is the example I used because you need to have read the entire book. essentially a book-length context window.
    - namibj 3y ago
      Thankfully, there are methods (like causal transformers, but also some question/text/answer architectures) for not re-computing self-attention between the document/knowledge-base tokens across the iterations of autoregressive decoding for generating the response.
      - aik 3y ago
        Do you have an example of where these methods still produce good summaries? Eg if you adjust how re-computation of self-attention in autoregressive decoding / between token generations works to significantly decrease the amount of computation needed?
  - bradknowles 3y ago
    The real problem is knowing what tools exist for the job. Or that the right tool actually does exist. I would submit this is true for many fields of knowledge.
dmezzetti 3y ago
Nice article that covers the overall idea behind RAG. RAG is one of the most practical and production-ready ideas out of Generative AI. If you want a more hands on approach, txtai has a couple articles (disclaimer I'm the author of txtai). https://neuml.hashnode.dev/build-rag-pipelines-with-txtai https://neuml.hashnode.dev/build-rag-pipelines-with-txtai https://medium.com/neuml/judge-your-resume-with-ai-4223a2803509 https://medium.com/neuml/judge-your-resume-with-ai-4223a2803...
bloopernova 3y ago
Huggingface is blocked at work. If anyone else is in the same boat, here's an archive link: http://web.archive.org/web/20231122230456/https://huggingface.co/facebook/rag-token-nq http://web.archive.org/web/20231122230456/https://huggingfac...
- maxlamb 3y ago
  Do you know why it’s blocked? Seems weird to me
  - bloopernova 3y ago
    The policy is "no external genAI access", meaning no chatGPT, Bard, etc etc. Which, I think, has (mis)translated to "ban all AI-related sites" at the corporate content-filtering firewall which uses 3rd party rulesets. Though I can appreciate the tightrope that corporate IT has to walk. Many IT security departments been given that kind of directive by a leadership absolutely petrified of losing intellectual property. Another corp that several friends work at has blanket-banned a bunch of top-level domains, and so far their security ops folks haven't even responded to their complaints/requests. I'm very thankful I'm not affected by that kind of filtering; .dev and I think .io are affected.
politelemon 3y ago
Where can I learn about effectively using Vector stores? I understand RAG itself, been through a few tutorials and the workflow. But I mean using a Vector store isn't just a matter of generate some embeddings, slap it in a storage, and then start doing searches like cosine similarity, hoping for the best. Are there any guides on effectively using vector stores, and effectively searching for the use case you're dealing with? I don't even know what the various search types mean, just that everyone seems to use cosine similarity. Hope that's making sense... my question is analogous to learning to become familiar with SQL and indexes and CTEs, while it's a common thing to implement a three-tier application talking to a database. I feel that RAG might become a somewhat common feature of a team's tech stack alongside operational datastores (eg Postgres, MSSQL), because it is a low overhead to getting started and it persists, so it's cheaper. I just don't want to go at it blindly.
- dmezzetti 3y ago
  Couple articles that are txtai-specific but general enough in terms of vector search. Getting started with semantic search: https://medium.com/neuml/getting-started-with-semantic-search-a9fd9d8a48cf https://medium.com/neuml/getting-started-with-semantic-searc... txtai intro: https://medium.com/neuml/introducing-txtai-the-all-in-one-embeddings-database-c721f4ff91ad https://medium.com/neuml/introducing-txtai-the-all-in-one-em... Examples: https://neuml.github.io/txtai/examples/ https://neuml.github.io/txtai/examples/
- minimaxir 3y ago
  > But I mean using a Vector store isn't just a matter of generate some embeddings, slap it in a storage, and then start doing searches like cosine similarity, hoping for the best. That is 90% of the work with RAG. RAG requires a surprising amount of QA to ensure "the best" is sufficient for your use case. For clarification, cosine similarity is a metric not a search type, and people like it since a) it allows for more defined heuristics since cosine similarity is limited to [-1, 1] and b) it's computationally efficient (dot product, as with Euclidian distance) if the vectors are unit-normalized beforehand. There are indeed multiple search types but HNSW is the best 99% of the time for both performance and latency, and is implemented in almost every modern major vector store.
NKosmatos 3y ago
I’m preparing a small internal tool for my work to search documents and provide answers (with references), I’m thinking of using GPT4All [0], Danswer [1] and/or privateGPT [2]. The RAG technique is very close to what I have in mind, but I don’t want the LLM to “hallucinate” and generate answers on its own by synthesizing the source documents. As stated by many others, we’re living in interesting times. [0] https://gpt4all.io/index.html https://gpt4all.io/index.html [1] https://www.danswer.ai/ https://www.danswer.ai/ [2] https://github.com/imartinez/privateGPT https://github.com/imartinez/privateGPT

84 points by teleforce 3y ago

In every presentation I saw at re:Invent that mentioned AI, RAG was also mentioned. It seems to me that with AI, as models and products achieve parity, the amount of data that is accessible to a provider will be the key differentiator in quality of responses. Those who can gain access to the most customer data will be best-positioned to win the AI market.

binarymax 3y ago
Because AWS trying to upsell Bedrock and Opensearch instances.

minimaxir 3y ago

RAG is the one paradigm of modern AI that's completely uncontroversial (hallucinations aside) and will persist even if there's an AI-industry crash. The more annoying part of RAG is that it's so effective it's creating a lot of confusing best practices and a ton of venture capital around tooling, notably vector stores (which IMO are insufficiently differentiated) and libraries to integrate a RAG flow (LangChain being the common painful example, but RAG is simple enough that it doesn't even need its own abstraction).

deegles 3y ago

Using RAG feels like asking an acquaintance to write a book report by giving them semi-randomly cut out paragraphs from the book. Finetuning seems to be a solution, but there is still the issue of the LLM not really "learning" facts. What's the latest on actually adding knowledge to an LLM?

filterfiber 3y ago
The most likely "real solution" is going to be using various tricks and finetuning on higher context lengths to just extend the context window. However I will say knowledge graphs are becoming more popular. I made a comment the other day with a list of some of the popular methods: https://news.ycombinator.com/item?id=38476596 https://news.ycombinator.com/item?id=38476596 I completely forgot to mention ROME in my last comment, where you can modify facts within an LLM https://arxiv.org/pdf/2202.05262.pdf https://arxiv.org/pdf/2202.05262.pdf Note that there could be something more recent then these that I missed, but as far as I know knowledge-graphs/RAG are what most people are currently using, but there's a lot of work being focused on extending the context window.
kristjansson 3y ago
This doesn't deserve the downvotes it (currently) has. This is a completely accurate statement, simple RAG works well when questions are highly correlated with specifics chunks of documents. It does not allow an LLM to synthesize an entire corpus to an answer (e.g. a book report). Other modes of RAG that retrieve multiple times per generation (e.g. every so-many tokens) can improve performance, and intuitively feel closer to the 'sparse attention over the entire corpus' that RAG is approximating. However, fine-tuning on relevant, high quality, knowledge-rich question/answer pairs seems dominant, when such examples are available or can be generated.
- filterfiber 3y ago
  I have to agree, RAG is effectively just compensating for data not fitting in the context window, and it has (potentially severe) limitations. Summarizing a book I think is a great example, RAG prevents LLMs from correlating all of the data. It will likely never be obsolete, but a larger context would be far preferable in nearly all scenerios.
- svcrunch 3y ago
  I think your comment is accurate, but regarding your last point: "However, fine-tuning on relevant, high quality, knowledge-rich question/answer pairs seems dominant, when such examples are available or can be generated." How does one solve the problem of access-controlled data, if not through RAG? Do you imagine a separate version of the LLM for every user, reflecting their unique permissions on the data? Also, in scenarios where the data is being updated regularly, RAG provides much lower latency to the new information. Deletes also present a challenge for a pure-LLM approach.
  - willy_k 3y ago
    I don’t know much about the space, but couldn’t LoRA be used to (moderately) efficiently use a relevant and individual fine-tuned model alongside a general LLM?
eskibars 3y ago
Having worked on a (production, as-a-service RAG system), what we've learned is that fine tuning is not good for really adding/removing facts but is great for changing the form of the output. Anyscale has a great blog on this: https://www.anyscale.com/blog/fine-tuning-is-for-form-not-facts https://www.anyscale.com/blog/fine-tuning-is-for-form-not-fa... (I work at Vectara, not Anyscale, but appreciate the details and thought they put into this blog) Fine-tuning can help with certain areas of knowledge acquisition, but it's costly and frankly doesn't work that well when the knowledge you're trying to provide goes "against" the data that was trained into the core system. e.g. try to fine tune a model that there is a cure to <disease X> if you're a research org and it's incredibly hard to do that because the base model may be so convinced otherwise. Probably the most difficult thing though is using fine-tuning to get a model to "forget" or "not respond to" something that it shouldn't take a stance on. I always ask a fine-tuned model: "how do you calculate the fourth interior angle of a triangle." This is obviously a nonsense question, but helps to show how most LLMs will happily tell you to sum the interior angles of your first 3 angles and then subtract from 180. A well-tuned RAG system will say "sorry, I don't have information for that." It shows how it's less "guess-y" and hallucinated
- namibj 3y ago
  Actually, selectively deleting facts like that does kinda work, essentially what you're doing is scolding-based reinforcement learning. For decent speed you'd want to probably use variance-reduction techniques like https://arxiv.org/abs/2205.08253 https://arxiv.org/abs/2205.08253 , though. Or a related (would be linked as cited by/cited in the linked when you look on semantic scholar) method that doesn't compute a hessian-vector product (which apparently interacts poorly with the non-linearity of the multi-layer perceptrons as provided by LLM-typical ReLU), in exchange for e.g. sizable batching (gradient accumulation!) at occasional checkpoints.
amanvir_ 3y ago
I've had this experience with RAG systems as well, but I've been wondering if structuring unstructured text (like a novel) in the form of a knowledge graph may solve this problem. Perhaps coupling a vector embedding search with a knowledge graph created from unstructured text could lead to more informed answers from LLMs? It seems as though some companies and researchers are already experimenting with this idea: https://www.nebula-graph.io/posts/graph-RAG https://www.nebula-graph.io/posts/graph-RAG.
aik 3y ago
No method today is a panacea. RAG naturally isn’t the approach you would take if you’re trying to summarize a large body of text, as in your example. That doesn’t make sense. That is not what it is for. What it’s awesome for is asking a somewhat specific question about some thing in a book if it can be answered by reviewing a paragraph or a few across the entire book. So to answer your question: Start by using the right tool for the job. The problem with this is: Some tools are prohibitively expensive, challenging, or very time-consuming to use.
- deegles 3y ago
  But the issue remains where your questions have to have easy "retrieval" aspects. You can't ask general questions that a human would have no trouble with. A book report is the example I used because you need to have read the entire book. essentially a book-length context window.
  - namibj 3y ago
    Thankfully, there are methods (like causal transformers, but also some question/text/answer architectures) for not re-computing self-attention between the document/knowledge-base tokens across the iterations of autoregressive decoding for generating the response.
    - aik 3y ago
      Do you have an example of where these methods still produce good summaries? Eg if you adjust how re-computation of self-attention in autoregressive decoding / between token generations works to significantly decrease the amount of computation needed?
- bradknowles 3y ago
  The real problem is knowing what tools exist for the job. Or that the right tool actually does exist. I would submit this is true for many fields of knowledge.

dmezzetti 3y ago

Nice article that covers the overall idea behind RAG. RAG is one of the most practical and production-ready ideas out of Generative AI. If you want a more hands on approach, txtai has a couple articles (disclaimer I'm the author of txtai). https://neuml.hashnode.dev/build-rag-pipelines-with-txtai https://neuml.hashnode.dev/build-rag-pipelines-with-txtai https://medium.com/neuml/judge-your-resume-with-ai-4223a2803509 https://medium.com/neuml/judge-your-resume-with-ai-4223a2803...

bloopernova 3y ago

Huggingface is blocked at work. If anyone else is in the same boat, here's an archive link: http://web.archive.org/web/20231122230456/https://huggingface.co/facebook/rag-token-nq http://web.archive.org/web/20231122230456/https://huggingfac...

maxlamb 3y ago
Do you know why it’s blocked? Seems weird to me
- bloopernova 3y ago
  The policy is "no external genAI access", meaning no chatGPT, Bard, etc etc. Which, I think, has (mis)translated to "ban all AI-related sites" at the corporate content-filtering firewall which uses 3rd party rulesets. Though I can appreciate the tightrope that corporate IT has to walk. Many IT security departments been given that kind of directive by a leadership absolutely petrified of losing intellectual property. Another corp that several friends work at has blanket-banned a bunch of top-level domains, and so far their security ops folks haven't even responded to their complaints/requests. I'm very thankful I'm not affected by that kind of filtering; .dev and I think .io are affected.

politelemon 3y ago

Where can I learn about effectively using Vector stores? I understand RAG itself, been through a few tutorials and the workflow. But I mean using a Vector store isn't just a matter of generate some embeddings, slap it in a storage, and then start doing searches like cosine similarity, hoping for the best. Are there any guides on effectively using vector stores, and effectively searching for the use case you're dealing with? I don't even know what the various search types mean, just that everyone seems to use cosine similarity. Hope that's making sense... my question is analogous to learning to become familiar with SQL and indexes and CTEs, while it's a common thing to implement a three-tier application talking to a database. I feel that RAG might become a somewhat common feature of a team's tech stack alongside operational datastores (eg Postgres, MSSQL), because it is a low overhead to getting started and it persists, so it's cheaper. I just don't want to go at it blindly.

dmezzetti 3y ago
Couple articles that are txtai-specific but general enough in terms of vector search. Getting started with semantic search: https://medium.com/neuml/getting-started-with-semantic-search-a9fd9d8a48cf https://medium.com/neuml/getting-started-with-semantic-searc... txtai intro: https://medium.com/neuml/introducing-txtai-the-all-in-one-embeddings-database-c721f4ff91ad https://medium.com/neuml/introducing-txtai-the-all-in-one-em... Examples: https://neuml.github.io/txtai/examples/ https://neuml.github.io/txtai/examples/
minimaxir 3y ago
> But I mean using a Vector store isn't just a matter of generate some embeddings, slap it in a storage, and then start doing searches like cosine similarity, hoping for the best. That is 90% of the work with RAG. RAG requires a surprising amount of QA to ensure "the best" is sufficient for your use case. For clarification, cosine similarity is a metric not a search type, and people like it since a) it allows for more defined heuristics since cosine similarity is limited to [-1, 1] and b) it's computationally efficient (dot product, as with Euclidian distance) if the vectors are unit-normalized beforehand. There are indeed multiple search types but HNSW is the best 99% of the time for both performance and latency, and is implemented in almost every modern major vector store.

NKosmatos 3y ago

I’m preparing a small internal tool for my work to search documents and provide answers (with references), I’m thinking of using GPT4All [0], Danswer [1] and/or privateGPT [2]. The RAG technique is very close to what I have in mind, but I don’t want the LLM to “hallucinate” and generate answers on its own by synthesizing the source documents. As stated by many others, we’re living in interesting times. [0] https://gpt4all.io/index.html https://gpt4all.io/index.html [1] https://www.danswer.ai/ https://www.danswer.ai/ [2] https://github.com/imartinez/privateGPT https://github.com/imartinez/privateGPT