3 ms·

Show HN: Demystifying Advanced RAG Pipelines

131 points by pchunduri6 3y ago

I've built an advanced RAG (Retrieval-Augmented Generation) pipeline from scratch to demystify the complex mechanics of modern LLM-powered Question Answering systems. This repository features: -- An implementation of a sub-question query engine from scratch to answer complex user questions. -- Illustrative explanations that unveil the inner workings of the system. -- An analysis of the challenges I faced while working with the system, like prompt engineering and cost estimation. -- Qualitative comparison with similar frameworks like LlamaIndex, offering a broader perspective. Key Takeaway: While Modern QA pipelines with advanced RAG abstractions may seem complex, they are fundamentally powered by a series of LLM calls with meticulous prompt design. Hoping that this repository provides intuitive insights for building more robust and efficient RAG systems. All feedback is warmly welcomed!

nilsbunger 3y ago
I love this write up. Thank you ! I’m looking for more resources like this - clear examples of composing LLMs into useful systems. Some of the cookbook examples in langchain, chainlit , etc have been useful too.
- pchunduri6 3y ago
  Thanks for the kind words! +1 for chainlit. I love their documentation. Do you have any specific use-cases in mind that would benefit from such pipelines?
SubiculumCode 3y ago
As a researcher I've been interested in developing a RAG pipeline populated with research articles on my topic of study. Does it fit easily in the RAG approach to also return excerpts from the actual documents as to help me verify, at a glance the source and veracity of LLM outputs?
- tayo42 3y ago
  You can do the summarization part how ever you want. You don't even need have an llm summarize what the program found. The context that includes the answer, so you can just include that in your final response.
- pchunduri6 3y ago
  Yes, this is an excellent RAG use-case! The vector index that I use in the repository uses EvaDB [1] to retrieve the top-K matches to the user queries from the available data sources. So, you can manually inspect the best matches to your query from the research article and verify the correctness of the LLM responses. [1] https://github.com/georgia-tech-db/evadb https://github.com/georgia-tech-db/evadb
- jarulraj 3y ago
  It is currently not possible to get rigorous summaries of paper chunks using GPT-4.
jondwillis 3y ago
This is a great README! It clearly breaks down some approaches to RAG. I also approciate how you strive to de-mystify what’s going on under the hood, which is in many ways VERY simple. This seems very similar to LangSmith’s trace monitoring, which I have been leaning on heavily for observability. You also mention LlamaIndex— how do you see your project fitting into the ecosystem? I don’t think I would able to use this yet because it is serial. Is it possible to non-serially issue independent sub-question queries? In my experimental agent system, waggledance.ai[1], I have been working on a pre-agent step of picking and synthesizing the right context and tools[2] for a given subtask of a larger goal, and it seems to be boosting results. It looks like now I have to try sub-question answering in the mix as well. [1] demo - https://waggledance.ai https://waggledance.ai [2] relevant code sample - https://github.com/agi-merge/waggle-dance/blob/1b14163c24fd2c8f9689921ed2ef1e9451b00876/packages/agent/src/strategy/callExecutionAgent.ts#L175 https://github.com/agi-merge/waggle-dance/blob/1b14163c24fd2...
- pchunduri6 3y ago
  Thanks for the kind words and the great questions! -- LlamaIndex has some excellent abstractions. In fact, I started off this project with LlamaIndex using their sub-question query engine. However, I found that the abstractions often obfuscate the prompt templates and the pipeline itself from the user. I found that writing my own pipeline was easier than trying to figure out how to engineer the prompts that LlamaIndex was using. -- It is possible to non-serially issue independent sub-question queries (e.g., using async io). LlamaIndex does something similar. However, I would be extra careful while issuing parallel sub-queries due to the brittle nature of the system. -- Cool project! I like the fact that the agent decision-making is clearly shown in the UI. A few questions: 1) How do you handle LLM output inconsistencies? 2) Can the user change the prompts for tasks or sub-tasks if the output is not satisfactory? Overall, a great idea and this sub-question query engine might simplify some of the abstractions here.
  - jondwillis 3y ago
    1) What do you mean by LLM output inconsistencies? Most LLM responses are parsed, and then if that fails, an attempt to auto-fix them is made by re-running the previous output through a rewriting/schema prompt. 2) I want that feature too, and have it planned! I want to have a sort of knowledge / progress dashboard, where users can "chat their data". I also want to add to each sub-task the ability to restart from that point. Essentially, since the project is a running on an entirely serverless architecture, this means serializing everything important, canceling current functions, and then re-hydrating from a certain point and calling the serverless functions again.
ckrapu 3y ago
I’m really interested in content explaining how to navigate graphs of embedded items for Q/A. Any resources on how to do this or arguments for why it’s a bad approach? For example, if my top K docs aren’t answering the question but each are linked to neighbors, I’d want to know some folk wisdom or tricks for structuring the neighbor graph to cheaply expand the set of useful results.
- TruthWillHurt 3y ago
  Not using neighbour, but Autogen has a multi-agent pipeline where if a question is not answered by top K docs, the agent can request "UPDATE CONTEXT" and pull the next set of results. See example 5 here: https://github.com/microsoft/autogen/blob/main/notebook/agentchat_RetrieveChat.ipynb https://github.com/microsoft/autogen/blob/main/notebook/agen...
- jondwillis 3y ago
  You could in theory create a tool/function like “Context Retrieval”, give it to an Agent, and instruct the Agent to paginate through it as needed. This would add some errors due to LLM usage and latency though. And then of course you would still need to design the graph structure. Maybe neo4j or similar graph dbs would be useful? I have seen a langchain integration for instance: https://python.langchain.com/docs/integrations/providers/neo4j https://python.langchain.com/docs/integrations/providers/neo...
- jarulraj 3y ago
  Wouldn't more semantically related neighbors be retrieved by just increasing K?
  - ckrapu 3y ago
    Potentially, yes! The scenario I am imagining is that context A as context for a query yields results 1, 2, and 3. Sometimes, finding neighbors of 1 (ie not necessarily in the top K w.r.t A) instead of going to results 4, 5, 6 might be better.
Ephil012 3y ago
Pretty cool tutorial. As a side note, it is pretty hard to evaluate these pipelines for quality once you build them since there's not many standard practices yet given how new this all is. If it's helpful to anyone else, we built a free open source tool within my company that is basically a collection of premade metrics for determining the quality of these pipelines. https://github.com/TonicAI/tvalmetrics https://github.com/TonicAI/tvalmetrics
- pchunduri6 3y ago
  This is really useful! Using LLM-assisted evaluation seems like the way to go for evaluating RAG applications. One issue I've faced while evaluating responses using GPT-4 is that the evaluation cost can go out of hand rather quickly. Do you have any measures in place or ideas on how to handle this?
  - Ephil012 3y ago
    Unfortunately, right now the LLM cost is just a fundamental issue. I think it is hard to get around because comparing answer quality usually involves understanding the question and answer itself which is a task that's really well suited to LLMs. One thing we have considered is some forms of evaluation could be replaced simply with using the embeddings of the question, context, and answer instead of using the LLM model for analysis. The idea is you could compare all the embeddings to get a rough idea of the performance based on similarity. That should in theory reduce costs. The only other alternative is just to use less advanced models which are cheaper.
yding 3y ago
Great explanation!

131 points by pchunduri6 3y ago

nilsbunger 3y ago

I love this write up. Thank you ! I’m looking for more resources like this - clear examples of composing LLMs into useful systems. Some of the cookbook examples in langchain, chainlit , etc have been useful too.

pchunduri6 3y ago
Thanks for the kind words! +1 for chainlit. I love their documentation. Do you have any specific use-cases in mind that would benefit from such pipelines?

SubiculumCode 3y ago

As a researcher I've been interested in developing a RAG pipeline populated with research articles on my topic of study. Does it fit easily in the RAG approach to also return excerpts from the actual documents as to help me verify, at a glance the source and veracity of LLM outputs?

tayo42 3y ago
You can do the summarization part how ever you want. You don't even need have an llm summarize what the program found. The context that includes the answer, so you can just include that in your final response.
pchunduri6 3y ago
Yes, this is an excellent RAG use-case! The vector index that I use in the repository uses EvaDB [1] to retrieve the top-K matches to the user queries from the available data sources. So, you can manually inspect the best matches to your query from the research article and verify the correctness of the LLM responses. [1] https://github.com/georgia-tech-db/evadb https://github.com/georgia-tech-db/evadb
jarulraj 3y ago
It is currently not possible to get rigorous summaries of paper chunks using GPT-4.

jondwillis 3y ago

This is a great README! It clearly breaks down some approaches to RAG. I also approciate how you strive to de-mystify what’s going on under the hood, which is in many ways VERY simple. This seems very similar to LangSmith’s trace monitoring, which I have been leaning on heavily for observability. You also mention LlamaIndex— how do you see your project fitting into the ecosystem? I don’t think I would able to use this yet because it is serial. Is it possible to non-serially issue independent sub-question queries? In my experimental agent system, waggledance.ai[1], I have been working on a pre-agent step of picking and synthesizing the right context and tools[2] for a given subtask of a larger goal, and it seems to be boosting results. It looks like now I have to try sub-question answering in the mix as well. [1] demo - https://waggledance.ai https://waggledance.ai [2] relevant code sample - https://github.com/agi-merge/waggle-dance/blob/1b14163c24fd2c8f9689921ed2ef1e9451b00876/packages/agent/src/strategy/callExecutionAgent.ts#L175 https://github.com/agi-merge/waggle-dance/blob/1b14163c24fd2...

pchunduri6 3y ago
Thanks for the kind words and the great questions! -- LlamaIndex has some excellent abstractions. In fact, I started off this project with LlamaIndex using their sub-question query engine. However, I found that the abstractions often obfuscate the prompt templates and the pipeline itself from the user. I found that writing my own pipeline was easier than trying to figure out how to engineer the prompts that LlamaIndex was using. -- It is possible to non-serially issue independent sub-question queries (e.g., using async io). LlamaIndex does something similar. However, I would be extra careful while issuing parallel sub-queries due to the brittle nature of the system. -- Cool project! I like the fact that the agent decision-making is clearly shown in the UI. A few questions: 1) How do you handle LLM output inconsistencies? 2) Can the user change the prompts for tasks or sub-tasks if the output is not satisfactory? Overall, a great idea and this sub-question query engine might simplify some of the abstractions here.
- jondwillis 3y ago
  1) What do you mean by LLM output inconsistencies? Most LLM responses are parsed, and then if that fails, an attempt to auto-fix them is made by re-running the previous output through a rewriting/schema prompt. 2) I want that feature too, and have it planned! I want to have a sort of knowledge / progress dashboard, where users can "chat their data". I also want to add to each sub-task the ability to restart from that point. Essentially, since the project is a running on an entirely serverless architecture, this means serializing everything important, canceling current functions, and then re-hydrating from a certain point and calling the serverless functions again.

ckrapu 3y ago

I’m really interested in content explaining how to navigate graphs of embedded items for Q/A. Any resources on how to do this or arguments for why it’s a bad approach? For example, if my top K docs aren’t answering the question but each are linked to neighbors, I’d want to know some folk wisdom or tricks for structuring the neighbor graph to cheaply expand the set of useful results.

TruthWillHurt 3y ago
Not using neighbour, but Autogen has a multi-agent pipeline where if a question is not answered by top K docs, the agent can request "UPDATE CONTEXT" and pull the next set of results. See example 5 here: https://github.com/microsoft/autogen/blob/main/notebook/agentchat_RetrieveChat.ipynb https://github.com/microsoft/autogen/blob/main/notebook/agen...
jondwillis 3y ago
You could in theory create a tool/function like “Context Retrieval”, give it to an Agent, and instruct the Agent to paginate through it as needed. This would add some errors due to LLM usage and latency though. And then of course you would still need to design the graph structure. Maybe neo4j or similar graph dbs would be useful? I have seen a langchain integration for instance: https://python.langchain.com/docs/integrations/providers/neo4j https://python.langchain.com/docs/integrations/providers/neo...
jarulraj 3y ago
Wouldn't more semantically related neighbors be retrieved by just increasing K?
- ckrapu 3y ago
  Potentially, yes! The scenario I am imagining is that context A as context for a query yields results 1, 2, and 3. Sometimes, finding neighbors of 1 (ie not necessarily in the top K w.r.t A) instead of going to results 4, 5, 6 might be better.

Ephil012 3y ago

Pretty cool tutorial. As a side note, it is pretty hard to evaluate these pipelines for quality once you build them since there's not many standard practices yet given how new this all is. If it's helpful to anyone else, we built a free open source tool within my company that is basically a collection of premade metrics for determining the quality of these pipelines. https://github.com/TonicAI/tvalmetrics https://github.com/TonicAI/tvalmetrics

pchunduri6 3y ago
This is really useful! Using LLM-assisted evaluation seems like the way to go for evaluating RAG applications. One issue I've faced while evaluating responses using GPT-4 is that the evaluation cost can go out of hand rather quickly. Do you have any measures in place or ideas on how to handle this?
- Ephil012 3y ago
  Unfortunately, right now the LLM cost is just a fundamental issue. I think it is hard to get around because comparing answer quality usually involves understanding the question and answer itself which is a task that's really well suited to LLMs. One thing we have considered is some forms of evaluation could be replaced simply with using the embeddings of the question, context, and answer instead of using the LLM model for analysis. The idea is you could compare all the embeddings to get a rough idea of the performance based on similarity. That should in theory reduce costs. The only other alternative is just to use less advanced models which are cheaper.

yding 3y ago

Great explanation!