4 ms·

How does it handle context curation in long conversations? Does it detect user intent for each message to see if it can act purely on existing context, or if it

How does it handle context curation in long conversations? Does it detect user intent for each message to see if it can act purely on existing context, or if it needs to do another retrieval and/or purge/summarize previous discussion and context?

carlos_marcial 11mo ago
Great questions! Here's how ChatRAG handles context curation: 1. Intent Detection Per Message Every message is classified to decide the retrieval strategy: - conversational (greetings, small talk) → skip RAG entirely - document_search (explicit knowledge queries) → always retrieve - complex/exploratory → adaptive retrieval - tool_required → route to MCP tools instead This prevents unnecessary retrieval on follow-ups like "thanks" or "can you explain that differently?" 2. Adaptive Retrieval When retrieval is needed, the system adjusts based on query characteristics: - Similarity thresholds: 0.45 for general queries → 0.75 for exact data requests - Result limits: 15 for specific queries → 30 for broad exploration - Strategy: semantic-only vs. hybrid vs. temporal-boosted "What was Q3 2024 revenue?" triggers exact_match mode at 0.75 threshold, while "Tell me about the company" uses standard semantic search at 0.45. 3. Conversation History Full conversation history is passed to the model without truncation—modern LLMs have 100K-200K token windows, and RAG context is freshly curated each turn rather than accumulated. 4. Not Implemented (Yet) - Conversation summarization/compression - Explicit context window management - "Forget previous context" mechanisms These would matter at scale or for very long sessions. That said, ChatRAG is a fully customizable boilerplate, and the idea is for developers to keep building on top of it, as I will too.
- cluckindan 11mo ago
  Did you just copy-paste this answer from an LLM? It sounds exactly like a generative model, and there are obvious issues with the response. For example, ”RAG context is freshly curated each turn rather than accumulated” is nonsensical: it implies that context is curated instead of replaced, but also that the previous context is forgotten and replaced, and neither means that full conversation history (including context) is passed without truncation. Also, no mention of tool-based retrieval. I have a hunch this is not a real product, just AI slop. Prove me wrong.
  - carlos_marcial 11mo ago
    Yes, of course I used an LLM for the response, but not just any LLM. I used an instance of ChatRAG that has all the ChatRAG documentation built into it. I’m a solo developer trying to build software, do the marketing, handle customer support, and also be a dad and a husband. Of course I’m going to use my own product to help me out. This is precisely why I know ChatRAG is good, I use it myself daily. That said, let me clarify the points you raised. On "freshly curated each turn rather than accumulated": this is actually the correct behavior, though I can see how the phrasing was confusing. The key is that conversation history and RAG context are two different things. The full conversation history... all previous user and assistant message... is always passed to the model without truncation. But the retrieved document chunkz are a separate layer. Those get requeried based on the current message and injected into the system prompt fresh each turn. Previous RAG results dont accumulate, they are replaced with whatever is most relevant to the current query. Why this design? Say you ask about 'Q3 revenue' and then follow up with 'What about employe benefits?' If the system accumulated chunks, you'd have Q3 revenue data polluting the context when you're now asking about something completely different. Fresh retrieval ensures the context stays relevant to what you're actually asking about right now. On tool-based retrieval: the system does have this through MCP integration. Query classification detects when tools are needed (Gmail, Calendar, Drive, Zapier, etc.) and routes accordingly. The model receives both the RAG context and access to tools, so it can use document knowledge and execute external actions within the same response when needed Let me know if you have any other questions. I’ll be more than happy to answer, with the help of my ChatRAG instance, any other doubts you may have !

carlos_marcial 11mo ago

Great questions! Here's how ChatRAG handles context curation: 1. Intent Detection Per Message Every message is classified to decide the retrieval strategy: - conversational (greetings, small talk) → skip RAG entirely - document_search (explicit knowledge queries) → always retrieve - complex/exploratory → adaptive retrieval - tool_required → route to MCP tools instead This prevents unnecessary retrieval on follow-ups like "thanks" or "can you explain that differently?" 2. Adaptive Retrieval When retrieval is needed, the system adjusts based on query characteristics: - Similarity thresholds: 0.45 for general queries → 0.75 for exact data requests - Result limits: 15 for specific queries → 30 for broad exploration - Strategy: semantic-only vs. hybrid vs. temporal-boosted "What was Q3 2024 revenue?" triggers exact_match mode at 0.75 threshold, while "Tell me about the company" uses standard semantic search at 0.45. 3. Conversation History Full conversation history is passed to the model without truncation—modern LLMs have 100K-200K token windows, and RAG context is freshly curated each turn rather than accumulated. 4. Not Implemented (Yet) - Conversation summarization/compression - Explicit context window management - "Forget previous context" mechanisms These would matter at scale or for very long sessions. That said, ChatRAG is a fully customizable boilerplate, and the idea is for developers to keep building on top of it, as I will too.

cluckindan 11mo ago
Did you just copy-paste this answer from an LLM? It sounds exactly like a generative model, and there are obvious issues with the response. For example, ”RAG context is freshly curated each turn rather than accumulated” is nonsensical: it implies that context is curated instead of replaced, but also that the previous context is forgotten and replaced, and neither means that full conversation history (including context) is passed without truncation. Also, no mention of tool-based retrieval. I have a hunch this is not a real product, just AI slop. Prove me wrong.
- carlos_marcial 11mo ago
  Yes, of course I used an LLM for the response, but not just any LLM. I used an instance of ChatRAG that has all the ChatRAG documentation built into it. I’m a solo developer trying to build software, do the marketing, handle customer support, and also be a dad and a husband. Of course I’m going to use my own product to help me out. This is precisely why I know ChatRAG is good, I use it myself daily. That said, let me clarify the points you raised. On "freshly curated each turn rather than accumulated": this is actually the correct behavior, though I can see how the phrasing was confusing. The key is that conversation history and RAG context are two different things. The full conversation history... all previous user and assistant message... is always passed to the model without truncation. But the retrieved document chunkz are a separate layer. Those get requeried based on the current message and injected into the system prompt fresh each turn. Previous RAG results dont accumulate, they are replaced with whatever is most relevant to the current query. Why this design? Say you ask about 'Q3 revenue' and then follow up with 'What about employe benefits?' If the system accumulated chunks, you'd have Q3 revenue data polluting the context when you're now asking about something completely different. Fresh retrieval ensures the context stays relevant to what you're actually asking about right now. On tool-based retrieval: the system does have this through MCP integration. Query classification detects when tools are needed (Gmail, Calendar, Drive, Zapier, etc.) and routes accordingly. The model receives both the RAG context and access to tools, so it can use document knowledge and execute external actions within the same response when needed Let me know if you have any other questions. I’ll be more than happy to answer, with the help of my ChatRAG instance, any other doubts you may have !