6 ms·

Docs bots like these are deceptively hard to get right in production. Retrieval is super sensitive to how you chunk/parse documentation and how you end up struc

• skrebbel 1y ago
  Why RAG at all? We concatenated all our docs and tutorials into a text file, piped it all into the AI right along with the question, and the answers are pretty great. Cost was, last I checked, roughly 50c per question. Probably scales linearly with how much docs you have. This feels expensive but compared to a human writing an answer it's peanuts. Plus (assuming the customer can choose to use the AI or a human), it's great customer experience because the answer is there that much faster. I feel like this is a no-brainer. Tbh with the context windows we have these days, I don't completely understand why RAG is a thing anymore for support tools.
  • Rygian 1y ago
    What you describe sounds like poor man's RAG. Or lazy man's. You're just doing the augmentation at each prompt.
  • cluckindan 1y ago
    With RAG the cost per question would be low single-digit pennies.
  • IceDane 1y ago
    Because llms still suck at actually using all that context at once. And surely you can see yourself that your solution doesn't scale. It's great that it works for your specific case but I'm sure you can come up with a scenario where it's just not feasible.
  • cube2222 1y ago
    This works as long as your docs are below the max context size (and even then, as you approach larger context sizes, quality degrades). Re cost though, you can usually reduce the cost significantly with context caching here. However, in general, I’ve been positively surprised with how effective Claude Code is at grep’ing through huge codebases. Thus, I think just putting a Claude Code-like agent in a loop, with a grep tool on your docs, and a system prompt that contains just a brief overview of your product and brief summaries of all the docs pages, would likely be my go to.
    • bee_rider 1y ago
      Oh man, maybe this would cause people to write docs that are easy to grep through. Let’s start up that feedback loop immediately, please.
      • cluckindan 1y ago
        How will you grep synonyms or phrases with different word choices?
        • bee_rider 1y ago
          I’m hoping that the documentation will be structured in a way such that Claude can easily come up with good grep regexes. If Claude can do it, I can probably do it only a little bit worse.
  • emil_sorensen 1y ago
    Accuracy drops hard with context length still. Especially in more technical domains. Plus latency and cost.
  • llm_nerd 1y ago
    What you described is RAG. Inefficient RAG, but still RAG. And it's inefficient in two ways- -you're using extra tokens for every query, which adds up. -you're making the LLM less precise by overloading it with potentially irrelevant extra info making it harder for it to needle in a haystack the specific relevant answer. Filtering (e.g. embedding similarity & BM25) and re-ranking/pruning what you provide to RAG is an optimization. It optimizes the tokens, the processing time, and optimizes the answer in an ideal world. Most LLMs are far more effective if your RAG is limited to what is relevant to the question.
    • TZubiri 1y ago
      I don't think it's RAG, RAG is specifically separating the search space from the LLM context-window or training set and giving the LLM tools to search in inference-time.
      • llm_nerd 1y ago
        In this case their Retrieval stage is "SELECT *", basically, so sure I'm being loose with the terminology, but otherwise it's just a non-selective RAG. Okay ..AG. RAG is selecting pertinent information to supply to the LLM with your query. In this case they decided that everything was pertinent, and the net result is just reduced efficiency. But if it works for them, eh.
        • TZubiri 1y ago
          I'm not sure we are talking about the same thing. The root comment talks about concatenating all doc files into a loong text string, and adding that as a system/user prompt to the LLM at inference time before the actual question. You mention the retrieval stage being a SELECT *? I don't think there's any SQL involved here.
          • llm_nerd 1y ago
            I was being rhetorical. The R in RAG is filtering augmentation data (the A) for things that might or might not be related to the query. Including everything is just a lazy form of RAG -- the rhetorical SELECT *. >and adding that as a system/user prompt to the LLM at inference time You understand this is all RAG is, right? RAG is any additional system to provide contextually relevant (and often more timely) supporting information to a baked model. People sometimes project RAG out to be a specific combination of embeddings, chunking, vector DBs, etc. But that is ancillary. RAG is simply selecting the augmentation data and supplying it with the question. Anyways, I think this thread has reached a conclusion and there really isn't much more value in it. Cheers.
  • TZubiri 1y ago
    That is not particularly cheap, especially since it scales linearly with doc size, and therefore time. Additionally the quality of loading the context-window decreases linearly as well, just because your model can handle 1M tokens it doesn't mean that it WILL remember 1M tokens, it just means that it CAN RAG fixes this, in the simplest configuration a RAG can be an index, and the only context you give the LLM is the table of contents, and you let it search through the index. Should it be a surprise that this is cheaper and more efficient? Loading the context window is like a library having every book open at every page at the same time instead of using the dewey decimal system
• chrismorgan 1y ago
  Why do you say “deceptively hard” instead of “fundamentally impossible”? You can increase the probability it’ll give good answers, but you can never guarantee it. It’s then a question of what degree of wrongness is acceptable, and how you signal that. In this specific case, what it said sounds to me (as a Shopify non-user) entirely reasonable, it’s just wrong in a subtle but rather crucial way, which is also mildly tricky to test.
  • whatsgonewrongg 1y ago
    A human answering every question is also not guaranteed to give good answers; anyone that has communicated with customer service knows that. So calling it impossible may be correct, but not useful. (We tend to have far fewer evals for such humans though.)
    • girvo 1y ago
      A human will tell you “I am not sure, and will have to ask engineering and get back to you in a few days”. None of these LLMs do that yet, they’re biased towards giving some answer, any answer.
      • whatsgonewrongg 1y ago
        You’re right that some humans will, and most LLMs won’t. But humans can be just as confidently wrong. And we incentivize them to make decisions quickly, in a way that costs the company less money.
      • dcre 1y ago
        This is not really true. If you give a decent model docs in the prompt and tell them to answer based on the docs and say “I don’t know” if the answer isn’t there, they do it (most of the time).
        • SecretDreams 1y ago
          > most of the time This is doing some heavy lifting
• PeterStuer 1y ago
  Indeed. Dabbling in 'RAG' (which for better or worse has become a tag for anything context retrieval) for more complex documentation and more intricate questions, you will very quickly realize that you really need to go far beyond simple 'chunking', and end up with a subsystem that constructs more than one very intricate knowledge graphs for supporting different kinds of questions the users might ask. For example: a simple question such as "What exactly is an 'Essential Entity'? is better handled by Knowledge Representation A as opposed to "Can you provide a gap and risk analysis on my 2025 draft compliance statement (uploaded) in light of the current GDPR, NIS-2 and the AI Act?" (My domain is regulatory compliance, so maybe this goes beyond pure documentation but I'm guessing pushed far enough the same complexities arise)
  • dingnuts 1y ago
    This is sort of hilarious; to use an LLM as a good search interface first build.. a search engine. I guess this is why Kagi Quick Answer has consistently been one of the best AI tools I use. The search is good, so their agent is getting the best context for the summaries. Makes sense.
    • PeterStuer 1y ago
      It is building a system that amplifies the strengths of the LLM by feeding it the right knowledge in the right format at inference time. Context design is both a search (as a generic term for everything retrieval) and a representation problem. Just dumping raw reams of text into the 'prompt' isn't the best way to great results. Now I am fully aware that anything I can do on my side of the API, the LLM provider can and eventually will do as well. After all, Search also evolved beyond 'pagerank' to thousands of specialized heuristic subsystems.
  • J_Shelby_J 1y ago
    “It’s just a chat bot Michael, how much can it cost?” A philosophy degree later… I ended up just generating a summary of each of our 1k docs, using the summaries for retrieval, running a filter to confirm the doc is relevant, and finally using the actual doc to generate an answers.
• deleted 1y ago
  [deleted]