7 ms·

Is the doc bot docs, or not?

189 points by tobr 1y ago

BossingAround 1y ago
It's probably docs... If it can hallucinate an answer, it's docs with probably the most infuriating UX one can imagine. I remember being taught that no docs is better (i.e. less frustrating to the user) than bad/incorrect docs.
- pmg101 1y ago
  "Documentation - or, as I like to call it, lies." After a certain number of years you learn that source code comments so often fall out of synch with the code itself that they're more of a liability than an asset.
  - taneq 1y ago
    “There’s lies, damn lies, and datasheets.” Although, “All datasheets are wrong. Some datasheets are useful.”
  - walthamstow 1y ago
    At my last place the docs were in the repo with the code, and if you didn't update the docs in the same PR as the code it wouldn't get approved. My current place? It's in Confluence, miles away from code and with no review mechanism.
domk 1y ago
Working with Shopify is an example of something where a good mental model of how it works under the hood is often required. This type of mistake, not realising that the tag is added by an app after an order is created and won't be available when sending the confirmation email, is an easy one to make, both for a human or an LLM just reading the docs. This is where AI that just reads the available docs is going to struggle, and won't replace actual experience with the platform.
bravesoul2 1y ago
Need a real CC to test. Right there makes me lose respect for shopify if true. Even stripe let's you test :)
- Bewelge 1y ago
  Not sure if I'm missing something but the way I'd always test orders is generate some 100% discount. You don't need any payment info then. I only ever needed a CC if I wanted to actually test something relating to payment. And on test stores you can mock a CC
  - bravesoul2 1y ago
    That's a good way too for most cases. Unless you need there to be an amount
PeterStuer 1y ago
Confused. I just tried it in the Shopify Assistant and got: There is no built-in Liquid property to directly detect Shopify Collective fulfillment in email notifications. You can use the Admin GraphQL API to programmatically detect fulfillment source. In Liquid, you must rely on tags, metafields, or custom properties that you set up yourself to mark Collective items. If you want to automate this, consider tagging products or orders associated with Shopify Collective, or using an app to set a metafield, and then check for that in your Liquid templates. What you can do in Liquid (email notifications): If Shopify exposes a tag, property, or metafield on the order or line item that marks it as a Shopify Collective item, you could check for that in Liquid. For example, if you tag orders or products with "Collective", you could use: {% if order.tags contains "Collective" %}  {% endif %} or for line items: {% for line_item in line_items %} {% if line_item.product.tags contains "Collective" %}  {% endif %} {% endfor %} In the author's 'wrong' vs 'seems to work' answer, the only difference is the tag on the line items vs, the order. The flow (template? as he refers to it as 'some other cryptic Shopify process' ) he uses in his tests does seem to add the 'Shopify Collective' tag to the line items, and potentially also to the order if the whole order is Shopify Collective fullfilled, but without further info we can only guess his setup. While using AI can always lead to non-perfect results, I feel the evidence presented here does not support the conclusion. P.S. Given the reference to 'cryptic Shopify processes', I wonder how far the author would get with 'just the docs'.
- redhale 1y ago
  I think you're making the author's point, though. If two users ask the bot the same question and get different answers, is the bot valuable? A dice roll that might be (or is even _probably_) correct is not what I want when going directly to the official docs.
  - PeterStuer 1y ago
    Not sure the author is giving the full account though, as his answer snippet was probably just a part of the same answer I got, framed and interpreted differently (the AI's are never this terse as to just whip out a few lines of code). Besides, it is not even incorrect in the way he states it is. It is fully dependent on how he added the tags in his flow, as the complete answer correctly stated. He speculates on some timing issue in some 'cryptic Shopify process' adding the tag at a later stage, but this is clearly wrong as his "working answer" (which is also in the Assistant reply) does rely on the tag having been added at the same point in the process. My pure and exaggerated on purpose speculation: He just blindly copied some flow template, then from the (same as I got?) Assistant's answer copy/pasted the first Liquid code box, tested on one order and found it not doing what he wanted, this suited his confirmation bias regarding AI, later tried pasting the second Liquid code box (or the same answer you will get from Gemini through Google Search) and found 'it worked' on his one test order, still blamed the Assistant for being 'wrong'.
deepdarkforest 1y ago
I mean that's the dirty secret of any RAG chatbot. The concept of "grounding" is arbitrary. It doesn't matter if you use embeddings, or use a tool that uses your usual search and gets the top items, like most web search tools or google's. Is still relies on the model to not hallucinate given this info, which is very hard since too much info -> model gets confused, but too little info -> model assumes the info might not be there so useless. The fine balance depends on the user's query, and all approaches like score cutoff for embeddings etc just don't generalize. This is the same exact problem in coding assistants when they hallucinate functions or cannot find the needed dependencies etc. There are better and more complex approaches that use multiple agents to summarize different smaller queries and then iteratively buildup etc, internally we and a lot of companies have them, but for external customer queries, way too expensive. You can't spend 30 cents on every query
Bewelge 1y ago
To be fair, for me at least, that weird chat bot only appears on https://help.shopify.com/ https://help.shopify.com/ while the technical documentation is on shopify.dev/. Everytime I land on help.shopify.com I get the feeling it's one of those "Doc pages for sales people". Like it's meant to show "We have great documentation and you can do all these things" but never actually explains how to do anything. I tried that bot a couple of months ago and it was utterly useless: question: When using discountRedeemCodeBulkAdd there's a limit to add 100 codes to a discount. Is this a limit on the API or on the discount? So can I add 100 codes to the same discount multiple times? answer: I wasn't able to find any results for that. Can you tell me a little bit more about what you're looking for? Telling it more did not help. To me that seemed like the bot didn't even have access to the technical documentation. Finding it hard to believe that any search engine can miss a word like discountRedeemCodeBulkAdd if it actually is in the dataset: https://shopify.dev/docs/api/admin-graphql/latest/mutations/discountredeemcodebulkadd https://shopify.dev/docs/api/admin-graphql/latest/mutations/... So it's a bit like asking sales people technical questions. edit: Okay, I should have tried that before commenting. They seem to have updated it. When I ask the same question now it answers correctly (weirdly in German) : Die Begrenzung von 100 Codes bei der Verwendung von discountRedeemCodeBulkAdd bezieht sich auf die Anzahl der Codes, die Sie in einem einzelnen API-Aufruf hinzufügen können, nicht auf die Gesamtanzahl der Codes, die einem Rabatt zugeordnet werden können. Ein Rabattcode kann bis zu 20.000.000 eindeutige Rabattcodes enthalten. Daher können Sie mehrfach jeweils 100 Codes zum selben Rabatt hinzufügen, bis Sie das Limit von 20.000.000 Codes erreicht haben. Beachten Sie, dass Drittanbieter-Apps oder benutzerdefinierte Lösungen dieses Limit nicht umgehen oder erhöhen können. ~= It's a limit on the API endpoint, you can add up to 20M to a single discount.
- delusional 1y ago
  > So it's a bit like asking sales people technical questions. Maybe that's the best anthropomorphic analogy of LLMs. Like good sales people completely disconnected from reality, but finely tuned to give you just the answer you want.
  - WJW 1y ago
    Well no, the problem was that the bot didn't give them the answer they wanted. It's more like "finely tuned to waffle around pretending to be knowledgeable, but lacking technical substance". Kind of like a bad salesperson, the best salespeople I've had the pleasure of knowing were not afraid to learn the technical background of their products.
anentropic 1y ago
I would guess these narrow docs bots probably perform worse than ChatGPT et al in 'search' mode
emil_sorensen 1y ago
Docs bots like these are deceptively hard to get right in production. Retrieval is super sensitive to how you chunk/parse documentation and how you end up structuring documentation in the first place (see frontpage post from a few weeks ago: https://news.ycombinator.com/item?id=44311217 https://news.ycombinator.com/item?id=44311217). You want grounded RAG systems like Shopify's here to rely strongly on the underlying documents, but also still sprinkle a bit of the magic of the latent LLM knowledge too. The only way to get that balance right is evals. Lots of them. It gets even harder when you are dealing with GraphQL schema like Shopify has since most models struggle with that syntax moreso than REST APIs. FYI I'm biased: Founder of kapa.ai here (we build docs AI assistants for +200 companies incl. Sentry, Grafana, Docker, the largest Apache projects etc).
- skrebbel 1y ago
  Why RAG at all? We concatenated all our docs and tutorials into a text file, piped it all into the AI right along with the question, and the answers are pretty great. Cost was, last I checked, roughly 50c per question. Probably scales linearly with how much docs you have. This feels expensive but compared to a human writing an answer it's peanuts. Plus (assuming the customer can choose to use the AI or a human), it's great customer experience because the answer is there that much faster. I feel like this is a no-brainer. Tbh with the context windows we have these days, I don't completely understand why RAG is a thing anymore for support tools.
  - Rygian 1y ago
    What you describe sounds like poor man's RAG. Or lazy man's. You're just doing the augmentation at each prompt.
  - cluckindan 1y ago
    With RAG the cost per question would be low single-digit pennies.
  - IceDane 1y ago
    Because llms still suck at actually using all that context at once. And surely you can see yourself that your solution doesn't scale. It's great that it works for your specific case but I'm sure you can come up with a scenario where it's just not feasible.
schnable 1y ago
Reminds me of when I asked Gemini how to do some stuff in Google Docs App Script, and it just hallucinated the capability and code to make it work. Turns out what I wanted to do isn't supported at all. I feel like we aren't properly using AI in products yet.
- hnlmorg 1y ago
  I’ve found LLMs (or at least everyone I’ve tried this on) will always assume the customer is correct and thus even if they’re flat out wrong, the LLM will make up some bullshit to confirm the costumer is still correct. It’s great when you’re looking to do creative stuff. But terrible when you’re looking to confirm the correctness of an approach or asking for support on something that you weren’t even aware of its nonexistence.
  - dworks 1y ago
    that's because its "answers" are actually "completions". cant escape that fact - LLMs will always "hallucinate".
- aDyslecticCrow 1y ago
  I asked about a nieche json library for c. It apparently wasn't in the training data so it just invented how it feels like a json library would work. Ive also had alot of issues with cmake that it just invents syntax and functions. Every new question has to be made in a new chat context to clear the context poisoning. Its the things that lack good docs i want to ask about. But that's where its most likley to fail.
  - dingnuts 1y ago
    I think users should get a refund on the tokens when this happens
    - Night_Thastus 1y ago
      That would turn a business model that is already questionable in terms of profitability to one that would never, ever be profitable. Just sayin.
simonw 1y ago
This is a great example of the kind of question I'd love to be able to ask these documentation bots but that I don't trust them to be able to get right (yet): > What’s the syntax, in Liquid, to detect whether an order in an email notification contains items that will be fulfilled through Shopify Collective? I suspect the best possible implementation of a documentation bot with respect to questions like this one would be an "agent" style bot that has the ability to spin up its own environment and actually test the code it's offering in the answer before confidently stating that it works. That's really hard to do - Robin in this case could only test the result by placing and then refunding an order! - but the effort involved in providing a simulated environment for the bot to try things out in might make the difference in terms of producing more reliable results.
- dworks 1y ago
  get a second agent to validate the return from the first agent. but it might get it wrong because reasons, so you need a third agent just to make sure. and then a fourth. and so on. this is obviously not a working direction.
  - simonw 1y ago
    That's why you give them the ability to actually execute the code in a sandbox. Then it's not AI checking AI, you're mixing something deterministic into the loop.
    - dworks 1y ago
      the return may still not reflect the sandbox reality.
    - kmoser 1y ago
      That may certainly increase the agent's ability to get it right, but there will always be cases where the code it generates mimics the correct response, i.e. produces the output asked for, without actually working as intended, as LLMs tend to want to please as much as be correct.
dworks 1y ago
We're going to see increasingly more of these, and it's going to cause a big scandal at one point, that pops the current AI bubble. It's really obvious that you can't use non-deterministic systems this way but companies are hellbent on doing it anyway. This is why I won't take a role to implement "AI" in an existing product.
- pinoy420 1y ago
  [dead]
- crystal_revenge 1y ago
  I don’t understand why people seem to be attacking the “non-determinism” of LLMs. First, I think most people are confusing “probabilistic” with “non-deterministic” which have very distinct meanings in CS/ML. Non-deterministic typically entails following multiple paths at once. Consider regex matching with NFAs or even the particular view of a list as a monad. The only case where LLMs are “non-deterministic” is when using sampling algorithms like beam search where multiple paths are considered simultaneously. But most LLM usage being discussed doesn’t involve beam search. But even if one assumes people mean “probabilistic”, that’s also an odd critique given how probabilistic software has pretty much eaten the world. Most of my career has been building reliable product using probabilistic models. Finally, there’s nothing inherently probabilistic or non-deterministic about LLM generation, these are properties of the sampler applied. I did quite a lot of LLM benchmarking in recent years and almost always used greedy sampling both for performance (doing things like GSM8K strong benefits from choosing the maximum likely path) and reproducibility. You can absolutely set up LLM tools that have perfectly reproducible results. LLMs have many issues but their probabilistic nature is not one of them.
  - TZubiri 1y ago
    It's not entirely unrelated, the fact that the system is non-deterministic means that it necessarily is probabilistic. A business can reduce temperature to 0 and choose a specific seed, and it's the correct approach in most cases, but still the answers might change! On the other hand, it's true that there is some probability that is independent of determinism, for example maybe changing the order of some words might yield different answers, this might be a deterministic machines, but there's millions of ways to frame a question, if the answer depends on trivial details of the question formatting, there's a randomness there. Similar to how there is randomness in who will win a chess match between two equally rated players, despite the game being deterministic.
trjordan 1y ago
The core argument here is: LLM docbots are wrong sometimes. Docs are not. That's not acceptable. But that's not true! Docs are sometimes wrong, and even more so if you could errors of omission. From a users perspective, dense / poorly structured docs are wrong, because they lead users to think the docs don't have the answer. If they're confusing enough, they may even mislead users. There's always an error rate. DocBots are almost certainly wrong more frequently, but they're also almost certainly much much faster than reading the docs. Given that the standard recommendation is to test your code before jamming it in production, that seems like a reasonable tradeoff. YMMV! (One level down: the feedback loop for getting docbots corrected is _far_ worse. You can complain to support that the docs are wrong, and most orgs will at least try to fix it. We, as an industry, are not fully confident in how to fix a wrong LLM response reliably in the same way.)
- mananaysiempre 1y ago
  Docs are reliably fixable, so with enough effort they will converge to correctness. Doc bots are not and will not.
- apnorton 1y ago
  > There's always an error rate. DocBots are almost certainly wrong more frequently, but they're also almost certainly much much faster than reading the docs. A lot of the discourse around LLM tooling right now boils down to "it's ok to be a bit wrong if you're wrong quickly" ... and then what follows is an ever-further bounds-pushing on how big "a bit" can be. The promise of AI is "human-level (or greater)" --- we should only be using AI when it's as accurate (or more accurate) as human-generated docs, but the tech simply isn't there yet.
ngriffiths 1y ago
The doc bot goes in the same category as asking a human who has read the docs. In order of helpfulness you could get: - "Oh yeah just write this," except the person is not an expert and it's either wrong or not idiomatic - An answer that is reliably correct enough of the time - An answer in the form "read this page" or quotes the docs The last one is so much better because it directly solves the problem, which is fundamentally a search problem. And it places the responsibility for accuracy where it belongs (on the written docs).
- bee_rider 1y ago
  I think the name, doc-bot, is just bad (actually I don’t know what Shopify even calls their thing, so maybe the confusion is on the part of the author of the post, and not some misleading thing from Shopify). A bot like that could fulfill the role of the community forum, which certainly isn’t nothing! But of course it isn’t the documentation.
schaum 1y ago
There is also https://gurubase.io/ https://gurubase.io/ Which is sometimes used as a kind of talk with the documentation, it claims to validate the response somehow
shlomo_z 1y ago
> so I did my customary dance of order-refund, order-refund, order-refund. My credit card is going to get locked one of these days. I don't know the first thing about Shopify, but perhaps you can create a free "test" item so you don't actually need to make a credit card transaction.
- dpifke 1y ago
  You elided the part where TFA claims you can't test "unconventional email formats" via test orders. The full quote is: Shopify doesn’t provide a way to test unconventional email formats without actually placing real orders, so I did my customary dance of order-refund, order-refund, order-refund. My credit card is going to get locked one of these days. The person who wrote the above knows a lot about Shopify, so if you're going to contradict them, it'd be nice to point to some evidence as to why you think they're wrong.
  - Groxx 1y ago
    Anyone who has used these kinds of systems will have encountered tons of cases where those test systems weren't enough, so they use real purchases too The test systems are broadly good and worth using, but no. Everyone uses real purchases too.
TZubiri 1y ago
nots
nickphx 1y ago
Placing live orders on your card is a violation of Shopify, Shopify merchant, and card holder terms..
goroutines 1y ago
sounds like a good time to plug install.md (precise step-by-step docs / guides as MCP, with simple RAG) - which I think is the right direction when paired with coding agents.
ysofunny 1y ago
it's lossy docs. docs with JPEG artifacts, the more you zoom, the more specific your query, the worse the noise becomes
cco 1y ago
At CURRENT_CO we're going through another evaluation of several LLM bots you can add into your docs or Slack etc. We've done three trials since 2023 and each time we've found them not good enough to put in front of our customers. Usually the distribution has been about 60% good answers, 20% neutral to bad, 20% actively harmful that wastes the user's time. Really hoping we'll see better results this time but so far nothing has beat the recommendation to add our docs to your local LLM IDE of choice (Cursor etc) and then ask it questions with your own codebase as context.
deleted 1y ago
[deleted]
jasonm23 1y ago
If you're using an LLM and you are surprised it makes things up, I don't know who to blame? The overblown claims? The systems prompt "team" that ensure "I don't know" can never be uttered? Or user expectations? I find it unfair to blame users, but I tend to think creating a system prompt that accentuates "with an air of certainty" over ... straightforward honesty, both telling of the org culture, and the wider trend of "look like you know"...

189 points by tobr 1y ago

It's probably docs... If it can hallucinate an answer, it's docs with probably the most infuriating UX one can imagine. I remember being taught that no docs is better (i.e. less frustrating to the user) than bad/incorrect docs.

pmg101 1y ago
"Documentation - or, as I like to call it, lies." After a certain number of years you learn that source code comments so often fall out of synch with the code itself that they're more of a liability than an asset.
- taneq 1y ago
  “There’s lies, damn lies, and datasheets.” Although, “All datasheets are wrong. Some datasheets are useful.”
- walthamstow 1y ago
  At my last place the docs were in the repo with the code, and if you didn't update the docs in the same PR as the code it wouldn't get approved. My current place? It's in Confluence, miles away from code and with no review mechanism.

domk 1y ago

Working with Shopify is an example of something where a good mental model of how it works under the hood is often required. This type of mistake, not realising that the tag is added by an app after an order is created and won't be available when sending the confirmation email, is an easy one to make, both for a human or an LLM just reading the docs. This is where AI that just reads the available docs is going to struggle, and won't replace actual experience with the platform.

bravesoul2 1y ago

Need a real CC to test. Right there makes me lose respect for shopify if true. Even stripe let's you test :)

Bewelge 1y ago
Not sure if I'm missing something but the way I'd always test orders is generate some 100% discount. You don't need any payment info then. I only ever needed a CC if I wanted to actually test something relating to payment. And on test stores you can mock a CC
- bravesoul2 1y ago
  That's a good way too for most cases. Unless you need there to be an amount

PeterStuer 1y ago

Confused. I just tried it in the Shopify Assistant and got: There is no built-in Liquid property to directly detect Shopify Collective fulfillment in email notifications. You can use the Admin GraphQL API to programmatically detect fulfillment source. In Liquid, you must rely on tags, metafields, or custom properties that you set up yourself to mark Collective items. If you want to automate this, consider tagging products or orders associated with Shopify Collective, or using an app to set a metafield, and then check for that in your Liquid templates. What you can do in Liquid (email notifications): If Shopify exposes a tag, property, or metafield on the order or line item that marks it as a Shopify Collective item, you could check for that in Liquid. For example, if you tag orders or products with "Collective", you could use: {% if order.tags contains "Collective" %}  {% endif %} or for line items: {% for line_item in line_items %} {% if line_item.product.tags contains "Collective" %}  {% endif %} {% endfor %} In the author's 'wrong' vs 'seems to work' answer, the only difference is the tag on the line items vs, the order. The flow (template? as he refers to it as 'some other cryptic Shopify process' ) he uses in his tests does seem to add the 'Shopify Collective' tag to the line items, and potentially also to the order if the whole order is Shopify Collective fullfilled, but without further info we can only guess his setup. While using AI can always lead to non-perfect results, I feel the evidence presented here does not support the conclusion. P.S. Given the reference to 'cryptic Shopify processes', I wonder how far the author would get with 'just the docs'.

redhale 1y ago
I think you're making the author's point, though. If two users ask the bot the same question and get different answers, is the bot valuable? A dice roll that might be (or is even _probably_) correct is not what I want when going directly to the official docs.
- PeterStuer 1y ago
  Not sure the author is giving the full account though, as his answer snippet was probably just a part of the same answer I got, framed and interpreted differently (the AI's are never this terse as to just whip out a few lines of code). Besides, it is not even incorrect in the way he states it is. It is fully dependent on how he added the tags in his flow, as the complete answer correctly stated. He speculates on some timing issue in some 'cryptic Shopify process' adding the tag at a later stage, but this is clearly wrong as his "working answer" (which is also in the Assistant reply) does rely on the tag having been added at the same point in the process. My pure and exaggerated on purpose speculation: He just blindly copied some flow template, then from the (same as I got?) Assistant's answer copy/pasted the first Liquid code box, tested on one order and found it not doing what he wanted, this suited his confirmation bias regarding AI, later tried pasting the second Liquid code box (or the same answer you will get from Gemini through Google Search) and found 'it worked' on his one test order, still blamed the Assistant for being 'wrong'.

deepdarkforest 1y ago

I mean that's the dirty secret of any RAG chatbot. The concept of "grounding" is arbitrary. It doesn't matter if you use embeddings, or use a tool that uses your usual search and gets the top items, like most web search tools or google's. Is still relies on the model to not hallucinate given this info, which is very hard since too much info -> model gets confused, but too little info -> model assumes the info might not be there so useless. The fine balance depends on the user's query, and all approaches like score cutoff for embeddings etc just don't generalize. This is the same exact problem in coding assistants when they hallucinate functions or cannot find the needed dependencies etc. There are better and more complex approaches that use multiple agents to summarize different smaller queries and then iteratively buildup etc, internally we and a lot of companies have them, but for external customer queries, way too expensive. You can't spend 30 cents on every query

Bewelge 1y ago

To be fair, for me at least, that weird chat bot only appears on https://help.shopify.com/ https://help.shopify.com/ while the technical documentation is on shopify.dev/. Everytime I land on help.shopify.com I get the feeling it's one of those "Doc pages for sales people". Like it's meant to show "We have great documentation and you can do all these things" but never actually explains how to do anything. I tried that bot a couple of months ago and it was utterly useless: question: When using discountRedeemCodeBulkAdd there's a limit to add 100 codes to a discount. Is this a limit on the API or on the discount? So can I add 100 codes to the same discount multiple times? answer: I wasn't able to find any results for that. Can you tell me a little bit more about what you're looking for? Telling it more did not help. To me that seemed like the bot didn't even have access to the technical documentation. Finding it hard to believe that any search engine can miss a word like discountRedeemCodeBulkAdd if it actually is in the dataset: https://shopify.dev/docs/api/admin-graphql/latest/mutations/discountredeemcodebulkadd https://shopify.dev/docs/api/admin-graphql/latest/mutations/... So it's a bit like asking sales people technical questions. edit: Okay, I should have tried that before commenting. They seem to have updated it. When I ask the same question now it answers correctly (weirdly in German) : Die Begrenzung von 100 Codes bei der Verwendung von discountRedeemCodeBulkAdd bezieht sich auf die Anzahl der Codes, die Sie in einem einzelnen API-Aufruf hinzufügen können, nicht auf die Gesamtanzahl der Codes, die einem Rabatt zugeordnet werden können. Ein Rabattcode kann bis zu 20.000.000 eindeutige Rabattcodes enthalten. Daher können Sie mehrfach jeweils 100 Codes zum selben Rabatt hinzufügen, bis Sie das Limit von 20.000.000 Codes erreicht haben. Beachten Sie, dass Drittanbieter-Apps oder benutzerdefinierte Lösungen dieses Limit nicht umgehen oder erhöhen können. ~= It's a limit on the API endpoint, you can add up to 20M to a single discount.

delusional 1y ago
> So it's a bit like asking sales people technical questions. Maybe that's the best anthropomorphic analogy of LLMs. Like good sales people completely disconnected from reality, but finely tuned to give you just the answer you want.
- WJW 1y ago
  Well no, the problem was that the bot didn't give them the answer they wanted. It's more like "finely tuned to waffle around pretending to be knowledgeable, but lacking technical substance". Kind of like a bad salesperson, the best salespeople I've had the pleasure of knowing were not afraid to learn the technical background of their products.

anentropic 1y ago

I would guess these narrow docs bots probably perform worse than ChatGPT et al in 'search' mode

emil_sorensen 1y ago

Docs bots like these are deceptively hard to get right in production. Retrieval is super sensitive to how you chunk/parse documentation and how you end up structuring documentation in the first place (see frontpage post from a few weeks ago: https://news.ycombinator.com/item?id=44311217 https://news.ycombinator.com/item?id=44311217). You want grounded RAG systems like Shopify's here to rely strongly on the underlying documents, but also still sprinkle a bit of the magic of the latent LLM knowledge too. The only way to get that balance right is evals. Lots of them. It gets even harder when you are dealing with GraphQL schema like Shopify has since most models struggle with that syntax moreso than REST APIs. FYI I'm biased: Founder of kapa.ai here (we build docs AI assistants for +200 companies incl. Sentry, Grafana, Docker, the largest Apache projects etc).

skrebbel 1y ago
Why RAG at all? We concatenated all our docs and tutorials into a text file, piped it all into the AI right along with the question, and the answers are pretty great. Cost was, last I checked, roughly 50c per question. Probably scales linearly with how much docs you have. This feels expensive but compared to a human writing an answer it's peanuts. Plus (assuming the customer can choose to use the AI or a human), it's great customer experience because the answer is there that much faster. I feel like this is a no-brainer. Tbh with the context windows we have these days, I don't completely understand why RAG is a thing anymore for support tools.
- Rygian 1y ago
  What you describe sounds like poor man's RAG. Or lazy man's. You're just doing the augmentation at each prompt.
- cluckindan 1y ago
  With RAG the cost per question would be low single-digit pennies.
- IceDane 1y ago
  Because llms still suck at actually using all that context at once. And surely you can see yourself that your solution doesn't scale. It's great that it works for your specific case but I'm sure you can come up with a scenario where it's just not feasible.

schnable 1y ago

Reminds me of when I asked Gemini how to do some stuff in Google Docs App Script, and it just hallucinated the capability and code to make it work. Turns out what I wanted to do isn't supported at all. I feel like we aren't properly using AI in products yet.

hnlmorg 1y ago
I’ve found LLMs (or at least everyone I’ve tried this on) will always assume the customer is correct and thus even if they’re flat out wrong, the LLM will make up some bullshit to confirm the costumer is still correct. It’s great when you’re looking to do creative stuff. But terrible when you’re looking to confirm the correctness of an approach or asking for support on something that you weren’t even aware of its nonexistence.
- dworks 1y ago
  that's because its "answers" are actually "completions". cant escape that fact - LLMs will always "hallucinate".
aDyslecticCrow 1y ago
I asked about a nieche json library for c. It apparently wasn't in the training data so it just invented how it feels like a json library would work. Ive also had alot of issues with cmake that it just invents syntax and functions. Every new question has to be made in a new chat context to clear the context poisoning. Its the things that lack good docs i want to ask about. But that's where its most likley to fail.
- dingnuts 1y ago
  I think users should get a refund on the tokens when this happens
  - Night_Thastus 1y ago
    That would turn a business model that is already questionable in terms of profitability to one that would never, ever be profitable. Just sayin.

simonw 1y ago

This is a great example of the kind of question I'd love to be able to ask these documentation bots but that I don't trust them to be able to get right (yet): > What’s the syntax, in Liquid, to detect whether an order in an email notification contains items that will be fulfilled through Shopify Collective? I suspect the best possible implementation of a documentation bot with respect to questions like this one would be an "agent" style bot that has the ability to spin up its own environment and actually test the code it's offering in the answer before confidently stating that it works. That's really hard to do - Robin in this case could only test the result by placing and then refunding an order! - but the effort involved in providing a simulated environment for the bot to try things out in might make the difference in terms of producing more reliable results.

dworks 1y ago
get a second agent to validate the return from the first agent. but it might get it wrong because reasons, so you need a third agent just to make sure. and then a fourth. and so on. this is obviously not a working direction.
- simonw 1y ago
  That's why you give them the ability to actually execute the code in a sandbox. Then it's not AI checking AI, you're mixing something deterministic into the loop.
  - dworks 1y ago
    the return may still not reflect the sandbox reality.
  - kmoser 1y ago
    That may certainly increase the agent's ability to get it right, but there will always be cases where the code it generates mimics the correct response, i.e. produces the output asked for, without actually working as intended, as LLMs tend to want to please as much as be correct.

dworks 1y ago

We're going to see increasingly more of these, and it's going to cause a big scandal at one point, that pops the current AI bubble. It's really obvious that you can't use non-deterministic systems this way but companies are hellbent on doing it anyway. This is why I won't take a role to implement "AI" in an existing product.

pinoy420 1y ago
[dead]
crystal_revenge 1y ago
I don’t understand why people seem to be attacking the “non-determinism” of LLMs. First, I think most people are confusing “probabilistic” with “non-deterministic” which have very distinct meanings in CS/ML. Non-deterministic typically entails following multiple paths at once. Consider regex matching with NFAs or even the particular view of a list as a monad. The only case where LLMs are “non-deterministic” is when using sampling algorithms like beam search where multiple paths are considered simultaneously. But most LLM usage being discussed doesn’t involve beam search. But even if one assumes people mean “probabilistic”, that’s also an odd critique given how probabilistic software has pretty much eaten the world. Most of my career has been building reliable product using probabilistic models. Finally, there’s nothing inherently probabilistic or non-deterministic about LLM generation, these are properties of the sampler applied. I did quite a lot of LLM benchmarking in recent years and almost always used greedy sampling both for performance (doing things like GSM8K strong benefits from choosing the maximum likely path) and reproducibility. You can absolutely set up LLM tools that have perfectly reproducible results. LLMs have many issues but their probabilistic nature is not one of them.
- TZubiri 1y ago
  It's not entirely unrelated, the fact that the system is non-deterministic means that it necessarily is probabilistic. A business can reduce temperature to 0 and choose a specific seed, and it's the correct approach in most cases, but still the answers might change! On the other hand, it's true that there is some probability that is independent of determinism, for example maybe changing the order of some words might yield different answers, this might be a deterministic machines, but there's millions of ways to frame a question, if the answer depends on trivial details of the question formatting, there's a randomness there. Similar to how there is randomness in who will win a chess match between two equally rated players, despite the game being deterministic.

trjordan 1y ago

The core argument here is: LLM docbots are wrong sometimes. Docs are not. That's not acceptable. But that's not true! Docs are sometimes wrong, and even more so if you could errors of omission. From a users perspective, dense / poorly structured docs are wrong, because they lead users to think the docs don't have the answer. If they're confusing enough, they may even mislead users. There's always an error rate. DocBots are almost certainly wrong more frequently, but they're also almost certainly much much faster than reading the docs. Given that the standard recommendation is to test your code before jamming it in production, that seems like a reasonable tradeoff. YMMV! (One level down: the feedback loop for getting docbots corrected is _far_ worse. You can complain to support that the docs are wrong, and most orgs will at least try to fix it. We, as an industry, are not fully confident in how to fix a wrong LLM response reliably in the same way.)

mananaysiempre 1y ago
Docs are reliably fixable, so with enough effort they will converge to correctness. Doc bots are not and will not.
apnorton 1y ago
> There's always an error rate. DocBots are almost certainly wrong more frequently, but they're also almost certainly much much faster than reading the docs. A lot of the discourse around LLM tooling right now boils down to "it's ok to be a bit wrong if you're wrong quickly" ... and then what follows is an ever-further bounds-pushing on how big "a bit" can be. The promise of AI is "human-level (or greater)" --- we should only be using AI when it's as accurate (or more accurate) as human-generated docs, but the tech simply isn't there yet.

ngriffiths 1y ago

The doc bot goes in the same category as asking a human who has read the docs. In order of helpfulness you could get: - "Oh yeah just write this," except the person is not an expert and it's either wrong or not idiomatic - An answer that is reliably correct enough of the time - An answer in the form "read this page" or quotes the docs The last one is so much better because it directly solves the problem, which is fundamentally a search problem. And it places the responsibility for accuracy where it belongs (on the written docs).

bee_rider 1y ago
I think the name, doc-bot, is just bad (actually I don’t know what Shopify even calls their thing, so maybe the confusion is on the part of the author of the post, and not some misleading thing from Shopify). A bot like that could fulfill the role of the community forum, which certainly isn’t nothing! But of course it isn’t the documentation.

schaum 1y ago

There is also https://gurubase.io/ https://gurubase.io/ Which is sometimes used as a kind of talk with the documentation, it claims to validate the response somehow

shlomo_z 1y ago

> so I did my customary dance of order-refund, order-refund, order-refund. My credit card is going to get locked one of these days. I don't know the first thing about Shopify, but perhaps you can create a free "test" item so you don't actually need to make a credit card transaction.

dpifke 1y ago
You elided the part where TFA claims you can't test "unconventional email formats" via test orders. The full quote is: Shopify doesn’t provide a way to test unconventional email formats without actually placing real orders, so I did my customary dance of order-refund, order-refund, order-refund. My credit card is going to get locked one of these days. The person who wrote the above knows a lot about Shopify, so if you're going to contradict them, it'd be nice to point to some evidence as to why you think they're wrong.
- Groxx 1y ago
  Anyone who has used these kinds of systems will have encountered tons of cases where those test systems weren't enough, so they use real purchases too The test systems are broadly good and worth using, but no. Everyone uses real purchases too.

TZubiri 1y ago

nots

nickphx 1y ago

Placing live orders on your card is a violation of Shopify, Shopify merchant, and card holder terms..

goroutines 1y ago

sounds like a good time to plug install.md (precise step-by-step docs / guides as MCP, with simple RAG) - which I think is the right direction when paired with coding agents.

ysofunny 1y ago

it's lossy docs. docs with JPEG artifacts, the more you zoom, the more specific your query, the worse the noise becomes

cco 1y ago

At CURRENT_CO we're going through another evaluation of several LLM bots you can add into your docs or Slack etc. We've done three trials since 2023 and each time we've found them not good enough to put in front of our customers. Usually the distribution has been about 60% good answers, 20% neutral to bad, 20% actively harmful that wastes the user's time. Really hoping we'll see better results this time but so far nothing has beat the recommendation to add our docs to your local LLM IDE of choice (Cursor etc) and then ask it questions with your own codebase as context.

deleted 1y ago

[deleted]

jasonm23 1y ago

If you're using an LLM and you are surprised it makes things up, I don't know who to blame? The overblown claims? The systems prompt "team" that ensure "I don't know" can never be uttered? Or user expectations? I find it unfair to blame users, but I tend to think creating a system prompt that accentuates "with an air of certainty" over ... straightforward honesty, both telling of the org culture, and the wider trend of "look like you know"...