3 ms·
I just noticed Q in the AWS docs and tried a few test questions, and was not impressed. It refused to answer or misunderstood some basic questions. Eventually I
by rurp 3y ago
I just noticed Q in the AWS docs and tried a few test questions, and was not impressed. It refused to answer or misunderstood some basic questions. Eventually I got it to answer how a few short SKs would be ordered in dynamo and the answer it gave was incorrect.
Technical documentation is probably one of the worst usecases for GenAI, I'm not sure why so many companies are rushing to add it.
- freshpots 3y ago"Technical documentation is probably one of the worst usecases for GenAI, I'm not sure why so many companies are rushing to add it." I am one of those people who think that it would help people summarize it, get better compliance with specs, etc. However, I am limited in my knowledge when it comes to GenAI. Why do you think it is one of the worst use cases?
- rurp 3y agoBecause being precisely correct is especially important in technical documentation. GenAI is great at producing loads of content that looks sort of right, but terrible at logical correctness and, to a lesser extent, brevity. In my original example I asked Q about sorting in DynamoDB. The answer it gave was categorically wrong! That's worse than useless, it's actively misleading. If it can't get a simple example correct I have no faith that it will be reliable for more comicated real world questions, but those mistakes will be harder to catch.
- lazystar 3y agoas another anecdote, Q advised me to update a security group with a rule to allow ingress traffic on specific ports from a Lambda function execution role. It's jumbling up documentation from two completely different security tools; it's only a matter of time before Q causes a prod incident somewhere.
- ChicagoBoy11 3y agoI mean, surely the answer has to be: "because it's a pain point and there's a market for it", no? In the sense that, I think your skepticism is (rightly) warranted, but only because of the outcome that you've seen so far has produced unsatisfying results. In a universe where this approach does yield consistently correct and succinct answers, having an AI read a large body of technical documentation and be able to serve you the exact answer that you need from it does seem like a solution with lots of takers!
- danielmarkbruce 3y ago> Technical documentation is probably one of the worst usecases for GenAI, I'm not sure why so many companies are rushing to add it. Why?
- chrisfowles 3y agoBecause it's unique to the use case, complex, and nuanced. It has to be exactly correct, and it has potentially serious ramifications if the answers are incorrect, potentially with legal ramifications that I'm keenly waiting to see the first cases of.
- acdha 3y agoI think docs are tempting because it’s a mountain of content, customers always ask about changes, and the senior managers tend not to respect documentation team and view them as pure cost.
- PeterisP 3y agoQuite the contrary, just as for many products already it was better to search stackoverflow than the technical documentation, already last year one of the (few) use cases where I found ChatGPT really useful was asking for examples for various basic things that kind of are decribed in the technical documentation (and also on stackoverlow, and on a bajillion blogs) but the usability of that documentation (and intentional choices of its design, what to include and what not) sucks so much that it's far, far more effective to just ask a chatbot than read the original docs.