7 ms·
You just have to double check the results whenever you tell Claude to extract lists and data. 99.9% of it will be correct, but sometimes 1 or 2 records are off
by wenc 2y ago
You just have to double check the results whenever you tell Claude to extract lists and data.
99.9% of it will be correct, but sometimes 1 or 2 records are off. This kind of error is especially hard to notice because you're so impressed that Claude managed to do the extract task at all -- plus the results look wholly plausible upon eyeballing -- that you wouldn't expect anything to be wrong at all.
But LLMs can get things ever slightly wrong when it comes to long lists/tables. I've been bitten by this before.
Trust but verify.
(edit: if the answer is "machine verifiable", one approach to ask an LLM to write a Python validator which it can execute internally. ChatGPT can execute code. I believe Sonnet 3.5 can too, but I haven't tried.)
- thelittleone 2y agoCan solve that to have a similar or lower error probability as a human by running the results through a verification agent.
- jph00 2y agoSonnet on Claude.ai can not execute code, although it often pretends otherwise.
- disillusioned 2y agoThe "off by one" predilection of LLMs is going to lead to this massive erosion of trust in whatever "Truth" is supposed to be, and it's terrifying and going to make for a bumpy couple of years. (Or the complete collapse of objective knowledge, on a long enough time horizon.) It's one thing to ask an LLM when George Washington was born, and have it return "May 20, 2020." It's another thing to ask it, and have it matter-of-factly hallucinate "February 20, 1733." At first glance, that... sounds right, right? President's Day is in February, and has something to do with his birthday? And that year seems to check out? Good enough! But it's not right. And it's the confidence and bravado with which LLMs report these "facts" that's terrifying. It just misstates information, calculations, and detail work, because the stochastic model compelled it to, and there wasn't sufficient checks in place to confirm or validate the information. Trust but verify is one of those things that's so paradoxical and cyclical: if I have to confirm every fact ChatGPT gives me with... what I hope is a higher source of truth like Wikipedia, before it's overrun with LLM outputs... then why don't I just start there? If I have to build a validator in Python to verify the output then... why not just start there? We're going to see some major issues crop up from this sort of insidious error, but the hard part about off-by-ones is that they're remarkably difficult to detect, and so what will happen is data will slowly corrupt and take us further and further off course, and we won't notice until it's too late. We should be so lucky that all of LLMs' garbage outputs look like glue on pizza recommendations, but the reality is, it'll be a slow, seeping poisoning of the well, and when this inaccurate output starts sneaking into parts of our lives that really matter... we're probably well and truly fucked.
- stavros 2y agoThis is semi-offtopic, but "trust but verify" is an oxymoron. Trusting something means I don't have to verify whether it's correct (I trust that it is), so the saying, in the end, is "don't verify but verify".
- cromka 2y agoA better phrase would be “Use it but verify”, simply.
- stavros 2y agoYes, which boils down to "verify".
- EGreg 2y agoI always wondered about that
- bilal9684 2y ago[flagged]
- cj 2y agoIt's possible to trust (or have faith) in my car being able to drive another 50k miles without breaking down. But if I bring it to a mechanic to have the car inspected just in case, does that mean I never had trust/faith in the car to begin with? "I trust my coworkers write good code, but I verify with code reviews" -- doing code reviews doesn't mean you don't trust your coworker. Yet another way to look at it: people can say things they believe to be true but are actually false (which isn't lying). When that happens, you can successfully trust someone in the sense that they're not lying to you, but the absence of a lie doesn't guarantee a truth, so verifying what you trust to be true doesn't invalidate your trust.
- 2y ago
- nabla9 2y agoThey are just human.
- IshKebab 2y agoThey aren't the same as humans. They definitely work differently. Also they've been trained to say something as plausible as possible. If it happens to be true, then that's great because it's extra plausible. If it's not true, no big deal. While I have worked with one awful human in the past who was like that, most thankfully aren't!
- mmahemoff 2y agoYes, probably better to get the LLM to write the script. Example, I was trying out two podcast apps and wanted to get a diff of the feeds I had subscribed to. I initially asked the LLM to compare the two OPML files but it got the results wrong. I could have spent the next 30 minutes prompt engineering and manually verifying results, but instead I asked it to write a script to compare two LLMs, which turned out fine. It's fairly easy to inspect a script and be confident it's _probably_ accurate compared to the tedious process of checking a complex output.
- sorokod 2y agoIsn't asking an LLM "to write a Python validator" suffers from the 99.9% (or whatever the error rate for validators written by Claude) problem?
- bonzini 2y agoThe difference is that you're asking it to perform one intellectual task (write a program) instead of 100 menial tasks (parse a file). To the LLM the two are the same level of complexity, so performing less work means less possibility of error. Also, the LLM is more likely to fail spectacularly by hallucinating APIs when writing a script, and more likely to fail subtly on parsing tasks.
- dbaupp 2y agoIn addition to what you say, it can also be easier for a (appropriately-skilled) human to verify a small program than to verify voluminous parsing output, plus, as you say, there's the semi-automated "verification" of a very-wrong program failing to execute.
- jessekv 2y agoAll tests have this problem. We still write them for the same reasons we do double-entry bookkeeping.
- Foobar8568 2y agoWell for my case, I have 0 trust in chatgpt ( or local LLM) to extract properly data from a PDF file, especially if it's over a few pages.
- dan-robertson 2y agoIf it’s correct 99.9% of the time, and the piano lessons are every two weeks, that’s one error in piano lesson scheduling over 40 years. That sounds good enough to me to not verify.
- MattGaiser 2y agoHave two LLMs do the task and compare.
- campers 2y agoOr three or four or five! https://openreview.net/pdf?id=zj7YuTE4t8 https://openreview.net/pdf?id=zj7YuTE4t8
- surfingdino 2y agoGoogle's Gemini dev docs contain a few of such warnings. For a good reason, these models make stuff up and the domain where ROI is positive is small.
- websap 2y agoIt would be interesting to see how well does a human do it? Are they correct more than 99.9% of the time?
- huijzer 2y agoI completely agree. If correctness matters, then it’s probably better to use LLMs to write the code than to let LLMs be the code.
- agumonkey 2y agoCan Claude tdd itself ? Lean-Claude
- wanderingmind 2y agoKind of a noob question, is it possible to design a GAN type network with LLM, where one (or many) LLMs generate outputs, while a few other LLMs validate or discriminate them and thus improving generator LLMs accuracy.
- oneshtein 2y agoYes, you can use AI to spot errors in AI output. Done before with good results, but it requires to run 2 different, but equally good, AI models in parallel, which is way more expensive than 1 model.
- ada1981 2y ago"which is way more expensive than 1 model." In this case "way more" means exactly 2x the cost.
- onlyrealcuzzo 2y ago> 99.9% of it will be correct, but sometimes 1 or 2 records are off. This kind of error is especially hard to notice because you're so impressed that Claude managed to do the extract task at all -- plus the results look wholly plausible upon eyeballing -- that you wouldn't expect anything to be wrong at all. That's far better than I would do on my own. I doubt I'd even be 99% accurate. If it's really 99.9% accurate for something like this - I'd gladly take it.
- croes 2y agoThe problem is that people could be impressed and use it for things where 0.01% could lead to people getting hurt or even get killed.
- onlyrealcuzzo 2y agoYeah, but that's probably not your own calendar reminders...
- taberiand 2y agoI call that problem "Doctor, doctor! It hurts when I do this!" If the risk exists with AI processing this kind of data, it exists with a human processing the data. The fail-safe processes in place for the human output need to be used for the AI output too, obviously - using the AI speeds up the initial process enormously though
- croes 2y agoThe problem is liability. Who is liable if the AI makes errors?
- deleted 2y ago[deleted]
- thegabriele 2y agoIt depends on the richness of the AI
- troupo 2y ago> You just have to double check the results whenever you tell Claude to extract lists and data. There's also the problem that they are tuned to be overly helpful. I tried a similar thing described in the OG article with some non-English data. I could not stop Claude from "helpfully" translating chunks of data into English. "include description from the image" would cause it to translate it, and "include description from the image, do not translate or summarize it" would cause it to just skip it.
- bhl 2y agoI wonder if tool-calling to output schema'd json would have a low error rate here. For each field, you could have a description of what is approximately right, and that should anchor the output better than a one-off prompt.
- stocknoob 2y agoHave N separate chats, and take the result that they all agree on. N=1 is like RAID0.
- sussmannbaka 2y ago> Trust but verify. This is why I don’t bother with GenAI. It’s a lot harder to verify data from a machine trained to generate plausible looking data (that happens to be correct most of the time, sure) than from real data. Ofc this is getting harder as “real data” gets drowned in GenAI output, which drives the animosity against GenAI in my circle.
- damnever 2y agoVerification comes with an additional cost. This makes me wonder: if we can verify, why use a LLM at all? An LLM might even cost more.