4 ms·
Models hallucinate plausible answers to why they did things. It might be true and it might be complete fiction.
by tarxzvf 23d ago
Models hallucinate plausible answers to why they did things. It might be true and it might be complete fiction.
- taeric 23d agoI'm growing increasingly confident that this is how people often work, as well.
- skinfaxi 23d agoPeople don't make rational decisions that make rationalized decisions. Is there any thought to pulling your hand off a hot surface?
- autoexec 23d agoPeople do both. Some choices aren't worth the time and effort of detailed analysis and contemplation and some are basically instinctual, but there are plenty of times that choices are carefully considered and well reasoned before being made and acted on.
- nemomarx 23d agoI kind of want my computer systems to be more reliable and predictable than paying an intern to manage something and asking why they messed up
- jaggederest 23d agoAt this point it very dramatically is more reliable and predictable than any human I've worked with. Do you know anyone who actually reads and adheres closely to all of the documentation every time it's changed?
- bluefirebrand 23d agoI don't know anyone who has that kind of time, no
- wccrawford 23d agoThat was my experience with Claude when my vibe-coded project was small. But now that I've been working on it a month and there's a lot of documentation, it's pretty clearly ignoring parts of the documentation and parts of the code. It will come up with some ridiculous statement about how something works, and I'll challenge it, and it'll admit I'm right. It definitely reads more documentation than any programmer I've ever worked with (myself included) but because it doesn't have a memory other than the documentation, it still makes mistakes like that. I haven't turned on "memory" or tried it with Codex, so I don't know how that'll change soon, though.
- jaggederest 23d agoYeah the biggest task these days that I do manually is curating the documentation. AGENTS.md in every major directory, and a variety of reference docs that are explicitly referenced in those files. # See DOC-ITEM-NAME DOC-ITEM-NAME.md When referencing documents, always use the exact syntax See <TAG> - this is enforced by a lint on precommit And those doc items are basically all of the values, architectural, strategic, and tactical items. It's a poor man's in-repo RAG but it's shockingly effective, especially if you keep them small. I may migrate some/all of them to skills over time, but I usually update them biweekly, and I only allow agents to make small edits or propose new notes. And typically I go through and delete or curate any agent edits before merge. Depending on language I've seen this scale past multiple millions of lines of code, as long as you pair it with all of the linting and tooling that you can possibly build.
- voakbasda 23d agoEvery time I hear someone complain about hallucinations, I laugh at the total lack of self awareness about our species. Humans are just as bad (now, probably worse) at telling the truth, whether due to intention or poor memory.
- lathiat 23d ago“Post Hoc Rationalisation” https://www.patheos.com/blogs/tippling/2013/11/14/post-hoc-rationalisation-reasoning-our-intuition-and-changing-our-minds/ https://www.patheos.com/blogs/tippling/2013/11/14/post-hoc-r...
- BurningFrog 23d agoI learned this from "The Elephant in the Brain", which I strongly recommend: https://amzn.to/4iSyLX8 https://amzn.to/4iSyLX8
- jameshart 23d agoI learned this from Dirk Gently’s Holistic Detective Agency, which I strongly recommend.
- BurningFrog 22d agoI read that book and strongly recommend it too! Don't remember that part though. It's been a few decades.
- supern0va 23d agoExactly. I am becoming increasingly convinced that this is actually just a part of how intelligence/cognition works.
- astrobe_ 23d agoBut is it really what we want, machines with the same defects as humans? I don't want a pocket calculator that make mistakes "sometimes" so I have to double-check the results, I want a pocket calculator that works (to those who want to argue that pocket calculators don't give the correct result for (1/3)*3: STFU).
- CookieCrisp 23d agoNo, but it makes sense to me that we’d need to go through this step to get where we want to go
- bmacho 23d agoNo. People have an inner monologue, partial results and ideas and they remember that. If they've worked some minutes/hours/weeks on something and you ask them why did they do that, they will either answer honestly and truthfully, lie, or say "I missed that/didn't seem important so I just chose something at random". None of these cases are similar to how AI works.
- yawnxyz 23d agoI think this is true with some people, but I don't think this holds true for some (or even most) people across the US (at least not all the folks I've worked with)
- croo 23d agoActually split brain experiments tells a different story. The left hemisphere actively confabulates, inventing plausible explanations for actions it didn’t initiate, suggesting that much of human self-narrative may be post-hoc storytelling.
- satvikpendem 23d agoIndeed. People literally make stuff up when their corpus callosum is severed.
- bmacho 23d agoAand what if their corpus callosum isn't severed?
- satvikpendem 23d agoThey also make stuff up.
- deleted 23d ago[deleted]
- bmacho 23d ago
- yonatan8070 23d agoAFAIK this really is true. I've seen some videos about patients who had the connecting part between the left and right halves of the brain cut as a (archaic) treatment for epilepsy. While it did help the epilepsy, their brain was essentially two brains controlling two halves of the body. With one controlling speech. There were experiments where one eye was shown some instruction text, the corresponding hand performed that instruction, and when asked why they dix that action, the speaking half just made up some plausible, yet completely wrong reason, just like an LLM.
- deleted 23d ago[deleted]
- Terr_ 23d agoI see this observation frequently, and I dislike how it often has the unsound subtext of: "Therefore something is going well or at least not too badly." If you build a robot where a pressurized hose leaks causing fluid to destroy part of the circuitry, we don't praise it as progress towards the human ideal of having brain aneurysms. A similar failure-path is not a reliable indicator of a similar success-path.
- krupan 23d agoAll people at least some times, absolutely yes. Isn't it awesome we built machine that does the exact same thing, but even faster and more often? /s
- theluketaylor 23d agoTrue, but even a hallucinated explanation of where things went wrong added to the context can force the model down a better path over the next few inputs.
- embedding-shape 23d agoYou can also literally tell them: "Here is your session ID: $ID, lookup the .jsonl session, trace exactly why this decision was being made, present evidence and concrete proof, no guessing or assumptions" and you'll get an evidence-based report without guesses.
- zamadatix 23d agoIt can always hallucinate said report results/evidence/proof just the same. This approach tends to help reduce the hallucination rate though. You can extend this further by using an adversarial agent trying to find mistakes in the other instance's logs in a loop where a 3rd neutral agent weighs the claims of the other two. This is also just another step in reducing error, it does not guarantee elimination of such errors. The latter is an impossible guarantee, even for humans.
- bmacho 23d agoAsk it to build you a simple and deterministic citation checking extension to your IDE that puts source in meta to citations. E.g. color citations green/red depending if they are valid.
- zamadatix 23d agoSure, you can always validate what it's saying yourself at any point & you can have it try to make manual verification an easier process to complete via methods such as the above.
- Kiro 23d agoThe point of the parent post is that the explanation shows they made the error themselves, so it's immediately validated.
- jameshart 23d agoTo test this, change the history in the context to indicate that the model did or recommended something completely different than it actually did, and then ask it to explain why. You’ll still get a plausible explanation.
- JeremyNT 23d agoYes. But although they can't know "why" a specific "wrong" answer was selected, the response is often still informative, and it can highlight real weaknesses in process or code structure that should be addressed anyway.