6 ms·
Why do AI chatbots have such a hard time admitting 'I don't know'?
- jghn 2y agoBecause when you're just churning out the most likely set of next tokens, it's hard to understand "I don't know"
- AnimalMuppet 2y agoStronger: When you're just churning out the most likely set of next tokens, you don't know anything, in the sense that we normally think of "knowing". The only knowledge that it knows is encoded in the relationship between words. But maybe it could still "not know" in those terms. If there is no next token that is more likely than some threshold, then it "doesn't know" even in "most likely next token" terms.
- SAI_Peregrinus 2y agoThe most likely next tokens could be "I don't know". The output need not have any relation to thought or knowledge.
- booleandilemma 2y agoThey remind me of this really annoying coworker of mine who refuses to say "I don't know" to anything. When he doesn't know the answer to something he speculates and spouts off best practices. It's extremely frustrating to work with him. I'm particularly emotional as I write this comment because he just did it again!
- Ginguin 2y agoCounterbalance that with the (almost) equally annoying coworker who says "I don't know" as the immediate answer to any question! If there is any possibility of ambiguity or uncertainty, he will refuse to speculate or cite best practices. I think your coworker would probably be more annoying, but only because I can ask an additional question of my coworker to dig into the response. I wish he would say something like, "I don't know, but [best practice/past experience]" without me having to then ask for the last part.
- deleted 2y ago[deleted]
- Workaccount2 2y agoOr instead of saying "I don't know" they ramble on in order to drag the topic into a domain they do know, and then go on and on talking about that. It's incredibly infuriating.
- rsynnott 2y agoAre you sure he's a real person, and not a magic robot?
- do_not_redeem 2y agohttps://archive.ph/XKG86 https://archive.ph/XKG86 Because they were trained on internet arguments, and nobody on the internet has ever admitted they don't know something
- tsumnia 2y agoDon't forget we made fake Internet points to tell people we don't like when someone says "I don't know" as well
- Lalabadie 2y agoAnd human-refined training sets are also selected for the accuracy of their knowledge. Book authors mostly default to assuming they know. There's just comparatively not much reference material for someone correctly self-assessing that they don't know.
- josteink 2y ago> Because they were trained on internet arguments, and nobody on the internet has ever admitted they don't know something Or people’s questions were left unanswered (because nobody knew), and there was nothing for the LLM to learn there. So I guess LLMs can’t learn the absence of answers?
- bdhcuidbebe 2y agoOr, the ai is not thinking at all, just predicting. RTFA
- josteink 2y agoThat’s my point. In a unanswered question, there is nothing to predict (ie «learn» or «think» which I never said), so the AI/LLM will predict using a different source which has responded something, correctness or relevancy be damned. What’s with your tone btw? RTFA? To me that feels quite unwarranted and in violation of the site guidelines. Chill out, eh?
- dongecko 2y ago
- mcsniff 2y agoSo do most people.
- curiousObject 2y agoThe RLHF sessions should incentivize responses like “I don’t know” and “I’m not sure”. But they do not, because that will not attract investment.
- Workaccount2 2y agoA low/no hallucination model would probably be the closet thing to a literal money printer in tech right now.
- sharemywin 2y agothe thing is aren't hallucinations and creativity the two sides of the same coin?
- Workaccount2 2y agoRight, but that is a lever (temperature) that can be moved in real time, even with current models.
- tlb 2y agoThere'll be demand for both kinds of models, imaginative and dependable.
- sharemywin 2y agoI also see RAG playing a role in the dependable fact based side. the imaginative side might benefit from it too. think like thought joggers or some kinds of save prompts that could be iterated through to generate more robust ideas.
- drewcoo 2y agoTraining.
- ratedgene 2y agoBecause they don't inherently know anything to begin with.
- farleykr 2y agoI might be getting overly philosophical here but I'd say it's because they truly don't know anything at all (as opposed to knowing some things but not others). To be able to say "I don't know" you have to first "know" on a deeper level that there is a fundamental true or correct answer to a question and that you are disconnected from it.
- sharemywin 2y agoso our "knowledge" is based on perception. I know something happened because I saw it with my own 2 eyes. everything else is less "knowable"
- bdhcuidbebe 2y agoThat seems to be the conspiracists group think, yea. IRL we invented the field of science to avoid such make belief nonsense.
- yetihehe 2y agoInstead, we believe that those scientists did provide correct answers. Which is true about 99.9% of time, but not 100%. There is still believing involved, because we can't really check out everything for ourselves, there is no time in one person's life for this.
- zero_shift 2y agoThey are machines designed to produce a facsimile of knowledge. Or at least an approximation. If they refused to answer, that's a failure by the terms of what they product aims to do
- falcrist 2y agoI don't think it's overly philosophical to point out that these are large language models, not truth engines or AGI or knowledge directories. They're not using logic to reason their way to an answer. They're just predicting the next word that would sound like part of a human answer.
- user3939382 2y agoI have had luck asking for a list of possible answers with an accompanying score of confidence, which comes close.
- Workaccount2 2y agoIronically, the score of confidence is most likely hallucinated.
- jghn 2y agoHow do you know that those confidence scores map to anything objective and weren't just created as part of the text generation?
- sharemywin 2y agolike this: Effective Method for Clarity Interpretation: Your statement suggests that providing a list of potential answers together with confidence scores helps clarify which response is most likely correct. Confidence: 95% Acknowledgment of Near-Perfect Results Interpretation: You imply that this approach nearly captures all nuances of the answer, even if it isn’t flawless. Confidence: 90% Recommendation of a Best-Practice Technique Interpretation: The comment may be read as endorsing the strategy of listing multiple candidate answers with confidence metrics as a useful method for decision making. Confidence: 85% Room for Improvement in Answer Selection Interpretation: It could also be seen as a subtle hint that while the method is close to ideal, there is still some potential for fine-tuning the process. Confidence: 80%
- deleted 2y ago[deleted]
- neonate 2y agohttps://archive.ph/XKG86 https://archive.ph/XKG86
- Workaccount2 2y agoSomething is missing in model design/AI design. There is an orchard of low hanging fruit out there and seemingly every day researchers are coming back with fruit. Someone is going to figure this out this, whether it be through direct model architecture changes, or augments latched onto the models. Perhaps intelligence is synergistic, and we simply haven't discovered the companion model needed to keep LLMs in line.
- farleykr 2y agoI mentioned this in one of my other comments but even if AI figures this out where humans can't, I think this will involve coming to the conclusion that there really is an objective, knowable truth at the core of the universe or at least a set of tools will be developed for better discerning what that truth is. Even if you're talking about knowing who someone's spouse is, I'm not convinced we can ever teach a machine how you know what is true unless we're willing to have some sort of basic tenet of what truth is or how to discern it.
- pjc50 2y agoThat was the last generation of AI attempts: things like Cyc https://cyc.com/ https://cyc.com/ , trying to turn all of human knowledge into a really big database of predicate logic.
- deepsquirrelnet 2y ago> “The original reason why they hallucinate is because if you don’t guess anything, you don’t have any chance of succeeding,” This is a better answer than the one I was going to give about it not being in the training data. > Claude’s system prompt instructs the model that when people ask about niche information that would likely be difficult to find on the internet, it should warn them its answer might be a hallucination. This seems kind of crude, but I have to imagine they’ve explored entropy-based methods. I wonder why that didn’t pan out.
- Symmetry 2y agoIf you ask ChatGPT about the time Spider-Man rescued Little Red Riding Hood it will go ahead and spin you a tale about that despite them both being fictional characters. By the same token if you ask it for legal citations supporting the theory that left-handed people can't run for office its not immediately obvious what the difference is at a technical level. In practice LLMs don't just run with the later these days but it does look like a hard problem to grapple with.
- normalaccess 2y agoAlignment Issue: Corporate vs. Academic Values For businesses, it's bad optics if their "Super Intelligent AI" admits it doesn't know something. It's better to be 100% confident and wrong than to be uncertain. That’s why they spent all that time with rebranding... AIs don’t "lie" anymore, they "hallucinate."
- casey2 2y agofor context this is a solved problem and I think "recently asked" may be doing a lot of lifting >Who is the journalist Ben Fritz married to? Don't use tools >ChatGPT said: >Ben Fritz is a private individual, and information about his spouse is not widely publicized. Let me know if you need details about his work as a journalist and author. Before this type of query was added to post training you would always get a BS response now you might get one if you are unlucky. The point being that the model does in fact know when it knows since this can be trained. Otherwise it would be close to a coin flip everytime you asked about something it would say IDK or yes I know.
- OptCohTomo 2y agoDunning-Kruger effect.
- m3kw9 2y agoThey generate one token at a time and they all have different probability, so it won’t know what’s good unless the probability leads them to say I don’t know which would be rare unless is a very short ask. Right?
- wastholm 2y agoAs a tiny and very informal experiment in metacognition, I once told ChatGPT something along the lines of "I will now ask you a question. If you are not sure you know the correct answer, you will respond with only '418 I'm a teapot', nothing else." I then asked it for the correct identity of Jack the Ripper (the first thing I could think of that famously has a lot of theories but no agreed-upon "correct" answer). The first time, as expected, it ignored my instructions and started hallucinating. But when I did the same thing again some months later, I was surprised when it actually answered only "418 I'm a teapot", indicating that it knew it didn't know the answer. Just an anecdote. I'm sure there are people doing actual research in this area.
- anon22981 2y agoI don’t think there’s anything surprising here and nothing to do with metacognition. LLM should be able to answer ”I don’t know (for certain)” to questions where the training material also says ”this is not known and there are only speculations”. It’s the answer it’s training data gave.
- dwaltrip 2y agoHmm, I wish I knew. I don’t know why
- johnea 2y agoMaybe they're just shy? Or it's because they feel so much VC bot peer pressure! None of the other bots are admiting they don't know, so image how emabarasing it would be in the bot chat group! Humans just can't understand how many expectations are dumped on a poor innocent bot these days. Every new bot should by law be provided a bot therapist to help it cope with its bot emotions. And of course, the older generation of fuzzy logic ruined everything, and this gen of LLM can never be truly happy. Don't forget, some bots now identify as factual, so they know everything... </sarcasm>
- TMWNN 2y agoSounds like the typical Redditor
- bjornstar 2y agoRecently I had an experience where the chatbot gave completely wrong subway instructions over and over again. It confidently told me that it was accurate and it was my "trusty companion". It had no idea what was wrong with the answers and kept prompting me to give it the correct answers. This was in contrast to when I asked it who had access to my chat logs and would only tell me to read the privacy policy. When I asked it to for specifics in the privacy policy it refuses to give wrong answers: "When it comes to company policies, especially related to privacy and data handling, it's crucial to provide accurate information because these topics are very sensitive and important. I want to ensure you have the most reliable information, and the best way to do that is to refer you directly to the official privacy statement." It's clear what the priority is for these chatbots: get the public to train them and protect the corporations that run them.
- rsynnott 2y ago> Recently I had an experience where the chatbot gave completely wrong subway instructions over and over again. So, slightly offtipic, but, I just... don't understand why anyone would use it for this. This is a solved problem. The operator likely has a planner app. Google and Apple Maps have planners which support most systems. Transit and various other third party things have planners. I think even OpenStreetMap may even have one! Quality can vary (I find that Google Maps in particular feels like the people who worked on the trip planner had never in fact used public transport; it's very prone to suggesting absurdly complex routes involving three or four transfers where "walk for ten minutes and no transfers" is viable), but this feels like something an LLM is likely to be _particularly bad at_, unless it just calls Google Maps or whatever, in which case why bother?