6 ms·

The question is, why don’t LLMs admit when they don’t know an answer? If they can’t, how do we make them so they can? Plenty of humans seem to have no problem

The question is, why don’t LLMs admit when they don’t know an answer? If they can’t, how do we make them so they can? Plenty of humans seem to have no problem admitting when they don’t know something.

foobarbecue 3y ago
And the answer is: LLMs don't know ANY answers. They just know what answers tend to sound like. So it's a fundamental problem.
- SirMaster 3y ago
  Well I keep seeing people suggest that humans are also really just sophisticated LLMs. If that’s true then this behavior difference (identifying and admitting when you don’t know something) seems like a pretty fundamental difference between humans and LLMs at least at this point.
  - iainmerrick 3y ago
    Many people are bad at this too. It's an important and useful skill to learn, e.g. admitting in team meetings that you don't understand something. I wonder -- does that kind of interaction come up in spoken conversation more than in written text? Things like blog posts and tweets are often stated very confidently. If LLMs are trained mostly from those sorts of sources, maybe that's why they end up artificially overconfident.
- famouswaffles 3y ago
  Decidedly not true. https://news.ycombinator.com/item?id=37775805 https://news.ycombinator.com/item?id=37775805
  - not_enoch_wise 3y ago
    Thoroughly supported by arxiv links. Finally, a paper publishing site unburdened by peer review!
tkgally 3y ago
In my interactions with GPT-4 recently, it often says that it is unsure or doesn’t know. I got access yesterday to its image-interpretation function. Below are some results in which it expresses uncertainty about what appears in a photograph. https://www.gally.net/temp/20231015gpt4hedging/index.html https://www.gally.net/temp/20231015gpt4hedging/index.html The only confabulations are when reading text. In my tests so far, GPT-4 is not a good OCR engine and does make up a lot.
- wizofaus 3y ago
  Wow, it can't even recognise obviously Japanese characters?
  - tkgally 3y ago
    No. I also tested GPT-4 with a photo containing some very clear kanji and hiragana, and it misidentified every single character. For English text it worked okay with a contemporary document, but when I tried a 17th century text it hallucinated a lot of the words. Conventional character-by-character OCR is much more reliable. The samples I posted above don’t contain any major hallucinations about nontext image components, but other tests I’ve done with GPT-4 did. But GPT-4’s image recognition still seems better than Bard’s, which has been pretty bad in the tests I’ve done with it.
    - eurekin 3y ago
      I wouldn't be surprised if system token included "this photo contains following text: <output of tesseract ocr here>"
famouswaffles 3y ago
Because they don't care to. No seriously that's it. Lots of evidence indicating language models know when they hallucinate. GPT-4 logits calibration pre RLHF - [https://imgur.com/a/3gYel9r](https://imgur.com/a/3gYel9r) https://imgur.com/a/3gYel9r](https://imgur.com/a/3gYel9r) Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - [https://arxiv.org/abs/2305.14975](https://arxiv.org/abs/2305.14975) https://arxiv.org/abs/2305.14975](https://arxiv.org/abs/2305... Teaching Models to Express Their Uncertainty in Words - [https://arxiv.org/abs/2205.14334](https://arxiv.org/abs/2205.14334) https://arxiv.org/abs/2205.14334](https://arxiv.org/abs/2205... Language Models (Mostly) Know What They Know - [https://arxiv.org/abs/2207.05221](https://arxiv.org/abs/2207.05221) https://arxiv.org/abs/2207.05221](https://arxiv.org/abs/2207... TLDR; Hallucinations aren't a randomness or representation issue. The computation already knows. It just doesn't care about telling you this.
- SirMaster 3y ago
  "It just doesn't care about telling you this." But why? Why would the LLM creators want the LLM giving out knowingly bad information? This is what I don't understand. It makes me not want to use the product. Same as I don't like asking people things who I know like to just spew bad information when they don't know rather than simply saying they don't know.
  - famouswaffles 3y ago
    LLM creators train the model to predict the next token. They have no control over what it learns in the process. We're not programming anything into artificial neural networks so control or direction is limited.
    - SirMaster 3y ago
      It sure seems like they are programming things into the LLMs with how it often states "As a LLM I can't do that" sort of responses. So how do we get LLMs to admit when they don't know something rather than confabulate? Seems like that would be a really great improvement to me.

foobarbecue 3y ago

And the answer is: LLMs don't know ANY answers. They just know what answers tend to sound like. So it's a fundamental problem.

SirMaster 3y ago
Well I keep seeing people suggest that humans are also really just sophisticated LLMs. If that’s true then this behavior difference (identifying and admitting when you don’t know something) seems like a pretty fundamental difference between humans and LLMs at least at this point.
- iainmerrick 3y ago
  Many people are bad at this too. It's an important and useful skill to learn, e.g. admitting in team meetings that you don't understand something. I wonder -- does that kind of interaction come up in spoken conversation more than in written text? Things like blog posts and tweets are often stated very confidently. If LLMs are trained mostly from those sorts of sources, maybe that's why they end up artificially overconfident.
famouswaffles 3y ago
Decidedly not true. https://news.ycombinator.com/item?id=37775805 https://news.ycombinator.com/item?id=37775805
- not_enoch_wise 3y ago
  Thoroughly supported by arxiv links. Finally, a paper publishing site unburdened by peer review!

tkgally 3y ago

In my interactions with GPT-4 recently, it often says that it is unsure or doesn’t know. I got access yesterday to its image-interpretation function. Below are some results in which it expresses uncertainty about what appears in a photograph. https://www.gally.net/temp/20231015gpt4hedging/index.html https://www.gally.net/temp/20231015gpt4hedging/index.html The only confabulations are when reading text. In my tests so far, GPT-4 is not a good OCR engine and does make up a lot.

wizofaus 3y ago
Wow, it can't even recognise obviously Japanese characters?
- tkgally 3y ago
  No. I also tested GPT-4 with a photo containing some very clear kanji and hiragana, and it misidentified every single character. For English text it worked okay with a contemporary document, but when I tried a 17th century text it hallucinated a lot of the words. Conventional character-by-character OCR is much more reliable. The samples I posted above don’t contain any major hallucinations about nontext image components, but other tests I’ve done with GPT-4 did. But GPT-4’s image recognition still seems better than Bard’s, which has been pretty bad in the tests I’ve done with it.
  - eurekin 3y ago
    I wouldn't be surprised if system token included "this photo contains following text: <output of tesseract ocr here>"

famouswaffles 3y ago

Because they don't care to. No seriously that's it. Lots of evidence indicating language models know when they hallucinate. GPT-4 logits calibration pre RLHF - [https://imgur.com/a/3gYel9r](https://imgur.com/a/3gYel9r) https://imgur.com/a/3gYel9r](https://imgur.com/a/3gYel9r) Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - [https://arxiv.org/abs/2305.14975](https://arxiv.org/abs/2305.14975) https://arxiv.org/abs/2305.14975](https://arxiv.org/abs/2305... Teaching Models to Express Their Uncertainty in Words - [https://arxiv.org/abs/2205.14334](https://arxiv.org/abs/2205.14334) https://arxiv.org/abs/2205.14334](https://arxiv.org/abs/2205... Language Models (Mostly) Know What They Know - [https://arxiv.org/abs/2207.05221](https://arxiv.org/abs/2207.05221) https://arxiv.org/abs/2207.05221](https://arxiv.org/abs/2207... TLDR; Hallucinations aren't a randomness or representation issue. The computation already knows. It just doesn't care about telling you this.

SirMaster 3y ago
"It just doesn't care about telling you this." But why? Why would the LLM creators want the LLM giving out knowingly bad information? This is what I don't understand. It makes me not want to use the product. Same as I don't like asking people things who I know like to just spew bad information when they don't know rather than simply saying they don't know.
- famouswaffles 3y ago
  LLM creators train the model to predict the next token. They have no control over what it learns in the process. We're not programming anything into artificial neural networks so control or direction is limited.
  - SirMaster 3y ago
    It sure seems like they are programming things into the LLMs with how it often states "As a LLM I can't do that" sort of responses. So how do we get LLMs to admit when they don't know something rather than confabulate? Seems like that would be a really great improvement to me.