5 ms·

LLMs don’t know when they are hallucinating! They don’t know when they don’t know something. For example even when they’re told it’s ok to say “I don’t know” wh

LLMs don’t know when they are hallucinating! They don’t know when they don’t know something. For example even when they’re told it’s ok to say “I don’t know” when they do that, it’s because “I don’t know” is often a response to that question space. They know facts but they have no reasoning.

ars 2y ago
Can they not detect is a word has a high or low probability to result from the input? And if the probability is low it should consider that as hallucinating?
- creatonez 2y ago
  The "heat" value lets you control the algorithm to choose greater or lower probability outputs. Problem is, once it's already in a good range, it's not very predictable for increasing the quality of output. Often the hallucinatory results don't actually have a low probability, in which case the same hallucination will come out in multiple attempts. However, it might be interesting to try to make the model directly aware of these numbers by putting it into a post-message evaluation prompt, because currently the token randomization algorithm sits entirely behind the scenes.
- sqeaky 2y ago
  No. N/A They can't detect if a word is high or low probability because current LLMs can't detect anything. They don't reason as you or I understand it. They are really fancy word predictors and somewhere in the word prediction machinery it does have the probabilities stored but not in anyway accessible to the output. Imagine you are typing on your phone and the word autocomplete comes up and you just keep clicking the next word it suggests. What could you type to get at the inference data in your phone's keyboard? Of course a system that does this could be built, but that isn't how current LLMs or phone keyboards are built.
  - kolinko 2y ago
    What? Of course it is accessible. When you ask an LLM to estimate if a given answer was hallucinated, it converts internal probabilities into a token(s) that represent these probabilities. Here you have an output of chatGPT where it estimates it's probabilities of hallucination - which I would argue are quite close to correct: https://chatgpt.com/share/e/e10cbc17-cc35-432f-872b-cb06170026c6 https://chatgpt.com/share/e/e10cbc17-cc35-432f-872b-cb061700...
    - sqeaky 2y ago
      This is likely a hallucination. You are committing a type error or perhaps succumbing to circular reasoning. No modern LLM can query, let alone understand, the values in its own weights. They are simply inputs for the vector math. If they happen to be correct that is a either a coincidence or because it was trained with data including conversations about its past abilities. Consider this, do you thinks the internal weights are labeled and directly to some attribute of the world outside? If so then present evidence. They are not, so if it does attempt to query its own internal weights how does it know what to query? This could be an interesting academic problem and I would encourage you to present papers on this because last I looked this was and itractable as of 2023 for the much simpler neural networks of simple backpropagation networks from 20 years ago. Also, using ChatGPT as a reference here shows a major deficiency in your judgement. At best this is circular reasoning but likely you simply don't know how to allocate trust or how others do. The argument is X is unreliable then citing X is obvious not going to convince people who distrust X. Additionally, experts
      - sqeaky 2y ago
        To expand on this consider the amount of brains that can directly query the amount of a specific neurotransmitter used between two neurons and interpret what that means. That isn't exactly the same, but a hopefully illustrative analogy. An LLM's output is an emergent property of its structure and all the weights being multiplied together just as an organism's intelligence is an emergent property of its brain structure and communication between neurons. Neither of the emergent parts can currently reliably query the substrate they emerge from for similar reasons. That isn't to say that such an LLM or biological brain couldn't be built, but none presently are.
amelius 2y ago
But they can use tools such as databases and search engines, and I guess also logic engines.
Sharlin 2y ago
I don't think there's any fundamental reason why a LLM shouldn't be able to learn the distinction between being creative and being factual, at least much better than currently. It's just another classification problem (if a challenging one), and classification is what NNs are known for.
- szundi 2y ago
  Source of your disbelief?
  - Sharlin 2y ago
    Call it Bayesian intuition. I’d argue that at this point the specific claim "LLMs cannot solve the confabulation problem" is the one that requires hard evidence. These things forced almost all of us to update our priors on what glorified Markov chains are – and are not – capable of if you throw enough data and compute at them.
- b112 2y ago
  From what I see, there may be two issues here. 1) LLM has data from a large set of sources, but in some cases it has very little data on certain sub-topics. With so few sources of data, it is easier for some weird corner of the internet to have conflicting data, and some LLM models may randomly favour that instead of the valid answer. So not hallucination in this case, instead, a wrong answer from wrong sources. 2) LLM doesn't know, but has seen from human behaviour that just "making shit up, when you don't know" is entirely acceptable for many humans. Here I'm not referring to people saying (as I am) "may be", but instead "No, it's this!" with immense surety, in order to win discussions. Combine this with 'temperature' settings many model interfaces have, and there you go. 2b) Alternatively, #2 is inline with "I believe in flying spaghetti monsters", and other beliefs such as astrology, ghosts, and endless other "there is zero proof, but humans have immense weird data on it". So #2 is solvable by changing temperature, or keeping temperature but saying "Don't make things up". #1 may be temp + random choice of a lower probability pathway. #2b, not sure what to think here. How will LLMs ingest data, and use it to determine what is acceptable logic processing, when a lot of stuff in category 2b has zero proof behind it. And with zero proof, people are absolutely positive of the status of said thing? #2b trains LLMs that "just having a random idea and then even killing people that don't agree, seems OK for humans". (think religious wars, violent acts, burning witches, etc)
mseepgood 2y ago
> They don’t know when they don’t know something. Dunning-Kruger effect
317070 2y ago
> LLMs don’t know when they are hallucinating What makes you think that? The fact that you can ask them to stop hallucinating and that this does reduce hallucinations seems to me like a strong indication of the opposite? That is what the OP's article is about. > They don’t know when they don’t know something. The bigger models tend to be able to articulate some of their limits? See e.g this recent article [0]. > They know facts but they have no reasoning. Is that really true? They can solve many novel tasks that seem to require some level of reasoning. They do not possess the full range of reasoning capabilities most humans have, but do pretty well on the HellaSwag dataset which aims to test reasoning [1], even when test set leakage was not a big problem yet [2]. I have 10+ years of experience in machine learning, and I don't find your claims obvious at all. In fact, I see more and more indications of the opposite. Also if you look mechanistically at how LLMs work inside, there is nothing stopping self-knowledge, reasoning or hallucination-detection. LLMs are very black box. [0] https://arxiv.org/abs/2305.18153 https://arxiv.org/abs/2305.18153 [1] https://deepgram.com/learn/hellaswag-llm-benchmark-guide https://deepgram.com/learn/hellaswag-llm-benchmark-guide [2] https://paperswithcode.com/sota/sentence-completion-on-hellaswag https://paperswithcode.com/sota/sentence-completion-on-hella...
herbst 2y ago
Ask GPT4o to return a percentual score after each answer how likely something is 100% true. Did that for old regional stories and the higher the score the easier I could verify the story or parts of it. It seems to know, at least in some cases
- michaelt 2y ago
  I performed several thousand batch LLM queries a few months back and asked for such scores. My impression was, while the resulting scores weren't entirely random, they were pretty much random when the input was difficult. (I was mapping supermarket product descriptions to packaging materials - which is sometimes very easy, as sometimes it's obvious or the description includes explicit recycling information, and other times almost impossible, as for products like olive oil, glass and plastic bottles are equally plausible)
  - herbst 2y ago
    I guess that's way more complex because there is no one right answer. (Interesting use case btw) And if you ask gpt for an specific answer to something it can't actually know for sure it gets weird fast. Asking it to guess how much of what it told is based on actual things it 'knows' at least seems like a reasonable question for a LLM with only one right answer.
- user-the-name 2y ago
  [dead]
ben_w 2y ago
I was very into the occult as a teenager, thanks to my mother's eclectic attitudes: although theoretically Catholic, she also had Orthodox icons, Hindu statues, runic tarot kits, "magic" crystals, dowsing rods, homeopathic and Bach flower "remedies", books about leylines. I'm as sure as I can be that she didn't know she was wrong; I can definitely tell you that I didn't know that I was.
kolinko 2y ago
This has been disproven by multiple experiments, and also - there is nothing in the transformer architecture that would suggest this. On the contrary - even the most basic classifier neural networks by default produce not a binary answer, but an answer with confidence level.
- lewispollard 2y ago
  And if that confidence level is high enough for it to output an answer, but that answer is wrong or nonsensical, what do we call it?
  - ben_w 2y ago
    In humans? Overconfidence.
  - kolinko 2y ago
    LLMs have no idea if they hallucinate ≠ LLM make no mistakes. The former statement is a bold claim that would require some sort of evidence. The latter nobody really argues with.
    - lewispollard 2y ago
      "Hallucinate" is just a term we've come up with for LLMs making mistakes. If the LLM knew it was making a mistake, wouldn't it just not make it? Of course, it's not possible for an LLM to actually "know" anything in that sense of the word.
mewpmewp2 2y ago
Isn't hallucinating a very wrong word? Hallucinating should be about twisted input misinterpretation. E.g. if the text input you give them was malformed at some level. What they are doing is try to give you an answer based on compressed knowledge, which might be true, but might not because it's compressed and lossy. Because they are trained to bias to give an answer even if it's not 100% clear from compressed knowledge. The larger the model, the less compressed the knowledge and better they would be able to differ when it is something they know vs something that they likely don't.
- seba_dos1 2y ago
  It is, but that ship has sailed. "Confabulation" would be a better word to describe it.
amy-petrik-214 2y ago
>LLMs don’t know when they are hallucinating! Gosh, you sound like me at my last Phish concert. My partner was all like "you're tripping hard but you have no idea". In fairness I was, and I thought I was an AI at the time, or was I ? Anyway you can't really tell a sentience "hey you are hallucinating, stop hallucinating." It's like telling a woman "don't be angry baby", it only makes them angrier - similarly when you tell a person (or AI) "cut it out with the hallucinatin' " usually it causes them to trip harder.

ars 2y ago

Can they not detect is a word has a high or low probability to result from the input? And if the probability is low it should consider that as hallucinating?

creatonez 2y ago
The "heat" value lets you control the algorithm to choose greater or lower probability outputs. Problem is, once it's already in a good range, it's not very predictable for increasing the quality of output. Often the hallucinatory results don't actually have a low probability, in which case the same hallucination will come out in multiple attempts. However, it might be interesting to try to make the model directly aware of these numbers by putting it into a post-message evaluation prompt, because currently the token randomization algorithm sits entirely behind the scenes.
sqeaky 2y ago
No. N/A They can't detect if a word is high or low probability because current LLMs can't detect anything. They don't reason as you or I understand it. They are really fancy word predictors and somewhere in the word prediction machinery it does have the probabilities stored but not in anyway accessible to the output. Imagine you are typing on your phone and the word autocomplete comes up and you just keep clicking the next word it suggests. What could you type to get at the inference data in your phone's keyboard? Of course a system that does this could be built, but that isn't how current LLMs or phone keyboards are built.
- kolinko 2y ago
  What? Of course it is accessible. When you ask an LLM to estimate if a given answer was hallucinated, it converts internal probabilities into a token(s) that represent these probabilities. Here you have an output of chatGPT where it estimates it's probabilities of hallucination - which I would argue are quite close to correct: https://chatgpt.com/share/e/e10cbc17-cc35-432f-872b-cb06170026c6 https://chatgpt.com/share/e/e10cbc17-cc35-432f-872b-cb061700...
  - sqeaky 2y ago
    This is likely a hallucination. You are committing a type error or perhaps succumbing to circular reasoning. No modern LLM can query, let alone understand, the values in its own weights. They are simply inputs for the vector math. If they happen to be correct that is a either a coincidence or because it was trained with data including conversations about its past abilities. Consider this, do you thinks the internal weights are labeled and directly to some attribute of the world outside? If so then present evidence. They are not, so if it does attempt to query its own internal weights how does it know what to query? This could be an interesting academic problem and I would encourage you to present papers on this because last I looked this was and itractable as of 2023 for the much simpler neural networks of simple backpropagation networks from 20 years ago. Also, using ChatGPT as a reference here shows a major deficiency in your judgement. At best this is circular reasoning but likely you simply don't know how to allocate trust or how others do. The argument is X is unreliable then citing X is obvious not going to convince people who distrust X. Additionally, experts
    - sqeaky 2y ago
      To expand on this consider the amount of brains that can directly query the amount of a specific neurotransmitter used between two neurons and interpret what that means. That isn't exactly the same, but a hopefully illustrative analogy. An LLM's output is an emergent property of its structure and all the weights being multiplied together just as an organism's intelligence is an emergent property of its brain structure and communication between neurons. Neither of the emergent parts can currently reliably query the substrate they emerge from for similar reasons. That isn't to say that such an LLM or biological brain couldn't be built, but none presently are.

amelius 2y ago

But they can use tools such as databases and search engines, and I guess also logic engines.

Sharlin 2y ago

I don't think there's any fundamental reason why a LLM shouldn't be able to learn the distinction between being creative and being factual, at least much better than currently. It's just another classification problem (if a challenging one), and classification is what NNs are known for.

szundi 2y ago
Source of your disbelief?
- Sharlin 2y ago
  Call it Bayesian intuition. I’d argue that at this point the specific claim "LLMs cannot solve the confabulation problem" is the one that requires hard evidence. These things forced almost all of us to update our priors on what glorified Markov chains are – and are not – capable of if you throw enough data and compute at them.
b112 2y ago
From what I see, there may be two issues here. 1) LLM has data from a large set of sources, but in some cases it has very little data on certain sub-topics. With so few sources of data, it is easier for some weird corner of the internet to have conflicting data, and some LLM models may randomly favour that instead of the valid answer. So not hallucination in this case, instead, a wrong answer from wrong sources. 2) LLM doesn't know, but has seen from human behaviour that just "making shit up, when you don't know" is entirely acceptable for many humans. Here I'm not referring to people saying (as I am) "may be", but instead "No, it's this!" with immense surety, in order to win discussions. Combine this with 'temperature' settings many model interfaces have, and there you go. 2b) Alternatively, #2 is inline with "I believe in flying spaghetti monsters", and other beliefs such as astrology, ghosts, and endless other "there is zero proof, but humans have immense weird data on it". So #2 is solvable by changing temperature, or keeping temperature but saying "Don't make things up". #1 may be temp + random choice of a lower probability pathway. #2b, not sure what to think here. How will LLMs ingest data, and use it to determine what is acceptable logic processing, when a lot of stuff in category 2b has zero proof behind it. And with zero proof, people are absolutely positive of the status of said thing? #2b trains LLMs that "just having a random idea and then even killing people that don't agree, seems OK for humans". (think religious wars, violent acts, burning witches, etc)

mseepgood 2y ago

> They don’t know when they don’t know something. Dunning-Kruger effect

317070 2y ago

> LLMs don’t know when they are hallucinating What makes you think that? The fact that you can ask them to stop hallucinating and that this does reduce hallucinations seems to me like a strong indication of the opposite? That is what the OP's article is about. > They don’t know when they don’t know something. The bigger models tend to be able to articulate some of their limits? See e.g this recent article [0]. > They know facts but they have no reasoning. Is that really true? They can solve many novel tasks that seem to require some level of reasoning. They do not possess the full range of reasoning capabilities most humans have, but do pretty well on the HellaSwag dataset which aims to test reasoning [1], even when test set leakage was not a big problem yet [2]. I have 10+ years of experience in machine learning, and I don't find your claims obvious at all. In fact, I see more and more indications of the opposite. Also if you look mechanistically at how LLMs work inside, there is nothing stopping self-knowledge, reasoning or hallucination-detection. LLMs are very black box. [0] https://arxiv.org/abs/2305.18153 https://arxiv.org/abs/2305.18153 [1] https://deepgram.com/learn/hellaswag-llm-benchmark-guide https://deepgram.com/learn/hellaswag-llm-benchmark-guide [2] https://paperswithcode.com/sota/sentence-completion-on-hellaswag https://paperswithcode.com/sota/sentence-completion-on-hella...

herbst 2y ago

Ask GPT4o to return a percentual score after each answer how likely something is 100% true. Did that for old regional stories and the higher the score the easier I could verify the story or parts of it. It seems to know, at least in some cases

michaelt 2y ago
I performed several thousand batch LLM queries a few months back and asked for such scores. My impression was, while the resulting scores weren't entirely random, they were pretty much random when the input was difficult. (I was mapping supermarket product descriptions to packaging materials - which is sometimes very easy, as sometimes it's obvious or the description includes explicit recycling information, and other times almost impossible, as for products like olive oil, glass and plastic bottles are equally plausible)
- herbst 2y ago
  I guess that's way more complex because there is no one right answer. (Interesting use case btw) And if you ask gpt for an specific answer to something it can't actually know for sure it gets weird fast. Asking it to guess how much of what it told is based on actual things it 'knows' at least seems like a reasonable question for a LLM with only one right answer.
user-the-name 2y ago
[dead]

ben_w 2y ago

I was very into the occult as a teenager, thanks to my mother's eclectic attitudes: although theoretically Catholic, she also had Orthodox icons, Hindu statues, runic tarot kits, "magic" crystals, dowsing rods, homeopathic and Bach flower "remedies", books about leylines. I'm as sure as I can be that she didn't know she was wrong; I can definitely tell you that I didn't know that I was.

kolinko 2y ago

This has been disproven by multiple experiments, and also - there is nothing in the transformer architecture that would suggest this. On the contrary - even the most basic classifier neural networks by default produce not a binary answer, but an answer with confidence level.

lewispollard 2y ago
And if that confidence level is high enough for it to output an answer, but that answer is wrong or nonsensical, what do we call it?
- ben_w 2y ago
  In humans? Overconfidence.
- kolinko 2y ago
  LLMs have no idea if they hallucinate ≠ LLM make no mistakes. The former statement is a bold claim that would require some sort of evidence. The latter nobody really argues with.
  - lewispollard 2y ago
    "Hallucinate" is just a term we've come up with for LLMs making mistakes. If the LLM knew it was making a mistake, wouldn't it just not make it? Of course, it's not possible for an LLM to actually "know" anything in that sense of the word.

mewpmewp2 2y ago

Isn't hallucinating a very wrong word? Hallucinating should be about twisted input misinterpretation. E.g. if the text input you give them was malformed at some level. What they are doing is try to give you an answer based on compressed knowledge, which might be true, but might not because it's compressed and lossy. Because they are trained to bias to give an answer even if it's not 100% clear from compressed knowledge. The larger the model, the less compressed the knowledge and better they would be able to differ when it is something they know vs something that they likely don't.

seba_dos1 2y ago
It is, but that ship has sailed. "Confabulation" would be a better word to describe it.

amy-petrik-214 2y ago

>LLMs don’t know when they are hallucinating! Gosh, you sound like me at my last Phish concert. My partner was all like "you're tripping hard but you have no idea". In fairness I was, and I thought I was an AI at the time, or was I ? Anyway you can't really tell a sentience "hey you are hallucinating, stop hallucinating." It's like telling a woman "don't be angry baby", it only makes them angrier - similarly when you tell a person (or AI) "cut it out with the hallucinatin' " usually it causes them to trip harder.