6 ms·

Yes, exactly, it’s a post-facto value judgment, not a precise term. If I understand the meaning of the word, “hallucination” is all the model does. If it happen

by wrs 2y ago

Yes, exactly, it’s a post-facto value judgment, not a precise term. If I understand the meaning of the word, “hallucination” is all the model does. If it happens to hallucinate something we think is objectively true, we just decide not to call that a “hallucination”. But there’s literally no functional difference between that case and the case of the model saying something that’s objectively false, or something whose objective truth is unknown or undefinable. I haven’t read the paper yet, but if they resolve this definition usefully, that would be a good contribution.

diputsmonro 2y ago
Exactly this, I've been saying this since the beginning. Every response is a hallucination - a probabilistic string of words divorced from any concept of truth or reality. By total coincidence, some hallucinations happen to reflect the truth, but only because the training data happened to generally be truthful sentences. Therefore, creating something that imitates a truthful sentence will often happen to also be truthful, but there is absolutely no guarantee or any function that even attempts to enforce that. All responses are hallucinations. Some hallucinations happen to overlap the truth.
- fumeux_fume 2y ago
  Ok, but I think it would be more productive to educate people that LLMs have no concept of truth rather than insist they use the term "hallucinate" in an unintuitive way.
  - lolinder 2y ago
    I don't know about OP, but I'm suggesting that the term 'hallucinate' be abolished entirely as applies to LLMs, not redefined. It draws an arbitrary line in the middle of the set of problems that all amount to "how do we make sure that the output of an LLM is consistently acceptable" and will all be solved using the same techniques if at all.
  - aeternum 2y ago
    LLMs do now have a concept of truth now since much of the RLHF is focused on making them more accurate and true. I think the problem is that humanity has a poor concept of truth. We think of most things as true or not true when much of our reality is uncertain due to fundamental limitations or because we often just don't know yet. During covid for example humanity collectively hallucinated the importance of disinfecting groceries for awhile.
    - jbm 2y ago
      > humanity collectively hallucinated the importance of disinfecting groceries for awhile I reject this history. I homeschooled my kids during covid due to uncertainty and even I didn't reach that level, and nor did anyone I knew in person. A very tiny number who were egged on by some YouTubers did this, including one person I knew remotely. Unsurprisingly that person was based in SV.
      - tsimionescu 2y ago
        It's not some extremist on YouTube, disinfecting your groceries was the official recommendation of many countries worldwide, including most of Europe. I couldn't say how many people actually followed the recommendation , but I would bet it's way more than a tiny number.
        SirMaster 2y ago
        This is the first I’ve even heard of people disinfecting their groceries because of Covid. Honestly that sounds rather crazy to me.
        tsimionescu 2y ago
        There was a period near the start of the pandemic, especially while the medical establishment was trying to avoid ordinary people wearing masks in order to help stockpile them for high priority workers, when a lot of emphasis was put on surface contact. If it's extremely important to wear gloves and keep sanitizing your hands after touching every part of the supermarket, it stands to reason that you'd want to sanitize all of the outside packaging that others touched with their diseased hands as soon as you brought it into your house. Otherwise, you'd be expected to sanitize your hands every time you touched those items again, even at home, right? Of course, surface contact is actually a very minor avenue of infection, and pretty much limited to cases where someone has just sneezed or coughed on a surface that you are touching, and then putting your hand to your nose or maybe eyes or mouth soon after. So sanitizing groceries is essentially pointless, since it only slightly reduces an already very small risk.
  - diputsmonro 2y ago
    If people already understand what "hallucination" means, then I think it's perfectly intuitive and educational to say that, actually, the LLM is always doing that, just that some of those hallucinations happen to coincidentally describe something real. We need to dispell the notion that the LLM "knows" the truth, or is "smart". It's just a fancy stochastic parrot. Whether it's responses reflect a truthful reality or a fantasy it made up is just luck, weighted by (but not constrained to) its training data. Emphasizing that everything is a hallucination does that. I purposefully want to reframe how the word is used and how we think about LLMs.
- plaidfuji 2y ago
  In other words, all models are wrong, but some are useful.
  - diputsmonro 2y ago
    Precisely, I'm glad someone picked up on the reference!
- TeMPOraL 2y ago
  I think you're going too far here. > By total coincidence, some hallucinations happen to reflect the truth, but only because the training data happened to generally be truthful sentences. It's not a "total coincidence". It's the default. Thus, the model's responses aren't "divorced from any concept of truth or reality" - the whole distribution from which those responses are pulled is strongly aligned with reality. (Which is why people started using the term "hallucinations" to describe the failure mode, instead of "fishing a coherent and true sentence out of line noise" to describe the success mode - because success mode dominates.) Humans didn't invent language for no reason. They don't communicate to entertain themselves with meaningless noises. Most of communication - whether spoken or written - is deeply connected to reality. Language itself is deeply connected to reality. Even the most blatant lies, even all of fiction writing, they're all incorrect or fabricated only at the surface level - the whole thing, accounting for the utterance, what it is about, the meanings, the words, the grammar - is strongly correlated with truth and reality. So there's absolutely no coincidence that LLMs get things right more often than not. Truth is thoroughly baked into the training data, simply because it's a data set of real human communication, instead of randomly generated sentences.
  - MichaelZuo 2y ago
    Strongly correlated != 100% overlap A 99% overlap can still be coincidence. And even if it was ‘absolutely no coincidence’, that is still only reflective of the reality as perceived by the average of all the people from the training set.
  - zdragnar 2y ago
    The problem - as defined by how end users understand it - is that the model itself doesn't know the difference, and will proclaim bullshit with the same level of confidence that it does accurate information. That's how you end up with grocery store chatbots recommending mixing ammonia and bleach for a cocktail, or lawyers using chatbots to cite entirely fictional case law before a judge in court. Nothing that comes out of an LLM can be implicitly trusted, so your default assumption must be that everything it gives you needs verification from another source. Telling people "the truth is baked in" is just begging for a disaster.
- codr7 2y ago
  So much truth here, very refreshing to see! About time too, the sooner we can stop the madness the better, building a society on top of this technology is a movie I'd rather not see.
- SirMaster 2y ago
  Maybe they shouldn’t have mixed truthful data with obviously untruthful data in the same training data set? Why not make a model only from truthful data? Like exclude all fiction for example.
  - vanviegen 2y ago
    That wouldn't prevent hallucination. An LLM doesn't know what it doesn't know. It will always try to come up with a response that sounds plausible, based on its knowledge or lack thereof.
  - lolinder 2y ago
    1) It's impossible to get enough data to train one of these well while also curating it by hand. 2) Even if you could, randomly sampling from a probability distribution will cause it to make stuff up unless you overfitted on the training data. An example that's come up in thread is ISBNs—there isn't going to be enough signal in the training set to reliably encode sufficiently high probability strings for all known ISBNs, so sometimes it will just string together likely numbers.
ahepp 2y ago
maybe hallucination is all cognition is, and humans are just really good at it?
- deleted 2y ago
  [deleted]
- theturtle32 2y ago
  In my experience, humans are at least as bad at it as GPT-4, if not far worse. In terms, specifically, of being "factually accurate" and grounded in absolute reality. Humans operate entirely in the probabilistic realm of what seems right to us based on how we were educated, the values we were raised with, our religious beliefs, etc. -- Human beings are all over the map with this.
  - ruthmarx 2y ago
    > In my experience, humans are at least as bad at it as GPT-4, if not far worse. I had an argument with a former friend recently, because he read some comments on YouTube and was convinced a racoon raped a cat and produced some kind of hybrid offspring that was terrorizing a neighborhood. Trying to explain that different species can't procreate like that resulted in him pointing to the fact that other people believed it in the comments as proof. Say what you will about LLMs, but they seem to have a better basic education than an awful lot of adults, and certainly significantly better basic reasoning capabilities.
    - nmeagent 2y ago
      > Trying to explain that different species can't procreate like that resulted in him pointing to the fact that other people believed it in the comments as proof. Those two species can't interbreed apparently, but considering the number of species that can [1] produce hybrid offspring, some even from different families, it is reasonable to forgive people for entertaining the possibility. [1] https://en.m.wikipedia.org/wiki/List_of_genetic_hybrids https://en.m.wikipedia.org/wiki/List_of_genetic_hybrids
nprateem 2y ago
Exactly. Even a broken clock is right twice a day.
malshe 2y ago
Excellent point. I did not think about hallucination in this manner before.

by wrs 2y ago

diputsmonro 2y ago

Exactly this, I've been saying this since the beginning. Every response is a hallucination - a probabilistic string of words divorced from any concept of truth or reality. By total coincidence, some hallucinations happen to reflect the truth, but only because the training data happened to generally be truthful sentences. Therefore, creating something that imitates a truthful sentence will often happen to also be truthful, but there is absolutely no guarantee or any function that even attempts to enforce that. All responses are hallucinations. Some hallucinations happen to overlap the truth.

fumeux_fume 2y ago
Ok, but I think it would be more productive to educate people that LLMs have no concept of truth rather than insist they use the term "hallucinate" in an unintuitive way.
- lolinder 2y ago
  I don't know about OP, but I'm suggesting that the term 'hallucinate' be abolished entirely as applies to LLMs, not redefined. It draws an arbitrary line in the middle of the set of problems that all amount to "how do we make sure that the output of an LLM is consistently acceptable" and will all be solved using the same techniques if at all.
- aeternum 2y ago
  LLMs do now have a concept of truth now since much of the RLHF is focused on making them more accurate and true. I think the problem is that humanity has a poor concept of truth. We think of most things as true or not true when much of our reality is uncertain due to fundamental limitations or because we often just don't know yet. During covid for example humanity collectively hallucinated the importance of disinfecting groceries for awhile.
  - jbm 2y ago
    > humanity collectively hallucinated the importance of disinfecting groceries for awhile I reject this history. I homeschooled my kids during covid due to uncertainty and even I didn't reach that level, and nor did anyone I knew in person. A very tiny number who were egged on by some YouTubers did this, including one person I knew remotely. Unsurprisingly that person was based in SV.
    - tsimionescu 2y ago
      It's not some extremist on YouTube, disinfecting your groceries was the official recommendation of many countries worldwide, including most of Europe. I couldn't say how many people actually followed the recommendation , but I would bet it's way more than a tiny number.
      - SirMaster 2y ago
        This is the first I’ve even heard of people disinfecting their groceries because of Covid. Honestly that sounds rather crazy to me.
        tsimionescu 2y ago
        There was a period near the start of the pandemic, especially while the medical establishment was trying to avoid ordinary people wearing masks in order to help stockpile them for high priority workers, when a lot of emphasis was put on surface contact. If it's extremely important to wear gloves and keep sanitizing your hands after touching every part of the supermarket, it stands to reason that you'd want to sanitize all of the outside packaging that others touched with their diseased hands as soon as you brought it into your house. Otherwise, you'd be expected to sanitize your hands every time you touched those items again, even at home, right? Of course, surface contact is actually a very minor avenue of infection, and pretty much limited to cases where someone has just sneezed or coughed on a surface that you are touching, and then putting your hand to your nose or maybe eyes or mouth soon after. So sanitizing groceries is essentially pointless, since it only slightly reduces an already very small risk.
- diputsmonro 2y ago
  If people already understand what "hallucination" means, then I think it's perfectly intuitive and educational to say that, actually, the LLM is always doing that, just that some of those hallucinations happen to coincidentally describe something real. We need to dispell the notion that the LLM "knows" the truth, or is "smart". It's just a fancy stochastic parrot. Whether it's responses reflect a truthful reality or a fantasy it made up is just luck, weighted by (but not constrained to) its training data. Emphasizing that everything is a hallucination does that. I purposefully want to reframe how the word is used and how we think about LLMs.
plaidfuji 2y ago
In other words, all models are wrong, but some are useful.
- diputsmonro 2y ago
  Precisely, I'm glad someone picked up on the reference!
TeMPOraL 2y ago
I think you're going too far here. > By total coincidence, some hallucinations happen to reflect the truth, but only because the training data happened to generally be truthful sentences. It's not a "total coincidence". It's the default. Thus, the model's responses aren't "divorced from any concept of truth or reality" - the whole distribution from which those responses are pulled is strongly aligned with reality. (Which is why people started using the term "hallucinations" to describe the failure mode, instead of "fishing a coherent and true sentence out of line noise" to describe the success mode - because success mode dominates.) Humans didn't invent language for no reason. They don't communicate to entertain themselves with meaningless noises. Most of communication - whether spoken or written - is deeply connected to reality. Language itself is deeply connected to reality. Even the most blatant lies, even all of fiction writing, they're all incorrect or fabricated only at the surface level - the whole thing, accounting for the utterance, what it is about, the meanings, the words, the grammar - is strongly correlated with truth and reality. So there's absolutely no coincidence that LLMs get things right more often than not. Truth is thoroughly baked into the training data, simply because it's a data set of real human communication, instead of randomly generated sentences.
- MichaelZuo 2y ago
  Strongly correlated != 100% overlap A 99% overlap can still be coincidence. And even if it was ‘absolutely no coincidence’, that is still only reflective of the reality as perceived by the average of all the people from the training set.
- zdragnar 2y ago
  The problem - as defined by how end users understand it - is that the model itself doesn't know the difference, and will proclaim bullshit with the same level of confidence that it does accurate information. That's how you end up with grocery store chatbots recommending mixing ammonia and bleach for a cocktail, or lawyers using chatbots to cite entirely fictional case law before a judge in court. Nothing that comes out of an LLM can be implicitly trusted, so your default assumption must be that everything it gives you needs verification from another source. Telling people "the truth is baked in" is just begging for a disaster.
codr7 2y ago
So much truth here, very refreshing to see! About time too, the sooner we can stop the madness the better, building a society on top of this technology is a movie I'd rather not see.
SirMaster 2y ago
Maybe they shouldn’t have mixed truthful data with obviously untruthful data in the same training data set? Why not make a model only from truthful data? Like exclude all fiction for example.
- vanviegen 2y ago
  That wouldn't prevent hallucination. An LLM doesn't know what it doesn't know. It will always try to come up with a response that sounds plausible, based on its knowledge or lack thereof.
- lolinder 2y ago
  1) It's impossible to get enough data to train one of these well while also curating it by hand. 2) Even if you could, randomly sampling from a probability distribution will cause it to make stuff up unless you overfitted on the training data. An example that's come up in thread is ISBNs—there isn't going to be enough signal in the training set to reliably encode sufficiently high probability strings for all known ISBNs, so sometimes it will just string together likely numbers.

ahepp 2y ago

maybe hallucination is all cognition is, and humans are just really good at it?

deleted 2y ago
[deleted]
theturtle32 2y ago
In my experience, humans are at least as bad at it as GPT-4, if not far worse. In terms, specifically, of being "factually accurate" and grounded in absolute reality. Humans operate entirely in the probabilistic realm of what seems right to us based on how we were educated, the values we were raised with, our religious beliefs, etc. -- Human beings are all over the map with this.
- ruthmarx 2y ago
  > In my experience, humans are at least as bad at it as GPT-4, if not far worse. I had an argument with a former friend recently, because he read some comments on YouTube and was convinced a racoon raped a cat and produced some kind of hybrid offspring that was terrorizing a neighborhood. Trying to explain that different species can't procreate like that resulted in him pointing to the fact that other people believed it in the comments as proof. Say what you will about LLMs, but they seem to have a better basic education than an awful lot of adults, and certainly significantly better basic reasoning capabilities.
  - nmeagent 2y ago
    > Trying to explain that different species can't procreate like that resulted in him pointing to the fact that other people believed it in the comments as proof. Those two species can't interbreed apparently, but considering the number of species that can [1] produce hybrid offspring, some even from different families, it is reasonable to forgive people for entertaining the possibility. [1] https://en.m.wikipedia.org/wiki/List_of_genetic_hybrids https://en.m.wikipedia.org/wiki/List_of_genetic_hybrids

nprateem 2y ago

Exactly. Even a broken clock is right twice a day.

malshe 2y ago

Excellent point. I did not think about hallucination in this manner before.