5 ms·

Fowler is not really redefining "hallucination." He's using a form of irony that emphasises how fundamental "hallucinations" are to the operation of the system.

Fowler is not really redefining "hallucination." He's using a form of irony that emphasises how fundamental "hallucinations" are to the operation of the system. One might also say "you can't get rid of collateral damage from bombs. Indeed, collateral damage is the feature, it's just some of that is what we want to blow up." You're not meant to take it literally.

xyzzy123 1y ago
You might as well say it's interpolating or extrapolating. That's what people are usually doing too, even when recalling situations that they were personally involved in. I think we call it "hallucinating" when the machine does this in an un-human-like way.
- dapperdrake 1y ago
  The longer term for this is "stochastic parrot". See another HN comment here comparing LLMs to theater actors or movie actors. LLMs just spew words. It just so happens that human beings can decode them into something related, useful, and meaningful surprisingly often. Might even be a useful case of pareidolia (a term I dislike, because a world without any pattern matching whatsoever would not necessarily be "better").
  - jychang 1y ago
    I dislike the term “stochastic parrot”, because there’s plenty of evidence that LLMs do have an understanding of at least some things that they are saying. We can trace which neurons activate for a face recognition model and see that a certain neuron does light up when it sees a face. The correct features are active for the sentence “the word parrots is plural”. If you stop assuming the LLMs have no internal representations of the data, then everything makes a lot more sense! The LLM is FORCED to answer questions… just like a high school student filling out the SAT is forced to answer questions. If a high school student fills out the wrong answer on the SAT, is that a hallucination? Hallucinations are expected behaviors if you RLHF a high schooler to always guess an answer on the SAT because that’ll get them the highest score. This applies to ML model reward functions as well.
    - ACCount37 1y ago
      This matches what we know about LLMs and hallucination-avoidance behavior in LLMs. "Wrong answers on SAT" is also the leading hypothesis on why o3 was such an outlier - far more prone to hallucinations than either prior or following OpenAI models. On SAT, giving a random answer is right 20% of the time - more if you ruled at least one obviously wrong answer out. Saying "I don't know" and not answering is right 0% of the time. So if you RLVR on SAT-type tests, where any answer is better than no answer, you encourage hallucinations. Hallucination avoidance in LLMs is a fragile capability, and OpenAI has probably fried its o3 with too much careless RLVR. But another cause of hallucinations is limited self-awareness of modern LLMs. And I mean "self-awareness" in a very mechanical, no-nonsense fashion: "has information about itself and its own capabilities". LLMs have very little of that. Humans have some awareness of the limits of their knowledge - not at all perfect, but at least there's something. LLMs get much, much less of that. LLMs learn the bulk of their knowledge from pre-training data, but pre-training doesn't teach them a lot about where the limits of their knowledge lie.
      - neonspark 1y ago
        > But another cause of hallucinations is limited self-awareness of modern LLMs… Humans have some awareness of the limits of their knowledge Until you said that I didn’t realize just how much humans “hallucinate“ in just the same ways that AI does. I have a friend who is fluent in Spanish, a native speaker, but got a pretty weak grammar education when he was in high school. Also, he got no education at all in Critical thinking, at least not formally. So this guy is really, really fluent in his native language, but can often have a very difficult time explaining why he uses whatever grammar he uses. I think the whole world is realizing how little our brains can correctly explain and identify the grammar we use flawlessly. He helps me to improve my Spanish a lot, he can correct me with 100% accuracy of course, but I’ve noticed on many occasions, including this week, that when I ask a question about why he said something one way or another in Spanish, he will just make up some grammar rule that doesn’t actually exist, and is in fact not true. He said something like “you say it this way when you really know the person and you’re saying that the other way when it’s more formal“, but I think really it was just a slangy way to mis-stress something and it didn’t have to do with familiar/formal or not. I’ve learned not to challenge him on any of these grammar rules that he makes up, because he will dig his heels in, and I’ve learned just to ignore him because he won’t have remembered this made up grammar rule in a week anyway. This really feels like a very tight analogy with what my LLM does to me every day, except that when I challenge the LLM it will profusely apologize and declare itself incorrect even if it had been correct after all. Maybe LLMs are a little bit too humble. I imagine this is a very natural tendency in humans, and I imagine I do it much more than I’m aware of. So how do humans use self-awareness to reduce the odds of this happening? I think we mostly get trained in higher education to not trust the first thought that comes into our head, even if it feels self consistent and correct. We eventually learn to say “I don’t know” even if it’s about something that we are very, very good at.
  - aatd86 1y ago
    But..but...humans do the same thing? This is input/output with the output being formed from applying the function of our life experiences (training) to the input? They just don't have the hormonal circuitry to optimise for whatever our bodies are trying to optimise for when we take decisions.
    - cmiles74 1y ago
      People actually understand the input and the output. An LLM understand neither, it's generating output that is statistically likely, within some bounds. As Fowler said, it's a pleasant coincidence that some of this output has value to us. (For sure, arguments can be made that the relationship betweens the terms that the model has encoded could maybe be called "model thinking" or "model understanding" but it's not how people work.)
      - seunosewa 1y ago
        We do the same thing. We pick words that are statistically likely to get us what we want. And much of it is unconscious. You don't formally reason about every word you speak. You are focused on your objective, and your brain fills in the gaps.
        scott_w 1y ago
        We absolutely do not "pick words that are statistically likely to get us what we want." We use words to try to articulate (to varying levels of success) a message that we want to communicate. The words, tone, speed, pitch, etc. all convey meaning. > And much of it is unconscious. That does not mean we're "picking words statistically likely to get us what we want," it means "our brains do a lot of work subconsciously." Nothing more. > You are focused on your objective, and your brain fills in the gaps. This is a total contradiction of what you said at the start. LLMs are not focused on an objective, they are using very complex statistical algorithms to determine output strings. There is no objective to an LLM's output.
        aatd86 1y ago
  - naasking 1y ago
    > LLMs just spew words. It just so happens that human beings can decode them into something related, useful, and meaningful surprisingly often. This sentence is inherently contradictory. If LLM output is meaningful more than chance, then it's literally not "just spewing words". Therefore whatever model it is using to generate that meaning must contain some semantic content, even if it's not semantic content that's as rich as humans are capable of. The "stochastic parrot" term is thus silly.
    - thrawa8387336 1y ago
      It's a sufficiently large N shannonizer. Nothing more
      - jonnycomputer 1y ago
        so are we, maybe.
        naasking 1y ago
        Definitely true at some level: https://en.wikipedia.org/wiki/Predictive_coding https://en.wikipedia.org/wiki/Predictive_coding

xyzzy123 1y ago

You might as well say it's interpolating or extrapolating. That's what people are usually doing too, even when recalling situations that they were personally involved in. I think we call it "hallucinating" when the machine does this in an un-human-like way.

dapperdrake 1y ago
The longer term for this is "stochastic parrot". See another HN comment here comparing LLMs to theater actors or movie actors. LLMs just spew words. It just so happens that human beings can decode them into something related, useful, and meaningful surprisingly often. Might even be a useful case of pareidolia (a term I dislike, because a world without any pattern matching whatsoever would not necessarily be "better").
- jychang 1y ago
  I dislike the term “stochastic parrot”, because there’s plenty of evidence that LLMs do have an understanding of at least some things that they are saying. We can trace which neurons activate for a face recognition model and see that a certain neuron does light up when it sees a face. The correct features are active for the sentence “the word parrots is plural”. If you stop assuming the LLMs have no internal representations of the data, then everything makes a lot more sense! The LLM is FORCED to answer questions… just like a high school student filling out the SAT is forced to answer questions. If a high school student fills out the wrong answer on the SAT, is that a hallucination? Hallucinations are expected behaviors if you RLHF a high schooler to always guess an answer on the SAT because that’ll get them the highest score. This applies to ML model reward functions as well.
  - ACCount37 1y ago
    This matches what we know about LLMs and hallucination-avoidance behavior in LLMs. "Wrong answers on SAT" is also the leading hypothesis on why o3 was such an outlier - far more prone to hallucinations than either prior or following OpenAI models. On SAT, giving a random answer is right 20% of the time - more if you ruled at least one obviously wrong answer out. Saying "I don't know" and not answering is right 0% of the time. So if you RLVR on SAT-type tests, where any answer is better than no answer, you encourage hallucinations. Hallucination avoidance in LLMs is a fragile capability, and OpenAI has probably fried its o3 with too much careless RLVR. But another cause of hallucinations is limited self-awareness of modern LLMs. And I mean "self-awareness" in a very mechanical, no-nonsense fashion: "has information about itself and its own capabilities". LLMs have very little of that. Humans have some awareness of the limits of their knowledge - not at all perfect, but at least there's something. LLMs get much, much less of that. LLMs learn the bulk of their knowledge from pre-training data, but pre-training doesn't teach them a lot about where the limits of their knowledge lie.
    - neonspark 1y ago
      > But another cause of hallucinations is limited self-awareness of modern LLMs… Humans have some awareness of the limits of their knowledge Until you said that I didn’t realize just how much humans “hallucinate“ in just the same ways that AI does. I have a friend who is fluent in Spanish, a native speaker, but got a pretty weak grammar education when he was in high school. Also, he got no education at all in Critical thinking, at least not formally. So this guy is really, really fluent in his native language, but can often have a very difficult time explaining why he uses whatever grammar he uses. I think the whole world is realizing how little our brains can correctly explain and identify the grammar we use flawlessly. He helps me to improve my Spanish a lot, he can correct me with 100% accuracy of course, but I’ve noticed on many occasions, including this week, that when I ask a question about why he said something one way or another in Spanish, he will just make up some grammar rule that doesn’t actually exist, and is in fact not true. He said something like “you say it this way when you really know the person and you’re saying that the other way when it’s more formal“, but I think really it was just a slangy way to mis-stress something and it didn’t have to do with familiar/formal or not. I’ve learned not to challenge him on any of these grammar rules that he makes up, because he will dig his heels in, and I’ve learned just to ignore him because he won’t have remembered this made up grammar rule in a week anyway. This really feels like a very tight analogy with what my LLM does to me every day, except that when I challenge the LLM it will profusely apologize and declare itself incorrect even if it had been correct after all. Maybe LLMs are a little bit too humble. I imagine this is a very natural tendency in humans, and I imagine I do it much more than I’m aware of. So how do humans use self-awareness to reduce the odds of this happening? I think we mostly get trained in higher education to not trust the first thought that comes into our head, even if it feels self consistent and correct. We eventually learn to say “I don’t know” even if it’s about something that we are very, very good at.
- aatd86 1y ago
  But..but...humans do the same thing? This is input/output with the output being formed from applying the function of our life experiences (training) to the input? They just don't have the hormonal circuitry to optimise for whatever our bodies are trying to optimise for when we take decisions.
  - cmiles74 1y ago
    People actually understand the input and the output. An LLM understand neither, it's generating output that is statistically likely, within some bounds. As Fowler said, it's a pleasant coincidence that some of this output has value to us. (For sure, arguments can be made that the relationship betweens the terms that the model has encoded could maybe be called "model thinking" or "model understanding" but it's not how people work.)
    - seunosewa 1y ago
      We do the same thing. We pick words that are statistically likely to get us what we want. And much of it is unconscious. You don't formally reason about every word you speak. You are focused on your objective, and your brain fills in the gaps.
      - scott_w 1y ago
        We absolutely do not "pick words that are statistically likely to get us what we want." We use words to try to articulate (to varying levels of success) a message that we want to communicate. The words, tone, speed, pitch, etc. all convey meaning. > And much of it is unconscious. That does not mean we're "picking words statistically likely to get us what we want," it means "our brains do a lot of work subconsciously." Nothing more. > You are focused on your objective, and your brain fills in the gaps. This is a total contradiction of what you said at the start. LLMs are not focused on an objective, they are using very complex statistical algorithms to determine output strings. There is no objective to an LLM's output.
        aatd86 1y ago
- naasking 1y ago
  > LLMs just spew words. It just so happens that human beings can decode them into something related, useful, and meaningful surprisingly often. This sentence is inherently contradictory. If LLM output is meaningful more than chance, then it's literally not "just spewing words". Therefore whatever model it is using to generate that meaning must contain some semantic content, even if it's not semantic content that's as rich as humans are capable of. The "stochastic parrot" term is thus silly.
  - thrawa8387336 1y ago
    It's a sufficiently large N shannonizer. Nothing more
    - jonnycomputer 1y ago
      so are we, maybe.
      - naasking 1y ago
        Definitely true at some level: https://en.wikipedia.org/wiki/Predictive_coding https://en.wikipedia.org/wiki/Predictive_coding