4 ms·

> My former colleague Rebecca Parsons, has been saying for a long time that hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature

> My former colleague Rebecca Parsons, has been saying for a long time that hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful. This is an example of my least favorite style of feigned insight: redefining a term into meaninglessness just so you can say something that sounds different while not actually saying anything new. Yes, if you redefine "hallucination" from "produce output containing detailed information despite that information not being grounded in external reality, in a manner distantly analogous to a human reporting sense data produced by a literal hallucination rather than the external inputs that are presumed normally to ground sense data" to "produce output", its true that all LLMs do is "hallucinate", and that "hallucinating" is not a undesirable behavior. But you haven't said anything new about the thing that was called "hallucination" by everyone else, or about the thing--LLM output in general--that you have called "hallucination". Everyone already knew that producing output wasn't undesirable. You've just taken the label conventionally attached to a bad behavior, attached it to a broader category that includes all behavior, and used the power of equivocation to make something that sounds novel without saying anything new.

scott_w 1y ago
Fowler is not really redefining "hallucination." He's using a form of irony that emphasises how fundamental "hallucinations" are to the operation of the system. One might also say "you can't get rid of collateral damage from bombs. Indeed, collateral damage is the feature, it's just some of that is what we want to blow up." You're not meant to take it literally.
- xyzzy123 1y ago
  You might as well say it's interpolating or extrapolating. That's what people are usually doing too, even when recalling situations that they were personally involved in. I think we call it "hallucinating" when the machine does this in an un-human-like way.
  - dapperdrake 1y ago
    The longer term for this is "stochastic parrot". See another HN comment here comparing LLMs to theater actors or movie actors. LLMs just spew words. It just so happens that human beings can decode them into something related, useful, and meaningful surprisingly often. Might even be a useful case of pareidolia (a term I dislike, because a world without any pattern matching whatsoever would not necessarily be "better").
    - jychang 1y ago
      I dislike the term “stochastic parrot”, because there’s plenty of evidence that LLMs do have an understanding of at least some things that they are saying. We can trace which neurons activate for a face recognition model and see that a certain neuron does light up when it sees a face. The correct features are active for the sentence “the word parrots is plural”. If you stop assuming the LLMs have no internal representations of the data, then everything makes a lot more sense! The LLM is FORCED to answer questions… just like a high school student filling out the SAT is forced to answer questions. If a high school student fills out the wrong answer on the SAT, is that a hallucination? Hallucinations are expected behaviors if you RLHF a high schooler to always guess an answer on the SAT because that’ll get them the highest score. This applies to ML model reward functions as well.
      - ACCount37 1y ago
        This matches what we know about LLMs and hallucination-avoidance behavior in LLMs. "Wrong answers on SAT" is also the leading hypothesis on why o3 was such an outlier - far more prone to hallucinations than either prior or following OpenAI models. On SAT, giving a random answer is right 20% of the time - more if you ruled at least one obviously wrong answer out. Saying "I don't know" and not answering is right 0% of the time. So if you RLVR on SAT-type tests, where any answer is better than no answer, you encourage hallucinations. Hallucination avoidance in LLMs is a fragile capability, and OpenAI has probably fried its o3 with too much careless RLVR. But another cause of hallucinations is limited self-awareness of modern LLMs. And I mean "self-awareness" in a very mechanical, no-nonsense fashion: "has information about itself and its own capabilities". LLMs have very little of that. Humans have some awareness of the limits of their knowledge - not at all perfect, but at least there's something. LLMs get much, much less of that. LLMs learn the bulk of their knowledge from pre-training data, but pre-training doesn't teach them a lot about where the limits of their knowledge lie.
bartread 1y ago
I’ve never liked that this behaviour is described using the term “hallucination”. If a human being talked confidently about something that they were just making up out of thin air by synthesizing based (consciously or unconsciously) on other information they know you wouldn’t call it “hallucination”: you’d call it “bullshit”. And, honestly, “bullshit” is a much more helpful way of thinking about this behaviour because it somewhat nullifies the arguments people make against the use of LLMs due to this behaviour. Fundamentally, if you don’t want to work with LLMs because they sometimes “bullshit”, are you planning on no longer working with human beings as well? It doesn’t hold up. But, more than that, going back to your point: it’s much harder to redefine the term “bullshit” to mean something different to the common understanding. All of that said, I don’t mind the piece and, honestly, the “I haven’t the foggiest” comment about the future of software development as a career is well made. I guess it’s just a somewhat useful collection of scattered thoughts on LLMs and, as such, an example of a piece where the “thoughts on” title fits well. I don’t think the author is trying to be particularly authoritative.
- dragonwriter 1y ago
  > I’ve never liked that this behaviour is described using the term “hallucination”. I have a standard canned rant about "confabulation" is a much better metaphor, but it wasn't the point I was focussed on here. > Fundamentally, if you don’t want to work with LLMs because they sometimes “bullshit”, are you planning on no longer working with human beings as well? I will very much not voluntarily rely on a human for particular tasks if that human has demonstrated a pattern of bullshitting me when given that kind of task, yes, especially if, on top of the opportunity cost inherent in relying on a person for a particular task, I am also required to compensate them—e.g., financially—for their notional attention to the task.
- movpasd 1y ago
  At the risk of sounding woo, I find some parallels in how LLMs work to my experiences with meditation and writing. My subjective experience of it is that there is some unconscious part of my brain that supplies a scattered stream of words as the sentence forms --- without knowing the neuroscience of it, I could speculate it is a "neurological transformer", some statistical model that has memorised a combination of the grammar and contextual semantic meaning of language. The difference is that the LLM is _only that part_. In producing language as a human, I filter these words, I go back and think of new phrasings, I iterate --- in writing consciously, in speech unconsciously. So rather than a sequence it is a scattered tree filled with rhetorical dead ends, pruned through interaction with my world-model and other intellectual faculties. You can pull on one thread of words as though it were fully-formed already as a kind of Surrealist exercise (like a one-person cadavre exquis), and the result feels similar to an LLM with the temperature turned up too high. But if nothing else, this highlights to me how easily the process of word generation may be decoupled from meaning. And it serves to explain another kind of common human experience, which feels terribly similar to the phenomenon of LLM hallucination: the "word vomit" of social anxiety. In this process it suddenly becomes less important that the words you produce are anchored to truth, and instead the language-system becomes tuned to produce any socially plausible output at all. That seems to me to be the most apt analogy.
ljm 1y ago
If everything is a hallucination then nothing is a hallucination. Thanks for listening to my Ted Talk.
belter 1y ago
> You've just taken the label conventionally attached to a bad behavior, attached it to a broader category that includes all behavior, and used the power of equivocation to make something that sounds novel without saying anything new. You mean like every evangelist says AI changes everything every 5 min...When in reality what they mean is neural nets statistical code generators are getting pretty good? because that is almost all AI there is at the moment? Just to make the current AI sound bigger than it is?
deleted 1y ago
[deleted]
cess11 1y ago
The point, though awkwardly stated, is that there is no difference between 'hallucination' output and any other output from these compressed databases, just like there is no such difference between two queries on a typical RDBMS. It's a good point.
- dragonwriter 1y ago
  > The point, though awkwardly stated, is that there is no difference between 'hallucination' output and any other output from these compressed databases, But there is, except when you redefine "hallucination" so there isn't. And, when you retain the definition where there is a difference, you find there are techniques by which you can reduce hallucinations, which is important and useful. Changing the definition to eliminate the distinction is actively harmful to understanding and productive use of LLMs, for the benefit of making what superficially seems like an insightful comment.
  - cess11 1y ago
    Unless you are willing to use the same term when you botch an SQL query and don't get the result you want, don't use it at all.
sceptic123 1y ago
I actually found that comment interesting. It's pointing towards something I've struggled with around LLMs. They are (currently) incapable of knowing if what they output is correct, so the idea that "it's all hallucinations" acknowledges that point and gives useful context for anyone using LLMs for software development.
- Eridrus 1y ago
  Humans are also incapable of knowing whether their output is correct. We merely convince ourselves that it is and then put our thoughts in contact with the external world and other people to see if we actually are.
  - jimbokun 1y ago
    So then we are capable of knowing whether our output is correct, by putting it into contact with the external world.
  - dghlsakjg 1y ago
    Humans are capable of using formal logic to reach a provably correct conclusion from valid premises. That conclusion can then be used as premises for a further conclusion. Follow the logic the other way and using our powers of observation, and sensing, we can gather base level truths and know that what we are saying is correct in an absolute sense. Computers use formal logic all the time to output truth, LLMs, however, do not.
thwarted 1y ago
It's meant to contrast/correct the claim that "LLMs produce either truth or they produce hallucinations." Claims worded like that give the impression that if we can just reduce/eliminate the hallucinations, all that will remain will be the truth. But that's not the case. What is the case is that all the output is the same thing, hallucination, and that some of those hallucinations just so happen to reflect reality (or expectations) so appears to embody truth. It's like rolling a die and wanting one to come up, and when it does saying "the die knew what I wanted". An infinite number of monkeys typing on an infinite number of typewriters will eventually produce the script for Hamlet. That doesn't mean the monkeys know about Shakespeare, or Hamlet, or even words, or even what they are doing being chained to typewriters. We've found a way to optimize the infinite typing monkeys to output something that passes for Hamlet much sooner than infinity.
deleted 1y ago
[deleted]
redditor98654 1y ago
The way you have expressed this, I am borrowing it for myself. Many times i run into these kind of situations and I fail to explain why doing something like this is frustrating and actually useless. Thank you.

scott_w 1y ago

Fowler is not really redefining "hallucination." He's using a form of irony that emphasises how fundamental "hallucinations" are to the operation of the system. One might also say "you can't get rid of collateral damage from bombs. Indeed, collateral damage is the feature, it's just some of that is what we want to blow up." You're not meant to take it literally.

xyzzy123 1y ago
You might as well say it's interpolating or extrapolating. That's what people are usually doing too, even when recalling situations that they were personally involved in. I think we call it "hallucinating" when the machine does this in an un-human-like way.
- dapperdrake 1y ago
  The longer term for this is "stochastic parrot". See another HN comment here comparing LLMs to theater actors or movie actors. LLMs just spew words. It just so happens that human beings can decode them into something related, useful, and meaningful surprisingly often. Might even be a useful case of pareidolia (a term I dislike, because a world without any pattern matching whatsoever would not necessarily be "better").
  - jychang 1y ago
    I dislike the term “stochastic parrot”, because there’s plenty of evidence that LLMs do have an understanding of at least some things that they are saying. We can trace which neurons activate for a face recognition model and see that a certain neuron does light up when it sees a face. The correct features are active for the sentence “the word parrots is plural”. If you stop assuming the LLMs have no internal representations of the data, then everything makes a lot more sense! The LLM is FORCED to answer questions… just like a high school student filling out the SAT is forced to answer questions. If a high school student fills out the wrong answer on the SAT, is that a hallucination? Hallucinations are expected behaviors if you RLHF a high schooler to always guess an answer on the SAT because that’ll get them the highest score. This applies to ML model reward functions as well.
    - ACCount37 1y ago
      This matches what we know about LLMs and hallucination-avoidance behavior in LLMs. "Wrong answers on SAT" is also the leading hypothesis on why o3 was such an outlier - far more prone to hallucinations than either prior or following OpenAI models. On SAT, giving a random answer is right 20% of the time - more if you ruled at least one obviously wrong answer out. Saying "I don't know" and not answering is right 0% of the time. So if you RLVR on SAT-type tests, where any answer is better than no answer, you encourage hallucinations. Hallucination avoidance in LLMs is a fragile capability, and OpenAI has probably fried its o3 with too much careless RLVR. But another cause of hallucinations is limited self-awareness of modern LLMs. And I mean "self-awareness" in a very mechanical, no-nonsense fashion: "has information about itself and its own capabilities". LLMs have very little of that. Humans have some awareness of the limits of their knowledge - not at all perfect, but at least there's something. LLMs get much, much less of that. LLMs learn the bulk of their knowledge from pre-training data, but pre-training doesn't teach them a lot about where the limits of their knowledge lie.

bartread 1y ago

I’ve never liked that this behaviour is described using the term “hallucination”. If a human being talked confidently about something that they were just making up out of thin air by synthesizing based (consciously or unconsciously) on other information they know you wouldn’t call it “hallucination”: you’d call it “bullshit”. And, honestly, “bullshit” is a much more helpful way of thinking about this behaviour because it somewhat nullifies the arguments people make against the use of LLMs due to this behaviour. Fundamentally, if you don’t want to work with LLMs because they sometimes “bullshit”, are you planning on no longer working with human beings as well? It doesn’t hold up. But, more than that, going back to your point: it’s much harder to redefine the term “bullshit” to mean something different to the common understanding. All of that said, I don’t mind the piece and, honestly, the “I haven’t the foggiest” comment about the future of software development as a career is well made. I guess it’s just a somewhat useful collection of scattered thoughts on LLMs and, as such, an example of a piece where the “thoughts on” title fits well. I don’t think the author is trying to be particularly authoritative.

dragonwriter 1y ago
> I’ve never liked that this behaviour is described using the term “hallucination”. I have a standard canned rant about "confabulation" is a much better metaphor, but it wasn't the point I was focussed on here. > Fundamentally, if you don’t want to work with LLMs because they sometimes “bullshit”, are you planning on no longer working with human beings as well? I will very much not voluntarily rely on a human for particular tasks if that human has demonstrated a pattern of bullshitting me when given that kind of task, yes, especially if, on top of the opportunity cost inherent in relying on a person for a particular task, I am also required to compensate them—e.g., financially—for their notional attention to the task.
movpasd 1y ago
At the risk of sounding woo, I find some parallels in how LLMs work to my experiences with meditation and writing. My subjective experience of it is that there is some unconscious part of my brain that supplies a scattered stream of words as the sentence forms --- without knowing the neuroscience of it, I could speculate it is a "neurological transformer", some statistical model that has memorised a combination of the grammar and contextual semantic meaning of language. The difference is that the LLM is _only that part_. In producing language as a human, I filter these words, I go back and think of new phrasings, I iterate --- in writing consciously, in speech unconsciously. So rather than a sequence it is a scattered tree filled with rhetorical dead ends, pruned through interaction with my world-model and other intellectual faculties. You can pull on one thread of words as though it were fully-formed already as a kind of Surrealist exercise (like a one-person cadavre exquis), and the result feels similar to an LLM with the temperature turned up too high. But if nothing else, this highlights to me how easily the process of word generation may be decoupled from meaning. And it serves to explain another kind of common human experience, which feels terribly similar to the phenomenon of LLM hallucination: the "word vomit" of social anxiety. In this process it suddenly becomes less important that the words you produce are anchored to truth, and instead the language-system becomes tuned to produce any socially plausible output at all. That seems to me to be the most apt analogy.

ljm 1y ago

If everything is a hallucination then nothing is a hallucination. Thanks for listening to my Ted Talk.

belter 1y ago

> You've just taken the label conventionally attached to a bad behavior, attached it to a broader category that includes all behavior, and used the power of equivocation to make something that sounds novel without saying anything new. You mean like every evangelist says AI changes everything every 5 min...When in reality what they mean is neural nets statistical code generators are getting pretty good? because that is almost all AI there is at the moment? Just to make the current AI sound bigger than it is?

deleted 1y ago

[deleted]

cess11 1y ago

The point, though awkwardly stated, is that there is no difference between 'hallucination' output and any other output from these compressed databases, just like there is no such difference between two queries on a typical RDBMS. It's a good point.

dragonwriter 1y ago
> The point, though awkwardly stated, is that there is no difference between 'hallucination' output and any other output from these compressed databases, But there is, except when you redefine "hallucination" so there isn't. And, when you retain the definition where there is a difference, you find there are techniques by which you can reduce hallucinations, which is important and useful. Changing the definition to eliminate the distinction is actively harmful to understanding and productive use of LLMs, for the benefit of making what superficially seems like an insightful comment.
- cess11 1y ago
  Unless you are willing to use the same term when you botch an SQL query and don't get the result you want, don't use it at all.

sceptic123 1y ago

I actually found that comment interesting. It's pointing towards something I've struggled with around LLMs. They are (currently) incapable of knowing if what they output is correct, so the idea that "it's all hallucinations" acknowledges that point and gives useful context for anyone using LLMs for software development.

Eridrus 1y ago
Humans are also incapable of knowing whether their output is correct. We merely convince ourselves that it is and then put our thoughts in contact with the external world and other people to see if we actually are.
- jimbokun 1y ago
  So then we are capable of knowing whether our output is correct, by putting it into contact with the external world.
- dghlsakjg 1y ago
  Humans are capable of using formal logic to reach a provably correct conclusion from valid premises. That conclusion can then be used as premises for a further conclusion. Follow the logic the other way and using our powers of observation, and sensing, we can gather base level truths and know that what we are saying is correct in an absolute sense. Computers use formal logic all the time to output truth, LLMs, however, do not.

thwarted 1y ago

It's meant to contrast/correct the claim that "LLMs produce either truth or they produce hallucinations." Claims worded like that give the impression that if we can just reduce/eliminate the hallucinations, all that will remain will be the truth. But that's not the case. What is the case is that all the output is the same thing, hallucination, and that some of those hallucinations just so happen to reflect reality (or expectations) so appears to embody truth. It's like rolling a die and wanting one to come up, and when it does saying "the die knew what I wanted". An infinite number of monkeys typing on an infinite number of typewriters will eventually produce the script for Hamlet. That doesn't mean the monkeys know about Shakespeare, or Hamlet, or even words, or even what they are doing being chained to typewriters. We've found a way to optimize the infinite typing monkeys to output something that passes for Hamlet much sooner than infinity.

deleted 1y ago

[deleted]

redditor98654 1y ago

The way you have expressed this, I am borrowing it for myself. Many times i run into these kind of situations and I fail to explain why doing something like this is frustrating and actually useless. Thank you.