5 ms·

> hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some o

> hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful. Nice.

tptacek 1y ago
In that framing, you can look at an agent as simply a filter on those hallucinations.
- th0ma5 1y ago
  Yes yes, with yet to be discovered holes
- Lionga 1y ago
  Isn't an "agent" not just hallucinations layered on top of other random hallucinations to create new hallucinations?
  - tptacek 1y ago
    No, that's exactly what an agent isn't. What makes an agent an agent is all the not-LLM code. When an agent generates Golang code, it runs the Go compiler, which is in the agent's architecture an extension of the agent. The Go compiler does not hallucinate.
    - Lionga 1y ago
      The most common "agent" is an letting an LLM run a while loop (“multi-step agent”) [1] [1] https://huggingface.co/docs/smolagents/conceptual_guides/intro_agents https://huggingface.co/docs/smolagents/conceptual_guides/int...
      - tptacek 1y ago
        That's not how Claude Code works (or Gemini, Cursor, or Codex).
- armchairhacker 1y ago
  This vaguely relates to a theory about human thought: that our subconscious constantly comes up with random ideas, then filters the unreasonable ones, but in people with delusions (e.g. schizophrenia) the filter is broken. Salience (https://en.wikipedia.org/wiki/Salience_(neuroscience) https://en.wikipedia.org/wiki/Salience_(neuroscience)), "the property by which some thing stands out", is something LLMs have trouble with. Probably because they're trained on human text, which ranges from accurate descriptions of reality to nonsense.
- keeda 1y ago
  More of a error-correcting feedback loop rather than a filter, really. Which is very much what we do as humans, apparently. One recent theory of neuroscience that is becoming influential is Predictive Processing --https://en.wikipedia.org/wiki/Predictive_coding https://en.wikipedia.org/wiki/Predictive_coding -- this postulates that we also constantly generate a "mental model" of our environment (a literal "prediction") and use sensory inputs to correct and update it. So the only real difference between "perception" and a "hallucination" is whether it is supported by physical reality.
  - jonoc 1y ago
    thats a fascinating way to put it
  - sethammons 1y ago
    You ever see something out of the corner of your eye and you were mistaken? It feels like LLM hallucinations. "An orange cat on my counter?!" And an instant later, your brain has reclassified "a basketball on my counter" as that fits the environment model better as several instant-observations gather contexts: not moving, more round, not furry, I don't own a cat, yesterday my kid mentioned something about tryouts, boop insta-reclassification from cat to basketball. I can recognize my own meta cognition there. My model of reality course corrects the information feed interpretation on the fly. Optical illusions feel very similar whereby the inner reality model clashes with the observed. For general ai, it needs a world model that can be tested against and surprise is noted and models are updated. Looping llm output with test cases is a crude approximation of that world model.
    - osullivj 1y ago
      I leaned heavily on my own meta cognition recently when withdrawing from a bereavment benzo habit recently. The combo of flu symptoms, anxiety and hallucinations are fierce. I knew O was seeing things that were not real; light fittings turning into a rotating stained glass slideshow. So I'm totally on board with the visual model hypothesis. My own speculation is that audio perception is less predictive as audio structure persists deeper into my own DMT sessions than does vision, where perspective quickly collapses and vision becomes kaleidoscopic. Which may be a return to the vision we had as newborns. Maybe normal vision is only attained via socialisation?
ninetyninenine 1y ago
Nah I don't agree with this characterization. The problem is, the majority of those hallucinations are true. What was said would make more sense if the majority of the responses were, in fact, false, but this is not the case.
- deleted 1y ago
  [deleted]
- xmprt 1y ago
  I think you're both correct but have different definitions of hallucinations. You're judging it as a hallucination based on the veracity of the output. Whereas Fowler is judging it based on the method by which the output is achieved. By that judgement, everything is a hallucination because the user cannot differentiate between when the LLM is telling the truth and isn't. This is different from human hallucinations where it makes something up because of something wrong with the mind rather than some underlying issue with the brain's architecture.
  - ants_everywhere 1y ago
    an LLM hallucination is defined by its truth > In the field of artificial intelligence (AI), a hallucination or artificial hallucination (also called confabulation,[1] or delusion)[2] is a response generated by AI that contains false or misleading information presented as fact.[3][4] You say > This is different from human hallucinations where it makes something up because of something wrong with the mind rather than some underlying issue with the brain's architecture. For consistency you might as well say everything the human mind does is hallucination. It's the same sort of claim. This claim at least has the virtue of being taken seriously by people like Descartes. https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence) https://en.wikipedia.org/wiki/Hallucination_(artificial_inte...
    - ninetyninenine 1y ago
      Even the colloquial term outside of AI is characterized by the veracity of the output.
nine_k 1y ago
I'd rather say that LLMs live in a world that consists entirely of stories, nothing but words and their combinations. Thy have no other reality. So they are good at generating more stories that would sit well with the stories they already know. But the stories are often imprecise, and sometimes contradictory, so they have to guess. Also, LLMs don't know how to count, but they know that two usually follows one, and three is usually said to be larger than two, so they can speak in a way that mostly does not contradict this knowledge. They can use tools to count, like a human who knows digits would use a calculator. But much more than an arithmetic engine, the current crop of AI needs an epistemic engine, something that would help follow logic and avoid contradictions, to determine what is a well-established fact, and what is a shaky conjecture. Then we might start trusting the AI.
- gnerd00 1y ago
  this was true, but then it wasn't... the research world several years ago, had a moment when the machinery could reliably solve multi-step problems.. there had to be intermediary results; and machinery could solve problems in a domain where they were not trained specifically.. this caused a lot of excitement, and several hundred billion dollars in various investments.. Since no one actually knows how all of it works, not even the builders, here we are.
  - utyop22 1y ago
    "Since no one actually knows how all of it works, not even the builders, here we are." To me this is the most bizarre part. Have we ever had a technology deployed at this scale without a true understanding of its inner workings? My fear is that the general public perception of AI will be damaged since for most LLMs = AI.
    - riwsky 1y ago
      Humanity used fire for like a bazillion years before figuring out thermodynamics
    - SillyUsername 1y ago
anthem2025 1y ago
Isn’t that why people argue against calling them hallucinations? It implies that some parts of the output aren’t hallucinations, when the reality is that none of it has any thought behind it.
awesome_dude 1y ago
I have a very similar (probably unoriginal) thought about some human mental illnesses. So, we VALUE creativity, we claim that it helps us solve problems, improves our understanding of the universe, etc. BUT people with some mental illnesses, their brain is so creative that they lose the understanding of where reality is and where their imagination/creativity takes over. eg. Hearing voices? That's the brain conjuring up a voice - auditory and visual hallucinations are the easy example. But it goes further, depression is where people's brains create scenarios where there is no hope, and there's no escape. Anxiety too, the brain is conjuring up fears of what's to come
- malloryerik 1y ago
  You may like to check out Iain McGilchrist's take on schizophrenia, which essentially he says is a relative excess of rationality ("if then else" thinking) and a deficit of reasonableness (as in sensible context inhabiting).
  - awesome_dude 1y ago
    I shall have a read of that at some point.
keeda 1y ago
I've prefered to riff off of the other quote: "All (large language) model outputs are hallucinations, but some are useful." Some astonishingly large proportion of them, actually. Hence the AI boom.
jama211 1y ago
I find it a bit of a reductive way of looking at it personally

> hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful. Nice.

tptacek 1y ago

In that framing, you can look at an agent as simply a filter on those hallucinations.

th0ma5 1y ago
Yes yes, with yet to be discovered holes
Lionga 1y ago
Isn't an "agent" not just hallucinations layered on top of other random hallucinations to create new hallucinations?
- tptacek 1y ago
  No, that's exactly what an agent isn't. What makes an agent an agent is all the not-LLM code. When an agent generates Golang code, it runs the Go compiler, which is in the agent's architecture an extension of the agent. The Go compiler does not hallucinate.
  - Lionga 1y ago
    The most common "agent" is an letting an LLM run a while loop (“multi-step agent”) [1] [1] https://huggingface.co/docs/smolagents/conceptual_guides/intro_agents https://huggingface.co/docs/smolagents/conceptual_guides/int...
    - tptacek 1y ago
      That's not how Claude Code works (or Gemini, Cursor, or Codex).
armchairhacker 1y ago
This vaguely relates to a theory about human thought: that our subconscious constantly comes up with random ideas, then filters the unreasonable ones, but in people with delusions (e.g. schizophrenia) the filter is broken. Salience (https://en.wikipedia.org/wiki/Salience_(neuroscience) https://en.wikipedia.org/wiki/Salience_(neuroscience)), "the property by which some thing stands out", is something LLMs have trouble with. Probably because they're trained on human text, which ranges from accurate descriptions of reality to nonsense.
keeda 1y ago
More of a error-correcting feedback loop rather than a filter, really. Which is very much what we do as humans, apparently. One recent theory of neuroscience that is becoming influential is Predictive Processing --https://en.wikipedia.org/wiki/Predictive_coding https://en.wikipedia.org/wiki/Predictive_coding -- this postulates that we also constantly generate a "mental model" of our environment (a literal "prediction") and use sensory inputs to correct and update it. So the only real difference between "perception" and a "hallucination" is whether it is supported by physical reality.
- jonoc 1y ago
  thats a fascinating way to put it
- sethammons 1y ago
  You ever see something out of the corner of your eye and you were mistaken? It feels like LLM hallucinations. "An orange cat on my counter?!" And an instant later, your brain has reclassified "a basketball on my counter" as that fits the environment model better as several instant-observations gather contexts: not moving, more round, not furry, I don't own a cat, yesterday my kid mentioned something about tryouts, boop insta-reclassification from cat to basketball. I can recognize my own meta cognition there. My model of reality course corrects the information feed interpretation on the fly. Optical illusions feel very similar whereby the inner reality model clashes with the observed. For general ai, it needs a world model that can be tested against and surprise is noted and models are updated. Looping llm output with test cases is a crude approximation of that world model.
  - osullivj 1y ago
    I leaned heavily on my own meta cognition recently when withdrawing from a bereavment benzo habit recently. The combo of flu symptoms, anxiety and hallucinations are fierce. I knew O was seeing things that were not real; light fittings turning into a rotating stained glass slideshow. So I'm totally on board with the visual model hypothesis. My own speculation is that audio perception is less predictive as audio structure persists deeper into my own DMT sessions than does vision, where perspective quickly collapses and vision becomes kaleidoscopic. Which may be a return to the vision we had as newborns. Maybe normal vision is only attained via socialisation?

ninetyninenine 1y ago

Nah I don't agree with this characterization. The problem is, the majority of those hallucinations are true. What was said would make more sense if the majority of the responses were, in fact, false, but this is not the case.

deleted 1y ago
[deleted]
xmprt 1y ago
I think you're both correct but have different definitions of hallucinations. You're judging it as a hallucination based on the veracity of the output. Whereas Fowler is judging it based on the method by which the output is achieved. By that judgement, everything is a hallucination because the user cannot differentiate between when the LLM is telling the truth and isn't. This is different from human hallucinations where it makes something up because of something wrong with the mind rather than some underlying issue with the brain's architecture.
- ants_everywhere 1y ago
  an LLM hallucination is defined by its truth > In the field of artificial intelligence (AI), a hallucination or artificial hallucination (also called confabulation,[1] or delusion)[2] is a response generated by AI that contains false or misleading information presented as fact.[3][4] You say > This is different from human hallucinations where it makes something up because of something wrong with the mind rather than some underlying issue with the brain's architecture. For consistency you might as well say everything the human mind does is hallucination. It's the same sort of claim. This claim at least has the virtue of being taken seriously by people like Descartes. https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence) https://en.wikipedia.org/wiki/Hallucination_(artificial_inte...
  - ninetyninenine 1y ago
    Even the colloquial term outside of AI is characterized by the veracity of the output.

nine_k 1y ago

I'd rather say that LLMs live in a world that consists entirely of stories, nothing but words and their combinations. Thy have no other reality. So they are good at generating more stories that would sit well with the stories they already know. But the stories are often imprecise, and sometimes contradictory, so they have to guess. Also, LLMs don't know how to count, but they know that two usually follows one, and three is usually said to be larger than two, so they can speak in a way that mostly does not contradict this knowledge. They can use tools to count, like a human who knows digits would use a calculator. But much more than an arithmetic engine, the current crop of AI needs an epistemic engine, something that would help follow logic and avoid contradictions, to determine what is a well-established fact, and what is a shaky conjecture. Then we might start trusting the AI.

gnerd00 1y ago
this was true, but then it wasn't... the research world several years ago, had a moment when the machinery could reliably solve multi-step problems.. there had to be intermediary results; and machinery could solve problems in a domain where they were not trained specifically.. this caused a lot of excitement, and several hundred billion dollars in various investments.. Since no one actually knows how all of it works, not even the builders, here we are.
- utyop22 1y ago
  "Since no one actually knows how all of it works, not even the builders, here we are." To me this is the most bizarre part. Have we ever had a technology deployed at this scale without a true understanding of its inner workings? My fear is that the general public perception of AI will be damaged since for most LLMs = AI.
  - riwsky 1y ago
    Humanity used fire for like a bazillion years before figuring out thermodynamics
  - SillyUsername 1y ago

anthem2025 1y ago

Isn’t that why people argue against calling them hallucinations? It implies that some parts of the output aren’t hallucinations, when the reality is that none of it has any thought behind it.

awesome_dude 1y ago

I have a very similar (probably unoriginal) thought about some human mental illnesses. So, we VALUE creativity, we claim that it helps us solve problems, improves our understanding of the universe, etc. BUT people with some mental illnesses, their brain is so creative that they lose the understanding of where reality is and where their imagination/creativity takes over. eg. Hearing voices? That's the brain conjuring up a voice - auditory and visual hallucinations are the easy example. But it goes further, depression is where people's brains create scenarios where there is no hope, and there's no escape. Anxiety too, the brain is conjuring up fears of what's to come

malloryerik 1y ago
You may like to check out Iain McGilchrist's take on schizophrenia, which essentially he says is a relative excess of rationality ("if then else" thinking) and a deficit of reasonableness (as in sensible context inhabiting).
- awesome_dude 1y ago
  I shall have a read of that at some point.

keeda 1y ago

I've prefered to riff off of the other quote: "All (large language) model outputs are hallucinations, but some are useful." Some astonishingly large proportion of them, actually. Hence the AI boom.

jama211 1y ago

I find it a bit of a reductive way of looking at it personally