3 ms·

On the Existence, Impact, and Origin of Hallucination-Associated Neurons in LLMs

30 points by bilsbie 10mo ago

buppermint 10mo ago
From a quick read, this is cool but maybe a little overstated. From Figure 3, completely suppressing these neurons only reduces hallucinations by like ~5% compared to their normal state. Table 1 is even more odd, H-neurons predicts hallucination ~75% of the time but a similar % of random neurons predict hallucinations ~60% of the time, which doesn't seem like a huge difference to me.
deleted 10mo ago
[deleted]
lukeinator42 10mo ago
It is fairly well established that neurons in these artificial neural networks are polysemantic and information is represented in directions in the activation embedding space rather than neurons independently representing information (which is why anthropic is doing things like training sparse autoencoders). I haven't read the paper in depth but it seems like it is based on a fundamental misunderstanding about neurons in ANNs vs the brain.
- readthenotes1 10mo ago
  It may be that there are just a few misfirings that dramatically degrade the results. Strokes in the human brain causing a few neurons to die can have outsized affects in the body...
GaggiX 10mo ago
The people talking about semantics in the comment section seems to completely ignore the positive correlation of LLMs between accuracy and stated confidence, this is called calibration and this "old" blog post from a year ago already showed it, LLMs can know what they know: https://openai.com/index/introducing-simpleqa/ https://openai.com/index/introducing-simpleqa/
jurystillout26 10mo ago
Many people seem to be claiming that "LLMs do what humans do / humans also hallucinate", as if the process of human knowledge is identical to the purely semantic knowledge of LLMs. No. Human beings have experiential, embodied, temporal knowledge of the world through our senses. That is why we can, say, empirically know something, which is vastly different than semantically or logically knowing something. Yes, human beings also have probabalistic ways of understanding the world and interacting with others. We have many other forms of knowledge as well and the LLM way of interpreting data is by no means the primary way in which we feel confident that something is true or false. That said, I don't get up in arms about the term "hallucination", although I prefer the term confabulation per neuroscientist Anil Seth. Many clunky metaphors are now mainstream, and as long as the engineers and researchers who study these kinds of things are ok with that, that's the most important thing. But what I think all these people who dismiss objections to the term as "arguing semantics" are missing is the fundamental point: LLMs have no intent, and they have no way of distinguishing what data is empirically true or not. This is why the framing, not just the semantics, of this piece is flawed. "Hallucinations" is a feature of LLMs that exists at the very conceptual level, not as a design flaw of current models. They have pattern recognition, which gets us very far in terms of knowing things, but people who only rely on such methods of knowing are most often referred to as conspiracy theorists.
- meowface 10mo ago
  On one hand this is true. On the other hand it does seem possible, in theory, that through enough post-training and other measures they could become a bit closer to human minds and not just be token guessers. The human brain may at its fundamental level operate on the principles of predictive processing (https://slatestarcodex.com/2017/09/05/book-review-surfing-uncertainty/ https://slatestarcodex.com/2017/09/05/book-review-surfing-un...). It might be that it has many layers surrounding that raw predictive core which develop us into epistemological beings. The LLMs we see today may be in the very early stages of a similar sort of (artificial) evolution. The reflexiveness with which even top models like Opus 4.5 will sometimes seamlessly confabulate things definitely does make it seem like it is a very deep problem, but I don't think it's necessarily unsolvable. I used to be among the vast majority of people who thought LLMs were not sufficient to get us to AGI/ASI, but I'm increasingly starting to feel that piling enough hacks atop LLMs might really be what gets there before anything else.
dang 10mo ago
[stub for offtopicness] [submitters: one reason for not editorializing titles is it makes the threads be about that!]
- bowsamic 10mo ago
  “Physical”? I don’t think that’s the right word to use
  - Miraltar 10mo ago
    Agreed, the original title is better (and typo free) although a bit long. H-Neurons: On the Existence, Impact, and Origin of Hallucination-Associated Neurons in LLMs
  - ZeroConcerns 10mo ago
    Yeah, actual title is "H-Neurons: On the Existence, Impact, and Origin of Hallucination-Associated Neurons in LLMs" But regardless of title this is all highly dubious...
- airhangerf15 10mo ago
  LLMs don't "hallucinate" or "lie." They have no intent. They're Weighted Random Word Generator Machines. They're train mathematically to create series of tokens. Whenever they get something "right," it's literally by accident. If you get that rate of accidental rightness up to 80%, and people suddenly thing the random word generator is some kind of oracle. It's not. It's a large model with an embedded space, tokens and a whole series of computationally expensive perceptron and attention blocks that generate output. The title/introduction is very baited, because it implies some "physical" connection to hallucinations in biological organism, but it's focused on trying to single out certain parts of the model. LLMs are absolutely nothing at all like a biological system, of which our brains are orders of magnitudes more complex than the machines we've built that we no longer fully understand. Believing in these LLMs as being some next stage in understanding intelligence is hubris.
  - pektezol 10mo ago

30 points by bilsbie 10mo ago

From a quick read, this is cool but maybe a little overstated. From Figure 3, completely suppressing these neurons only reduces hallucinations by like ~5% compared to their normal state. Table 1 is even more odd, H-neurons predicts hallucination ~75% of the time but a similar % of random neurons predict hallucinations ~60% of the time, which doesn't seem like a huge difference to me.

deleted 10mo ago

[deleted]

lukeinator42 10mo ago

It is fairly well established that neurons in these artificial neural networks are polysemantic and information is represented in directions in the activation embedding space rather than neurons independently representing information (which is why anthropic is doing things like training sparse autoencoders). I haven't read the paper in depth but it seems like it is based on a fundamental misunderstanding about neurons in ANNs vs the brain.

readthenotes1 10mo ago
It may be that there are just a few misfirings that dramatically degrade the results. Strokes in the human brain causing a few neurons to die can have outsized affects in the body...

GaggiX 10mo ago

The people talking about semantics in the comment section seems to completely ignore the positive correlation of LLMs between accuracy and stated confidence, this is called calibration and this "old" blog post from a year ago already showed it, LLMs can know what they know: https://openai.com/index/introducing-simpleqa/ https://openai.com/index/introducing-simpleqa/

jurystillout26 10mo ago

Many people seem to be claiming that "LLMs do what humans do / humans also hallucinate", as if the process of human knowledge is identical to the purely semantic knowledge of LLMs. No. Human beings have experiential, embodied, temporal knowledge of the world through our senses. That is why we can, say, empirically know something, which is vastly different than semantically or logically knowing something. Yes, human beings also have probabalistic ways of understanding the world and interacting with others. We have many other forms of knowledge as well and the LLM way of interpreting data is by no means the primary way in which we feel confident that something is true or false. That said, I don't get up in arms about the term "hallucination", although I prefer the term confabulation per neuroscientist Anil Seth. Many clunky metaphors are now mainstream, and as long as the engineers and researchers who study these kinds of things are ok with that, that's the most important thing. But what I think all these people who dismiss objections to the term as "arguing semantics" are missing is the fundamental point: LLMs have no intent, and they have no way of distinguishing what data is empirically true or not. This is why the framing, not just the semantics, of this piece is flawed. "Hallucinations" is a feature of LLMs that exists at the very conceptual level, not as a design flaw of current models. They have pattern recognition, which gets us very far in terms of knowing things, but people who only rely on such methods of knowing are most often referred to as conspiracy theorists.

meowface 10mo ago
On one hand this is true. On the other hand it does seem possible, in theory, that through enough post-training and other measures they could become a bit closer to human minds and not just be token guessers. The human brain may at its fundamental level operate on the principles of predictive processing (https://slatestarcodex.com/2017/09/05/book-review-surfing-uncertainty/ https://slatestarcodex.com/2017/09/05/book-review-surfing-un...). It might be that it has many layers surrounding that raw predictive core which develop us into epistemological beings. The LLMs we see today may be in the very early stages of a similar sort of (artificial) evolution. The reflexiveness with which even top models like Opus 4.5 will sometimes seamlessly confabulate things definitely does make it seem like it is a very deep problem, but I don't think it's necessarily unsolvable. I used to be among the vast majority of people who thought LLMs were not sufficient to get us to AGI/ASI, but I'm increasingly starting to feel that piling enough hacks atop LLMs might really be what gets there before anything else.

dang 10mo ago

[stub for offtopicness] [submitters: one reason for not editorializing titles is it makes the threads be about that!]

bowsamic 10mo ago
“Physical”? I don’t think that’s the right word to use
- Miraltar 10mo ago
  Agreed, the original title is better (and typo free) although a bit long. H-Neurons: On the Existence, Impact, and Origin of Hallucination-Associated Neurons in LLMs
- ZeroConcerns 10mo ago
  Yeah, actual title is "H-Neurons: On the Existence, Impact, and Origin of Hallucination-Associated Neurons in LLMs" But regardless of title this is all highly dubious...
airhangerf15 10mo ago
LLMs don't "hallucinate" or "lie." They have no intent. They're Weighted Random Word Generator Machines. They're train mathematically to create series of tokens. Whenever they get something "right," it's literally by accident. If you get that rate of accidental rightness up to 80%, and people suddenly thing the random word generator is some kind of oracle. It's not. It's a large model with an embedded space, tokens and a whole series of computationally expensive perceptron and attention blocks that generate output. The title/introduction is very baited, because it implies some "physical" connection to hallucinations in biological organism, but it's focused on trying to single out certain parts of the model. LLMs are absolutely nothing at all like a biological system, of which our brains are orders of magnitudes more complex than the machines we've built that we no longer fully understand. Believing in these LLMs as being some next stage in understanding intelligence is hubris.
- pektezol 10mo ago