6 ms·

Does anyone have any technical insight on what actually causes the hallucinations? I know it’s an ongoing area of research, but do we have a lead?

by rzz3 1y ago

pkaye 1y ago
Anthropic had a recent paper that might be of interest. https://www.anthropic.com/research/tracing-thoughts-language-model https://www.anthropic.com/research/tracing-thoughts-language...
- namaria 1y ago
  The interpretation this paper offers is very questionable. It observes a so-called "replacement model" as a stand-in because it has a different architecture than the common LLMs, and lends itself to observing some "activation" patterns. Then it liberally labels patterns observed in the "replacement model" with words borrowed from psychology, neuroscience and cognitive science. It's all very fanciful and clearly directed at being pointed at as evidence of something deeper or more complex than what LLMs plainly do: statistical modelling of languages. Calling LLMs "next token predictor" is a bit of a cynical take, because that would be like calling a gaming engine a "pixel color processor". It's simplistic, yeah. But the polar opposite of spraying the explanation with convoluted inductive reasoning is just as bereft of substance.
minimaxir 1y ago
At a high level, what causes hallucinations is an easier question than how to solve them. LLMs are pretrained to maximize the probability of predicting the n+1 token given n tokens. To do this reliably, the model learns statistical patterns in the source data and transformer models are very good at doing that when large enough and given enough data. It is therefore suspect to any statistical biases in the training data because despite many advances in guiding LLMs, e.g. RLHF, LLMs are not sentient and most approaches to get around that such as the current reasoning models are hacks over a fundamental problem with the approach. It also doesn't help that when sampling the tokens, the default temperature of most LLM UIs is 1.0, with the argument that it is better for creativity. If you have access to the API and want a specific answer more reliably, I recommend setting temperature = 0.0, in which case the model will always select the token with the highest probability and tends to be more correct.
- rzz3 1y ago
  Thanks for the explanation, it makes a lot of sense. One thing I keep wondering though is why we can’t teach them logic at some sort of base layer. Not to anthropomorphize, but couldn’t the actual next-token prediction itself embed some sort of consultation against another (non-LLM) model that can evaluate a probability of logicality of the overall output that would be produced be selecting next-token-A vs next-token-B? I’m sure it’s obviously way more complicated than this, but still.
  - actsasbuffoon 1y ago
    I know this is reductive, but at their core, LLMs really are just very fancy autocomplete. The process they go through bears no resemblance to reasoning. Grab the best model you’ve got access to (o4, Gemini Pro 2.5, Claude Sonnet 3.7, etc) and try playing chess with it. The results are astonishingly bad. It’s not just that LLMs make poorly thought out moves. They regularly make completely illegal moves (making a knight move like a pawn, for example), hallucinate new pieces into existence, change pieces from one color to another, etc. More words have probably been written about Chess than any other game. You can literally buy a book that’s just about chess openings. There has to be a staggering amount of text about chess in the training data set. And yet, even the best models are far worse at chess than I was in 3rd grade (and I’m not great). They do pretty well for the first few moves, often playing well known chess openings. But once they get to the part where you need to reason, they fall apart in the most outrageous ways. I suspect this reveals something about how they deal with other logic-based tasks. I’ve wondered for a while now if they basically code through using a vast amount of training data from Stack Overflow, which is why they’re so good at helping with common error codes, and so useless when doing something novel. These new chess experiments have been eye-opening about how incapable the best models are of basic reasoning. Unless something about that changes drastically, I think LLMs have pretty much plateaued. There will undoubtedly be some advancements, but LLMs are never going to reach AGI without reasoning, nor will they be able to do most jobs.
  - Jensson 1y ago
    > One thing I keep wondering though is why we can’t teach them logic at some sort of base layer That is what they did with these "chain of thought" models. Maybe they didn't do it in the optimal way, but they did train them on their ability to answer certain questions. So the low hanging fruits from this style of training has already been plucked.
vikramkr 1y ago
There's the anthropic paper someone else linked, but also it's pretty interesting to see the question framed as trying to understand what causes the hallucinations lol. It's a (very fancy) next word predictor - it's kind of amazing that it doesn't hallucinate! Like that paper showed that there were circuits that functionally actually do things resembling arithmetic and computation with lookup tables instead of just blindly 'guessing' a random number when asked what an arithmetic expression equals and that seems like the much more extraordinary thing that we want to figure out the cause of!
tripplyons 1y ago
In pretraining, the model is just trying to predict the most likely next token given the context. Its guesses are not always correct, which leads to hallucinations. Post-training often incentivizes the model to sound confident in its output, which can make the problem worse.
asadotzler 1y ago
All responses are hallucinations; some of them are close enough to what we want to be useful, others not so much.
- deleted 1y ago
  [deleted]
- tripplyons 1y ago
  Correct responses wouldn't be considered hallucinations.
  - CharlesW 1y ago
    Not by the human evaluator, assuming they know (or otherwise validate) that the generated output is correct. But an LLM alone doesn't differentiate between hallucinations and not-hallucinations.
  - Terr_ 1y ago
    ... But they ought to be considered hallucinations, it's the same algorithm at work, we just are biased in favor of certain results. Similarly, suppose I always roll dice to determine tomorrow's winning lottery-ticket number. Getting it right one day doesn't change the mechanism I used. Some people might assume I was psychic, but would be wrong.
  - Matthyze 1y ago
    You're absolutely correct and it's ridiculous that HN can't fathom that a hallucination, in both the traditional and LLM sense, is not characterized by its form or content per se but by the fact that the form or content is incongruent with some external factor.
- esafak 1y ago
  Would you say the same of humans?
  - doug_durham 1y ago
riwsky 1y ago
Hallucinations are kind of the default, like finding hay in a haystack. The leads (needles) people search for are on what causes non-hallucinations.
AIPedant 1y ago
One vague possibility: hallucinations are mitigated by looking at uncertainty in token prediction, the idea being that it’s a proxy for factual uncertainty, and the system is more likely to confabulate a court case or whatever if it only has a fuzzy idea of what comes next. But this won’t work for reasoning models which solve math problems: the next token for “x=“ is highly uncertain until you’ve done the computation!
- nullc 1y ago
  RLHF generally tends to make the model pretty confident making the entropy of it a not so useful predictor of when the model is going to get something wrong. You can do a kind of sensitivity analysis to see how sensitive the output is to small perturbations of the weights... but it's computationally expensive. Might be an interesting form of fine tuning to do distillation where the student's current sensitivity to noise (extracted from the backwards step of gradient descent) is used to shrink the predicted distribution towards uniform. It could be done very cheaply during training and perhaps could avoid the back propagation cost of doing it at inference time.
esafak 1y ago
"[Hallucinations] don't form from out-of-distribution samples — they arise due to spurious attractor states on the Energy Landscape. In other words, 'noisy bumps' in the loss. So in the physics approach, to characterize the training error, we ask how sensitive is the free energy to noise. This is why in statistical physics, the training and generalization errors are not defined distributionally. That is, we don't define them using a training set and a test set. Instead, the errors are found by using the free energy as a generating function and taking the appropriate partial derivative." https://www.linkedin.com/posts/charlesmartin14_talktochuck-theaiguy-activity-7307260688509349889-RHvF/ https://www.linkedin.com/posts/charlesmartin14_talktochuck-t...
jablongo 1y ago
There is not really some distinct pathology with hallucinations, its just how wrong answers (e.g. inaccuracies / faulty token prediction chains) manifest in the case of LLMs. In the case of a linear regression, a "hallucination" is when the predicted value was far from the actual value for a given sample.
anon373839 1y ago
Hallucinations are caused by humans anthropomorphizing LLMs and imagining that they possess properties that don't exist. For example, LLMs cannot test their thoughts against external evidence or other knowledge they may have (such as logic) to think before they output something. That's because they are a frozen computation graph with some random noise on top. Even chain of thought prompting or RL-based "reasoning" are just a pale imitation of the behavior we actually wish we could get. It is just a method of using the same model to generate some context that improves the odds of a good final result. But the model itself does not actually consider the thoughts it is <thinking>. These "thoughts" (and the response that follows them) can and do exhibit the same defects as hallucinations, because they are just more of the same. Of course, the field has made some strides in reducing hallucinations. And it's not a total mystery why some outputs make sense and others don't. For example, just like with any other statistical model, the likelihood of error increases as the input becomes more dissimilar to the training data. But also, similarity to specific training data can be a problem because of overfitting. In those cases, the model is likely to output the common pattern rather than the pattern that would make sense for the given input.
calf 1y ago
Yann LeCun gave a recent talk, at 10:30 he gives a good explanation that doesn't appeal to "what LLMs are trying to do on the inside": LeCun just observes that next-token prediction is exponentially diverging, since every next token with the smallest error e is still (roughly speaking) (1-e)^n for output of length n. I just thought it was a good complexity-theory based explanation (unlike most other statistical-parrot type arguments). It is a single slide, very helpful: https://www.youtube.com/watch?v=ETZfkkv6V7Y https://www.youtube.com/watch?v=ETZfkkv6V7Y
nullc 1y ago
Hallucinations are what make the models useful. Well, when we like them we call it intelligence "look it did something correct that was nowhere in the training data set! It's intelligent!" and when we don't like them we call it hallucinations "look, it did something that was nowhere in the training dataset because it was wrong!". But they are the same thing: the model is extrapolating. It doesn't know when its extrapolations are correct or not because an LLM doesn't have access to the outside world (except via you and whatever tools you give it). If it was free of extrapolations it would just be a search engine over the training data, and that would be less useful.

Does anyone have any technical insight on what actually causes the hallucinations? I know it’s an ongoing area of research, but do we have a lead?

by rzz3 1y ago

Does anyone have any technical insight on what actually causes the hallucinations? I know it’s an ongoing area of research, but do we have a lead?

pkaye 1y ago

Anthropic had a recent paper that might be of interest. https://www.anthropic.com/research/tracing-thoughts-language-model https://www.anthropic.com/research/tracing-thoughts-language...

namaria 1y ago
The interpretation this paper offers is very questionable. It observes a so-called "replacement model" as a stand-in because it has a different architecture than the common LLMs, and lends itself to observing some "activation" patterns. Then it liberally labels patterns observed in the "replacement model" with words borrowed from psychology, neuroscience and cognitive science. It's all very fanciful and clearly directed at being pointed at as evidence of something deeper or more complex than what LLMs plainly do: statistical modelling of languages. Calling LLMs "next token predictor" is a bit of a cynical take, because that would be like calling a gaming engine a "pixel color processor". It's simplistic, yeah. But the polar opposite of spraying the explanation with convoluted inductive reasoning is just as bereft of substance.

minimaxir 1y ago

At a high level, what causes hallucinations is an easier question than how to solve them. LLMs are pretrained to maximize the probability of predicting the n+1 token given n tokens. To do this reliably, the model learns statistical patterns in the source data and transformer models are very good at doing that when large enough and given enough data. It is therefore suspect to any statistical biases in the training data because despite many advances in guiding LLMs, e.g. RLHF, LLMs are not sentient and most approaches to get around that such as the current reasoning models are hacks over a fundamental problem with the approach. It also doesn't help that when sampling the tokens, the default temperature of most LLM UIs is 1.0, with the argument that it is better for creativity. If you have access to the API and want a specific answer more reliably, I recommend setting temperature = 0.0, in which case the model will always select the token with the highest probability and tends to be more correct.

rzz3 1y ago
Thanks for the explanation, it makes a lot of sense. One thing I keep wondering though is why we can’t teach them logic at some sort of base layer. Not to anthropomorphize, but couldn’t the actual next-token prediction itself embed some sort of consultation against another (non-LLM) model that can evaluate a probability of logicality of the overall output that would be produced be selecting next-token-A vs next-token-B? I’m sure it’s obviously way more complicated than this, but still.
- actsasbuffoon 1y ago
  I know this is reductive, but at their core, LLMs really are just very fancy autocomplete. The process they go through bears no resemblance to reasoning. Grab the best model you’ve got access to (o4, Gemini Pro 2.5, Claude Sonnet 3.7, etc) and try playing chess with it. The results are astonishingly bad. It’s not just that LLMs make poorly thought out moves. They regularly make completely illegal moves (making a knight move like a pawn, for example), hallucinate new pieces into existence, change pieces from one color to another, etc. More words have probably been written about Chess than any other game. You can literally buy a book that’s just about chess openings. There has to be a staggering amount of text about chess in the training data set. And yet, even the best models are far worse at chess than I was in 3rd grade (and I’m not great). They do pretty well for the first few moves, often playing well known chess openings. But once they get to the part where you need to reason, they fall apart in the most outrageous ways. I suspect this reveals something about how they deal with other logic-based tasks. I’ve wondered for a while now if they basically code through using a vast amount of training data from Stack Overflow, which is why they’re so good at helping with common error codes, and so useless when doing something novel. These new chess experiments have been eye-opening about how incapable the best models are of basic reasoning. Unless something about that changes drastically, I think LLMs have pretty much plateaued. There will undoubtedly be some advancements, but LLMs are never going to reach AGI without reasoning, nor will they be able to do most jobs.
- Jensson 1y ago
  > One thing I keep wondering though is why we can’t teach them logic at some sort of base layer That is what they did with these "chain of thought" models. Maybe they didn't do it in the optimal way, but they did train them on their ability to answer certain questions. So the low hanging fruits from this style of training has already been plucked.

vikramkr 1y ago

There's the anthropic paper someone else linked, but also it's pretty interesting to see the question framed as trying to understand what causes the hallucinations lol. It's a (very fancy) next word predictor - it's kind of amazing that it doesn't hallucinate! Like that paper showed that there were circuits that functionally actually do things resembling arithmetic and computation with lookup tables instead of just blindly 'guessing' a random number when asked what an arithmetic expression equals and that seems like the much more extraordinary thing that we want to figure out the cause of!

tripplyons 1y ago

In pretraining, the model is just trying to predict the most likely next token given the context. Its guesses are not always correct, which leads to hallucinations. Post-training often incentivizes the model to sound confident in its output, which can make the problem worse.

asadotzler 1y ago

All responses are hallucinations; some of them are close enough to what we want to be useful, others not so much.

deleted 1y ago
[deleted]
tripplyons 1y ago
Correct responses wouldn't be considered hallucinations.
- CharlesW 1y ago
  Not by the human evaluator, assuming they know (or otherwise validate) that the generated output is correct. But an LLM alone doesn't differentiate between hallucinations and not-hallucinations.
- Terr_ 1y ago
  ... But they ought to be considered hallucinations, it's the same algorithm at work, we just are biased in favor of certain results. Similarly, suppose I always roll dice to determine tomorrow's winning lottery-ticket number. Getting it right one day doesn't change the mechanism I used. Some people might assume I was psychic, but would be wrong.
- Matthyze 1y ago
  You're absolutely correct and it's ridiculous that HN can't fathom that a hallucination, in both the traditional and LLM sense, is not characterized by its form or content per se but by the fact that the form or content is incongruent with some external factor.
esafak 1y ago
Would you say the same of humans?
- doug_durham 1y ago

riwsky 1y ago

Hallucinations are kind of the default, like finding hay in a haystack. The leads (needles) people search for are on what causes non-hallucinations.

AIPedant 1y ago

One vague possibility: hallucinations are mitigated by looking at uncertainty in token prediction, the idea being that it’s a proxy for factual uncertainty, and the system is more likely to confabulate a court case or whatever if it only has a fuzzy idea of what comes next. But this won’t work for reasoning models which solve math problems: the next token for “x=“ is highly uncertain until you’ve done the computation!

nullc 1y ago
RLHF generally tends to make the model pretty confident making the entropy of it a not so useful predictor of when the model is going to get something wrong. You can do a kind of sensitivity analysis to see how sensitive the output is to small perturbations of the weights... but it's computationally expensive. Might be an interesting form of fine tuning to do distillation where the student's current sensitivity to noise (extracted from the backwards step of gradient descent) is used to shrink the predicted distribution towards uniform. It could be done very cheaply during training and perhaps could avoid the back propagation cost of doing it at inference time.

esafak 1y ago

"[Hallucinations] don't form from out-of-distribution samples — they arise due to spurious attractor states on the Energy Landscape. In other words, 'noisy bumps' in the loss. So in the physics approach, to characterize the training error, we ask how sensitive is the free energy to noise. This is why in statistical physics, the training and generalization errors are not defined distributionally. That is, we don't define them using a training set and a test set. Instead, the errors are found by using the free energy as a generating function and taking the appropriate partial derivative." https://www.linkedin.com/posts/charlesmartin14_talktochuck-theaiguy-activity-7307260688509349889-RHvF/ https://www.linkedin.com/posts/charlesmartin14_talktochuck-t...

jablongo 1y ago

There is not really some distinct pathology with hallucinations, its just how wrong answers (e.g. inaccuracies / faulty token prediction chains) manifest in the case of LLMs. In the case of a linear regression, a "hallucination" is when the predicted value was far from the actual value for a given sample.

anon373839 1y ago

Hallucinations are caused by humans anthropomorphizing LLMs and imagining that they possess properties that don't exist. For example, LLMs cannot test their thoughts against external evidence or other knowledge they may have (such as logic) to think before they output something. That's because they are a frozen computation graph with some random noise on top. Even chain of thought prompting or RL-based "reasoning" are just a pale imitation of the behavior we actually wish we could get. It is just a method of using the same model to generate some context that improves the odds of a good final result. But the model itself does not actually consider the thoughts it is <thinking>. These "thoughts" (and the response that follows them) can and do exhibit the same defects as hallucinations, because they are just more of the same. Of course, the field has made some strides in reducing hallucinations. And it's not a total mystery why some outputs make sense and others don't. For example, just like with any other statistical model, the likelihood of error increases as the input becomes more dissimilar to the training data. But also, similarity to specific training data can be a problem because of overfitting. In those cases, the model is likely to output the common pattern rather than the pattern that would make sense for the given input.

calf 1y ago

Yann LeCun gave a recent talk, at 10:30 he gives a good explanation that doesn't appeal to "what LLMs are trying to do on the inside": LeCun just observes that next-token prediction is exponentially diverging, since every next token with the smallest error e is still (roughly speaking) (1-e)^n for output of length n. I just thought it was a good complexity-theory based explanation (unlike most other statistical-parrot type arguments). It is a single slide, very helpful: https://www.youtube.com/watch?v=ETZfkkv6V7Y https://www.youtube.com/watch?v=ETZfkkv6V7Y

nullc 1y ago

Hallucinations are what make the models useful. Well, when we like them we call it intelligence "look it did something correct that was nowhere in the training data set! It's intelligent!" and when we don't like them we call it hallucinations "look, it did something that was nowhere in the training dataset because it was wrong!". But they are the same thing: the model is extrapolating. It doesn't know when its extrapolations are correct or not because an LLM doesn't have access to the outside world (except via you and whatever tools you give it). If it was free of extrapolations it would just be a search engine over the training data, and that would be less useful.