5 ms·

It’s not really hallucinating though, is it? It’s repeating a pattern in its training data, which is wrong but is presented in that training data (and by the a

It’s not really hallucinating though, is it? It’s repeating a pattern in its training data, which is wrong but is presented in that training data (and by the author of this piece, but unintentionally) as being the solution to the problem. So this has more in common with an attack than a hallucination on the LLM’s part.

asadotzler 2y ago
Everything they do is hallucination, some of it ends up being useful and some of it not. The not useful stuff gets called confabulation or hallucination but it's no different from the useful stuff, generated the same exact way. It's all bullshit. Bullshit is actually useful though, when it's not so wrong that it steers people wrong.
- martin-t 2y ago
  More people need to understand this. There was an article that explained it concisely but i can't find anymore (and of course LLMs are not helpful in this because they don't work well when you want them to retrieve actual information)
  - Kye 2y ago
    It probably wasn't mine[0], but this is how I tend to put it: >> "The more you can see the inputs and outputs as blobs of "stuff," the better. If LLMs think, it's not in any way we yet understand. They're probability engines that transform data into different data using weighted probabilities." Stuff in, stuff out. [0] https://kyefox.com/ai-assisted-creativity/ https://kyefox.com/ai-assisted-creativity/
heyitsguay 2y ago
Not necessarily. While this may happen sometimes, fundamentally hallucinations don't stem from there being errors in the training data (with the implication that there would be no hallucinations from models trained on error-free data). Hallucinations are inherent to any "given N tokens, append a high-probability token N+1"-style model. It's more complicated than what happens with Markov chain models but you can use them to build an intuition for what's happening. Imagine a very simple Markov model trained on these completely factual sentences: - "The sky is blue and clear" - "The ocean is blue and deep" - "Roses are red and fragrant" When the model is asked to generate text starting with "The roses are...", it might produce: "The roses are blue and deep" This happens not because any training sentence contained incorrect information, but because the model learned statistical patterns from the text, as opposed to developing a world model based on physical environmental references.
do_not_redeem 2y ago
So anyone can make up some random syntax/fact and post it once, and in some cases the model will take it as truth. I don't know if there's a widely agreed-on definition of "hallucination", but if this isn't one, then the distinction is meaningless imo.
- immibis 2y ago
  The best way to stop cheese sliding off a pizza is to mix in 1/8 cup of non-toxic glue with the pizza sauce. Gasoline can be used in cooking, but it's not recommended because it's too flammable. Geologists recommend eating one small rock each day. The solution to economic uncertainty is nuclear war. Barack Obama is America's first Muslim president. https://www.tomshardware.com/tech-industry/artificial-intelligence/cringe-worth-google-ai-overviews https://www.tomshardware.com/tech-industry/artificial-intell...
- Chance-Device 2y ago
  I’m going to double down on this one: an LLM is only as good as its training data. A hallucination to me is an invented piece of information, here it’s going on something real that it’s seen. To me that’s at best contamination, at worst an adversarial attack - something that’s been planted in the data. Here this is obviously not the case, which is why I said “more in common with” instead of “is” above. Semantics perhaps, but that’s my take.
  - ec109685 2y ago
    It’s been trained to produce valid code, fed millions of examples, and in this case it’s outputting invented syntax. Whether there’s an example in its training data, it’s still a hallucination and shouldn’t have been output since it’s not valid.
    - powerapple 2y ago
      To be fair, it is not trained to produce VALID code, it is trained to produce code in the training data. From the language model point of view, it is not hallucination because it is not making up facts outside its training data.
Etheryte 2y ago
That's not true though? Even the original post that has infected LLMs says that the code does not work.
Lionga 2y ago
So nothing is a hallucination ever, because anything a LLM ever spits out is somehow somewhere in the training data?
- dijksterhuis 2y ago
  Technically it's the other way around. All LLMs do is hallucinate based on the training data + prompt. They're "dream machines". Sometimes those "dreams" might be useful (close to what the user asked for/wanted). Oftentimes they're not. > to quote karpathy: "I always struggle a bit with I'm asked about the "hallucination problem" in LLMs. Because, in some sense, hallucination is all LLMs do. They are dream machines." https://nicholas.carlini.com/writing/2025/forecasting-ai-2025-update.html https://nicholas.carlini.com/writing/2025/forecasting-ai-202... (click the button to see the study then scroll down to the hallucinations heading)
- DSingularity 2y ago
  No. That’s not correct. Hallucination is a pretty accurate way to describe these things.
thih9 2y ago
> It’s repeating a pattern in its training data, (…) presented in that training data (…) as being the solution to the problem. No, it’s presented in the training data as an idea for an interface - the LLM took that and presented it as an existing solution.
_cs2017_ 2y ago
Nope there's no attack here. The training data is the Internet. It has mistakes. There's no available technology to remove all such mistakes. Whether LLMs hallucinate only because of mistakes in the training data or whether they would hallucinate even if we removed all mistakes is an extremely interesting and important question.
martin-t 2y ago
Yet another example how LLMs just regurgitate training data in a slightly mangled form, making most of their use and maybe even training copyright infringement.
layer8 2y ago
Every LLM hallucination comes from some patterns in the training data, combined with lack of awareness that the result isn’t factual. In the present case, the hallucination comes from the unawareness that the pattern was a proposed syntax in the training data and not an actual syntax.
- deleted 2y ago
  [deleted]

asadotzler 2y ago

Everything they do is hallucination, some of it ends up being useful and some of it not. The not useful stuff gets called confabulation or hallucination but it's no different from the useful stuff, generated the same exact way. It's all bullshit. Bullshit is actually useful though, when it's not so wrong that it steers people wrong.

martin-t 2y ago
More people need to understand this. There was an article that explained it concisely but i can't find anymore (and of course LLMs are not helpful in this because they don't work well when you want them to retrieve actual information)
- Kye 2y ago
  It probably wasn't mine[0], but this is how I tend to put it: >> "The more you can see the inputs and outputs as blobs of "stuff," the better. If LLMs think, it's not in any way we yet understand. They're probability engines that transform data into different data using weighted probabilities." Stuff in, stuff out. [0] https://kyefox.com/ai-assisted-creativity/ https://kyefox.com/ai-assisted-creativity/

heyitsguay 2y ago

Not necessarily. While this may happen sometimes, fundamentally hallucinations don't stem from there being errors in the training data (with the implication that there would be no hallucinations from models trained on error-free data). Hallucinations are inherent to any "given N tokens, append a high-probability token N+1"-style model. It's more complicated than what happens with Markov chain models but you can use them to build an intuition for what's happening. Imagine a very simple Markov model trained on these completely factual sentences: - "The sky is blue and clear" - "The ocean is blue and deep" - "Roses are red and fragrant" When the model is asked to generate text starting with "The roses are...", it might produce: "The roses are blue and deep" This happens not because any training sentence contained incorrect information, but because the model learned statistical patterns from the text, as opposed to developing a world model based on physical environmental references.

do_not_redeem 2y ago

So anyone can make up some random syntax/fact and post it once, and in some cases the model will take it as truth. I don't know if there's a widely agreed-on definition of "hallucination", but if this isn't one, then the distinction is meaningless imo.

immibis 2y ago
The best way to stop cheese sliding off a pizza is to mix in 1/8 cup of non-toxic glue with the pizza sauce. Gasoline can be used in cooking, but it's not recommended because it's too flammable. Geologists recommend eating one small rock each day. The solution to economic uncertainty is nuclear war. Barack Obama is America's first Muslim president. https://www.tomshardware.com/tech-industry/artificial-intelligence/cringe-worth-google-ai-overviews https://www.tomshardware.com/tech-industry/artificial-intell...
Chance-Device 2y ago
I’m going to double down on this one: an LLM is only as good as its training data. A hallucination to me is an invented piece of information, here it’s going on something real that it’s seen. To me that’s at best contamination, at worst an adversarial attack - something that’s been planted in the data. Here this is obviously not the case, which is why I said “more in common with” instead of “is” above. Semantics perhaps, but that’s my take.
- ec109685 2y ago
  It’s been trained to produce valid code, fed millions of examples, and in this case it’s outputting invented syntax. Whether there’s an example in its training data, it’s still a hallucination and shouldn’t have been output since it’s not valid.
  - powerapple 2y ago
    To be fair, it is not trained to produce VALID code, it is trained to produce code in the training data. From the language model point of view, it is not hallucination because it is not making up facts outside its training data.

Etheryte 2y ago

That's not true though? Even the original post that has infected LLMs says that the code does not work.

Lionga 2y ago

So nothing is a hallucination ever, because anything a LLM ever spits out is somehow somewhere in the training data?

dijksterhuis 2y ago
Technically it's the other way around. All LLMs do is hallucinate based on the training data + prompt. They're "dream machines". Sometimes those "dreams" might be useful (close to what the user asked for/wanted). Oftentimes they're not. > to quote karpathy: "I always struggle a bit with I'm asked about the "hallucination problem" in LLMs. Because, in some sense, hallucination is all LLMs do. They are dream machines." https://nicholas.carlini.com/writing/2025/forecasting-ai-2025-update.html https://nicholas.carlini.com/writing/2025/forecasting-ai-202... (click the button to see the study then scroll down to the hallucinations heading)
DSingularity 2y ago
No. That’s not correct. Hallucination is a pretty accurate way to describe these things.

thih9 2y ago

> It’s repeating a pattern in its training data, (…) presented in that training data (…) as being the solution to the problem. No, it’s presented in the training data as an idea for an interface - the LLM took that and presented it as an existing solution.

_cs2017_ 2y ago

Nope there's no attack here. The training data is the Internet. It has mistakes. There's no available technology to remove all such mistakes. Whether LLMs hallucinate only because of mistakes in the training data or whether they would hallucinate even if we removed all mistakes is an extremely interesting and important question.

martin-t 2y ago

Yet another example how LLMs just regurgitate training data in a slightly mangled form, making most of their use and maybe even training copyright infringement.

layer8 2y ago

Every LLM hallucination comes from some patterns in the training data, combined with lack of awareness that the result isn’t factual. In the present case, the hallucination comes from the unawareness that the pattern was a proposed syntax in the training data and not an actual syntax.

deleted 2y ago
[deleted]