5 ms·

There were also several hallucinations during the announcement. (I also see hallucinations every time I use Claude and GPT, which is several times a week. Paid

There were also several hallucinations during the announcement. (I also see hallucinations every time I use Claude and GPT, which is several times a week. Paid and free tiers) So not seeing them means either lying or incompetent. I always try to attribute to stupidity rather than malice (Hanlon's razor). The big problem of LLMs is that they optimize human preference. This means they optimize for hidden errors. Personally I'm really cautious about using tools that have stealthy failure modes. They just lead to many problems and lots of wasted hours debugging, even when failure rates are low. It just causes everything to slow down for me as I'm double checking everything and need to be much more meticulous if I know it's hard to see. It's like having a line of Python indented with an inconsistent white space character. Impossible to see. But what if you didn't have the interpreter telling you which line you failed on or being able to search or highlight these different characters. At least in this case you'd know there's an error. It's hard enough dealing with human generated invisible errors, but this just seems to perpetuate the LGTM crowd

hhh 1y ago
You can just have a different use case that surfaces hallucinations than someone, they don’t have to by evil.
simonw 1y ago
What were the hallucinations during the announcement? My incompetence here was that I was careless with my use of the term "hallucination" here. I assumed everyone else shared my exact definition - that a hallucination is when a model confidently states a fact that is entirely unconnected from reality, which is a different issue from a mistake ("how many Bs in blueberry" etc). It's clear that MANY people do not share my definition! I deeply regret including that note in my post.
- davidmurdoch 1y ago
  I subscribe to the same definition as you. I've actually never heard someone referring to the mistakes as hallucinating until now, but I can see how it's a bit of a grey area.
  - godelski 1y ago
    I'm actually curious about how you both come to those definitions of hallucinations. It gets very difficult to distinguish these things when you dig into them. Simon dropped this paper[0] in another thread[1] and while they provide a formal mathematical definition I don't think this makes it clear (I mean it is one person (who doesn't have a long publication record, PhD, or work at a university), but following their definition is still a bit muddy. They say the truth has to be in the training but don't clarify if they mean in training distribution or literal training example. To make a clear example, is the fact that when prompting GPT-5 with "Solve 5.x = x + 5.11" it answers "-0.21" (making the same mistake as when it GPT-4 says 5.11 > 5.9). Is that example specifically in the training data? Who knows! But are those types of problems in the training data? Absolutely! So is this a mistake or a hallucination? Should we really be using an answer that requires knowing the exact details of the training data? That would be fruitless and allow any hallucination to be claimed as a mistake. But in distribution? Well that works because we can know the types of problems trained on. It is also much more useful given that the reason we build these machines is for generalization. But even without that ambiguity I think it still gets difficult to differentiate a mistake from a hallucination. So it is unclear to me (and presumably others) what the precise distinction is to you and Simon. [0] https://arxiv.org/abs/2508.01781 https://arxiv.org/abs/2508.01781 [1] https://news.ycombinator.com/item?id=44831621 https://news.ycombinator.com/item?id=44831621
    - davidmurdoch 1y ago
      Yeah, I agree with you with you dig down into it. But I tend to instinctually (as a mere human) think of a "hallucination" as something more akin to a statement that feels like it could be true, and can't be verified by using only the surrounding context -- like when a human mis-remembers a fact on something they recently read, or extrapolates reasonably, but incorrectly. Example: GPT-5 just told me a few moments ago that webpack's "enhanced-resolve has an internal helper called getPackage.json". Webpack likely does contain logic that finds the package root, but it does not contain a file with this name, and never has. A reasonable person couldn't say with absolutely certainty that enhanced-resolve doesn't contain a file with that name. I think a "mistake" is classified as more of an error in computation, where all of the facts required to come up with a solution are present in the context of the conversation (simple arithmetic problems, "how many 'r's in strawberry", etc.), but it just does it wrong. I think of mistakes as something with one and only one valid answer. A person with the ability to make the computation themselves can recognize the mistake without further research. So hallucinations are more about conversational errors, and mistakes are more about computational errors, I guess? But again, I agree, it gets very difficult to distinguish these things when you dig into them.
- slagsnag 1y ago
  You must have missed the ridiculous graphs, or the Bernoulli error, while these corpo techno fascists were buying your dinner. https://news.ycombinator.com/item?id=44830684 https://news.ycombinator.com/item?id=44830684 https://news.ycombinator.com/item?id=44829144 https://news.ycombinator.com/item?id=44829144
  - simonw 1y ago
    The graphs were nothing to do with model hallucination, that was a crap design decision by a human being. The Bernoulli error was a case of a model spitting out widely believed existing misinformation. That doesn't fit my mental model of a "hallucination" either - I see a hallucination as a model inventing something that's not true with no basis in information it has been exposed to before. Here's an example of a hallucination in a demo: that time when Google Bard claimed that the James Webb Space Telescope was first to take pictures of planet outside Earth’s solar system. That's plain not true, and I doubt they had trained on text that said it was true.
    - gnowlescentic 1y ago
      I don't care what you call each failure mode. I want something that doesn't fail to give correct outputs 1/3 to 1/2 the time. Forget AI/AGI/ASI, forget "hallucinations", forget "scaling laws". Just give me software that does what it says it does, like writing code to spec.
      - godelski 1y ago
        Along those lines, I also want something that will correct me if I am wrong. The same way a human would or even the same way Google does because typing in something wrong usually has enough terms to get me to the right thing, though usually takes a bit longer. I definitely don't want something that will just go along with me when I'm wrong and reinforce a misconception. When I'm wrong I want to be corrected sooner than later, that's the only way to be less wrong.
        1y ago
- godelski 1y ago
  Considering your other comment, you may not consider it a hallucination but the fact about the airfoil was wrong. I'm sure there was information in the training that had the same mistake because that mistake exists in textbooks BUT I'm also confident that the correct fact is in the training as you can get GPT to reproduce the correct fact. The hallucination most likely happened because the prompt primes the model for the incorrect answer by asking about Bernoulli. But following that, the "airfoil" it generates for the simulation is symmetric. That is both inconsistent with its answers and inconsistent with reality, so I think that one is more clear. Similarly, in the coding demo the French guy even says that the snake doesn't look like a mouse haha.
- Breza 1y ago
  GPT-5 did the below today when I asked it to proofread an email. Would you consider it a hallucination? I'm asking genuinely because I'm a fan of your thoughts on tech and am curious where you draw the line. My wording: "Would you have time to talk the week of the 25th?" ChatGPT wording (elipses mine): "Could we schedule ~25 minutes the week of Aug 25 [...]? I’m free Tue 8/26 10:00–12:00 ET or Thu 8/28 2:00–4:00 ET, but happy to work around your calendar." I am not, in fact, free during those times. I have seen this exact kind of error multiple times.

hhh 1y ago

You can just have a different use case that surfaces hallucinations than someone, they don’t have to by evil.

simonw 1y ago

What were the hallucinations during the announcement? My incompetence here was that I was careless with my use of the term "hallucination" here. I assumed everyone else shared my exact definition - that a hallucination is when a model confidently states a fact that is entirely unconnected from reality, which is a different issue from a mistake ("how many Bs in blueberry" etc). It's clear that MANY people do not share my definition! I deeply regret including that note in my post.

davidmurdoch 1y ago
I subscribe to the same definition as you. I've actually never heard someone referring to the mistakes as hallucinating until now, but I can see how it's a bit of a grey area.
- godelski 1y ago
  I'm actually curious about how you both come to those definitions of hallucinations. It gets very difficult to distinguish these things when you dig into them. Simon dropped this paper[0] in another thread[1] and while they provide a formal mathematical definition I don't think this makes it clear (I mean it is one person (who doesn't have a long publication record, PhD, or work at a university), but following their definition is still a bit muddy. They say the truth has to be in the training but don't clarify if they mean in training distribution or literal training example. To make a clear example, is the fact that when prompting GPT-5 with "Solve 5.x = x + 5.11" it answers "-0.21" (making the same mistake as when it GPT-4 says 5.11 > 5.9). Is that example specifically in the training data? Who knows! But are those types of problems in the training data? Absolutely! So is this a mistake or a hallucination? Should we really be using an answer that requires knowing the exact details of the training data? That would be fruitless and allow any hallucination to be claimed as a mistake. But in distribution? Well that works because we can know the types of problems trained on. It is also much more useful given that the reason we build these machines is for generalization. But even without that ambiguity I think it still gets difficult to differentiate a mistake from a hallucination. So it is unclear to me (and presumably others) what the precise distinction is to you and Simon. [0] https://arxiv.org/abs/2508.01781 https://arxiv.org/abs/2508.01781 [1] https://news.ycombinator.com/item?id=44831621 https://news.ycombinator.com/item?id=44831621
  - davidmurdoch 1y ago
    Yeah, I agree with you with you dig down into it. But I tend to instinctually (as a mere human) think of a "hallucination" as something more akin to a statement that feels like it could be true, and can't be verified by using only the surrounding context -- like when a human mis-remembers a fact on something they recently read, or extrapolates reasonably, but incorrectly. Example: GPT-5 just told me a few moments ago that webpack's "enhanced-resolve has an internal helper called getPackage.json". Webpack likely does contain logic that finds the package root, but it does not contain a file with this name, and never has. A reasonable person couldn't say with absolutely certainty that enhanced-resolve doesn't contain a file with that name. I think a "mistake" is classified as more of an error in computation, where all of the facts required to come up with a solution are present in the context of the conversation (simple arithmetic problems, "how many 'r's in strawberry", etc.), but it just does it wrong. I think of mistakes as something with one and only one valid answer. A person with the ability to make the computation themselves can recognize the mistake without further research. So hallucinations are more about conversational errors, and mistakes are more about computational errors, I guess? But again, I agree, it gets very difficult to distinguish these things when you dig into them.
slagsnag 1y ago
You must have missed the ridiculous graphs, or the Bernoulli error, while these corpo techno fascists were buying your dinner. https://news.ycombinator.com/item?id=44830684 https://news.ycombinator.com/item?id=44830684 https://news.ycombinator.com/item?id=44829144 https://news.ycombinator.com/item?id=44829144
- simonw 1y ago
  The graphs were nothing to do with model hallucination, that was a crap design decision by a human being. The Bernoulli error was a case of a model spitting out widely believed existing misinformation. That doesn't fit my mental model of a "hallucination" either - I see a hallucination as a model inventing something that's not true with no basis in information it has been exposed to before. Here's an example of a hallucination in a demo: that time when Google Bard claimed that the James Webb Space Telescope was first to take pictures of planet outside Earth’s solar system. That's plain not true, and I doubt they had trained on text that said it was true.
  - gnowlescentic 1y ago
    I don't care what you call each failure mode. I want something that doesn't fail to give correct outputs 1/3 to 1/2 the time. Forget AI/AGI/ASI, forget "hallucinations", forget "scaling laws". Just give me software that does what it says it does, like writing code to spec.
    - godelski 1y ago
      Along those lines, I also want something that will correct me if I am wrong. The same way a human would or even the same way Google does because typing in something wrong usually has enough terms to get me to the right thing, though usually takes a bit longer. I definitely don't want something that will just go along with me when I'm wrong and reinforce a misconception. When I'm wrong I want to be corrected sooner than later, that's the only way to be less wrong.
      - 1y ago
godelski 1y ago
Considering your other comment, you may not consider it a hallucination but the fact about the airfoil was wrong. I'm sure there was information in the training that had the same mistake because that mistake exists in textbooks BUT I'm also confident that the correct fact is in the training as you can get GPT to reproduce the correct fact. The hallucination most likely happened because the prompt primes the model for the incorrect answer by asking about Bernoulli. But following that, the "airfoil" it generates for the simulation is symmetric. That is both inconsistent with its answers and inconsistent with reality, so I think that one is more clear. Similarly, in the coding demo the French guy even says that the snake doesn't look like a mouse haha.
Breza 1y ago
GPT-5 did the below today when I asked it to proofread an email. Would you consider it a hallucination? I'm asking genuinely because I'm a fan of your thoughts on tech and am curious where you draw the line. My wording: "Would you have time to talk the week of the 25th?" ChatGPT wording (elipses mine): "Could we schedule ~25 minutes the week of Aug 25 [...]? I’m free Tue 8/26 10:00–12:00 ET or Thu 8/28 2:00–4:00 ET, but happy to work around your calendar." I am not, in fact, free during those times. I have seen this exact kind of error multiple times.