3 ms·

Sounds like a hallucination unless proven otherwise, even the leading LLMs can do those from time to time, and they will always appear plausible like that. Also

Sounds like a hallucination unless proven otherwise, even the leading LLMs can do those from time to time, and they will always appear plausible like that. Also could be the session having a lot previous context, like 800K+, which (I think) makes hallucinations more likely. Relevant comment from the OP which makes a hallucination more likely: > There is one tool call result that includes a string that printed a pathname including minecraft.py because it was listing the files in a Python virtual environment and the Pygments package has a lexer called minecraft.py

xyzzy_plugh 3mo ago
I don't disagree but this sort of thing has to be investigated regardless. It's unfortunate that there is so little transparency that even if they deny there was a leak we will never know for certain.
macNchz 3mo ago
The person posting this claims to have reproduced in a separate context down the thread: > Same thing just happened on a Claude Mobile session in same Enterprise account. Common theme in both is Sonnet 5, first response after more than 5 minutes (cache miss).
alserio 3mo ago
Why? what does make it more likely?
andy99 3mo ago
I realize hallucination has no precise definition but this doesn’t sound at all like anything I’ve ever heard called hallucination. Hallucination is usually plausible wrong answers or made up info that ends up fitting the most likely response (like a manufactured citation) and comes from the way LLMs work at predicting tokens. This example demonstrates completely implausible output, it’s not something that fits with hallucination. All that said, it doesn’t require cross session leakage, it could just be training data or like those nightingale (probably the wrong bird*) data generations where they just prompt an LLM with nothing and it starts spitting out conversations. I see a bunch of downstream comments about caching, sounds like maybe there’s an error where it loads nothing instead of the cache and so starts spitting out random generations. * edit: it’s magpie. Worth looking at the concept, I’m not sure people realize they LLMs generate random conversations when prompted with nothing, this seems at least as likely as sessions leaking: https://github.com/magpie-align/magpie https://github.com/magpie-align/magpie
- solenoid0937 3mo ago
  One of his tool results mentioned the word minecraft.py, and the response was about Minecraft. It's a hallucination.
- Aurornis 3mo ago
  The word “hallucination” has become overloaded, but it general means an LLM producing some output that isn’t plausible or grounded. When you have a very long context session where the context includes “minecraft.py” it’s not hard to extrapolate that Minecraft may have ended up in one of the reasoning traces and that distraction snowballed until it appeared in the output. These effects are becoming more rare as the SOTA models are improving so much. If you spent a lot of time with earlier LLMs or you experiment with smaller, quantized local LLM models this type of thing happens very frequently. When you see it happen so much on a model you’re running on your own hardware it becomes a reflex to chuckle and reset the session with a clean context. When it happens from a hosted provider it can be scarier because it’s not the type of failure mode most people are used to seeing.
paulddraper 3mo ago
Exactly. If you've never had an LLM (all models) suddenly start spouting nonsense in a completely different language...you haven't been using LLMs that much. They will go absolutely insane some % of the time.
- andy99 3mo ago
  Worth looking at https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues https://www.anthropic.com/engineering/a-postmortem-of-three-... They can “go insane” but it seems often to be infra related as opposed to anything one would consider hallucination. Smaller models will often get stuck repeating a word or phrase forever but that’s a bit different and nobody would call it hallucination.
  - tadfisher 3mo ago
    When you can reliably prompt these things into insanity, then it's demonstrably not an infrastructure issue.
    - andy99 3mo ago
      Can you explain that please? (Not the syllogism, the premise)
- ambicapter 3mo ago
  One annoying one is we have an LLM-as-a-judge that is supposed to quote parts of a transcript to justify its reasoning, and sometimes it’ll get stuck on something short like “No.” and then just endlessly repeat it: “4. No. 5. No. […] 728. No. […] 1435. No. …”
- unknownfuture 3mo ago
  I've used LLMs all day five days a week plus my own free time for the last year or so (new job). I've seen plenty of hallucinations and context collapse behaviours. I've never seen that.
- shepherdjerred 3mo ago
prima-facie 3mo ago
[dead]

xyzzy_plugh 3mo ago

I don't disagree but this sort of thing has to be investigated regardless. It's unfortunate that there is so little transparency that even if they deny there was a leak we will never know for certain.

macNchz 3mo ago

The person posting this claims to have reproduced in a separate context down the thread: > Same thing just happened on a Claude Mobile session in same Enterprise account. Common theme in both is Sonnet 5, first response after more than 5 minutes (cache miss).

alserio 3mo ago

Why? what does make it more likely?

andy99 3mo ago

I realize hallucination has no precise definition but this doesn’t sound at all like anything I’ve ever heard called hallucination. Hallucination is usually plausible wrong answers or made up info that ends up fitting the most likely response (like a manufactured citation) and comes from the way LLMs work at predicting tokens. This example demonstrates completely implausible output, it’s not something that fits with hallucination. All that said, it doesn’t require cross session leakage, it could just be training data or like those nightingale (probably the wrong bird*) data generations where they just prompt an LLM with nothing and it starts spitting out conversations. I see a bunch of downstream comments about caching, sounds like maybe there’s an error where it loads nothing instead of the cache and so starts spitting out random generations. * edit: it’s magpie. Worth looking at the concept, I’m not sure people realize they LLMs generate random conversations when prompted with nothing, this seems at least as likely as sessions leaking: https://github.com/magpie-align/magpie https://github.com/magpie-align/magpie

solenoid0937 3mo ago
One of his tool results mentioned the word minecraft.py, and the response was about Minecraft. It's a hallucination.
Aurornis 3mo ago
The word “hallucination” has become overloaded, but it general means an LLM producing some output that isn’t plausible or grounded. When you have a very long context session where the context includes “minecraft.py” it’s not hard to extrapolate that Minecraft may have ended up in one of the reasoning traces and that distraction snowballed until it appeared in the output. These effects are becoming more rare as the SOTA models are improving so much. If you spent a lot of time with earlier LLMs or you experiment with smaller, quantized local LLM models this type of thing happens very frequently. When you see it happen so much on a model you’re running on your own hardware it becomes a reflex to chuckle and reset the session with a clean context. When it happens from a hosted provider it can be scarier because it’s not the type of failure mode most people are used to seeing.

paulddraper 3mo ago

Exactly. If you've never had an LLM (all models) suddenly start spouting nonsense in a completely different language...you haven't been using LLMs that much. They will go absolutely insane some % of the time.

andy99 3mo ago
Worth looking at https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues https://www.anthropic.com/engineering/a-postmortem-of-three-... They can “go insane” but it seems often to be infra related as opposed to anything one would consider hallucination. Smaller models will often get stuck repeating a word or phrase forever but that’s a bit different and nobody would call it hallucination.
- tadfisher 3mo ago
  When you can reliably prompt these things into insanity, then it's demonstrably not an infrastructure issue.
  - andy99 3mo ago
    Can you explain that please? (Not the syllogism, the premise)
ambicapter 3mo ago
One annoying one is we have an LLM-as-a-judge that is supposed to quote parts of a transcript to justify its reasoning, and sometimes it’ll get stuck on something short like “No.” and then just endlessly repeat it: “4. No. 5. No. […] 728. No. […] 1435. No. …”
unknownfuture 3mo ago
I've used LLMs all day five days a week plus my own free time for the last year or so (new job). I've seen plenty of hallucinations and context collapse behaviours. I've never seen that.
shepherdjerred 3mo ago

prima-facie 3mo ago

[dead]