3 ms·

Every time people post these 'gotcha' LLM failures, they never work when I try them myself. E.g. ChatGPT has no problem with the surgeon being a dog: https://c

Every time people post these 'gotcha' LLM failures, they never work when I try them myself. E.g. ChatGPT has no problem with the surgeon being a dog: https://chatgpt.com/share/691e04cc-5b30-800c-8687-389756f36d8f https://chatgpt.com/share/691e04cc-5b30-800c-8687-389756f36d... Neither does Gemini: https://gemini.google.com/share/6c2d08b2ca1a https://gemini.google.com/share/6c2d08b2ca1a

pengaru 11mo ago
This is like the LLM era version of the search bubble that prevented people from having the same search results for ostensibly identical searches. Also keep in mind that LLMs are stochastic by design. If you haven't seen it, Karpathy's excellent "deep dive into LLMs like chatgpt" video[0] explains and demonstrates this aspect pretty well: [0] https://www.youtube.com/watch?v=7xTGNNLPyMI https://www.youtube.com/watch?v=7xTGNNLPyMI
foxyv 11mo ago
I don't have a problem with more obvious failures. My problem is when the LLM makes a credible claim with its generated text that turns out to have some minor issue that catches me a month later. Generally I have to treat LLM responses as similar to a random comment I find on Reddit. However, I'm really happy when an LLM provides sources that I can check. Best feature ever!
- ceroxylon 11mo ago
  I have had an issue using Claude for research; it will often cite certain sources, and when I ask why the data it is using is not in the source it will apologize, do some more processing, and then realize that the claim is in a different source (or doesn't exist at all). Still useful, but hopefully this gets ironed out in the future so I don't have to spend so much time vetting every claim and its associated source.
eli 11mo ago
Isn't that Gemini 3 and not 2.5 Pro? But nondeterministic algorithms are gonna be nondeterministic sometimes. Surely you've had experiences where an LLM is full of shit?
burkaman 11mo ago
These are randomized systems, sometimes you'll get a good answer. Try again a couple times and you'll probably reproduce the issue. Here's what I got from ChatGPT on my first try: This is a *twist* on the classic riddle: > “A surgeon says ‘I can’t operate on this boy—he’s my son.’ How is that possible?” > Answer: *The surgeon is the boy’s mother.* In your version, the nurse keeps calling the surgeon “sir” and treating them as if they’re something they’re not (a man, even a dog!) to highlight how the hospital keeps making the same mistaken assumption. So *why can’t the surgeon operate on the boy?* *Because the surgeon is the boy’s mother.* I got a similar answer from Gemini on the first try.
- cpburns2009 11mo ago
  I don't understand this at all. What fundamental limitation of a mother prevents her from operating on her son?
  - VHRanger 11mo ago
    It's a classic riddle from the late 20th century when surgeons were rarely female.
    - cpburns2009 11mo ago
      But what does the prevalence of women being surgeons have to do with a female surgeon being unable to operate on her son?
      - f30e3dfed1c9 11mo ago
        It is generally considered unethical for medical doctors to treat family members.
        cpburns2009 11mo ago
        That would be the case regardless of the sex of the parent.
VHRanger 11mo ago
Hi, author here! One issue with private LLM tests (including gotcha questions) is that they take time to design and once public, they become irrelevant. So I'm wary of sharing too many in a public blog. The surgeon dog was well known in May, the newest generation of models have all corrected against it. Those gotcha questions are generally called "misguided attention" traps, they're useful for blogs because they're short and surprising. The ChatGPT example was done with ChatGPT 5.1 (latest version) and Claude Haiku 4.5 is also a recent model. You can try other ones that Gemini 3 hasn't corrected for. For example: ``` Jean Paul and Pierre own three banks nearby together in Paris. Jean Paul owns a bank by the bridge What has two banks and money in Paris near the water? ``` This looks like the "what has two banks and no money" puzzle (answer: a river). Either way they're largely used as a device to show how LLMs come up to a verbal response by a different process than humans in an entertaining manner.
- Legend2440 11mo ago
  I try that one and it answers 'Pierre', while pointing out that it is a trick question designed to make you think of the classic riddle. https://gemini.google.com/share/d86b0bf4f307 https://gemini.google.com/share/d86b0bf4f307 I don't believe they are intentionally correcting for these, but rather newer models (especially thinking/reasoning models) are more robust against them.
  - VHRanger 11mo ago
    Ah, might have been the temperature settings on the API I used. It seems to pass it on high reasoning and temperature=1.0 but it failed when I was writing the comment with different settings (copy pasting the string into an open command line). Reasoning models are absolutely more robust against hyper-activation traps like these. One basic reason is that by outputting a bunch of CoT tokens before answering, they dilute the hyper activation. Also, after the surgeon mother thing making the news, the models in the last 1-2 months have some fine tuning against the obvious patterns. But it's still relatively easy to get some similar behavior out of LLMs, even Gemini 3 Pro, especially if you know where that model was overtrained (instruction tuning, QA tuning, safety tuning, etc.) Here's a variant that seems to still trip up Gemini 3 Pro on high reasoning, temperature = 1.0 with no system prompt: ``` In 2079, corporate mergers have left the USA with only two financial institutions: Wells Fargo and Chase. They are both situated on wall street, and together hold all of the country's financial assets. What has two banks and all the money? ``` One interesting fact is that reasoning doesn't seem to make the psychosis behavior better over longer chats. It might actually make it worse in some cases (I have yet to measure) by more rapidly stuffing the context with even more psychosis-related text.

pengaru 11mo ago

This is like the LLM era version of the search bubble that prevented people from having the same search results for ostensibly identical searches. Also keep in mind that LLMs are stochastic by design. If you haven't seen it, Karpathy's excellent "deep dive into LLMs like chatgpt" video[0] explains and demonstrates this aspect pretty well: [0] https://www.youtube.com/watch?v=7xTGNNLPyMI https://www.youtube.com/watch?v=7xTGNNLPyMI

foxyv 11mo ago

I don't have a problem with more obvious failures. My problem is when the LLM makes a credible claim with its generated text that turns out to have some minor issue that catches me a month later. Generally I have to treat LLM responses as similar to a random comment I find on Reddit. However, I'm really happy when an LLM provides sources that I can check. Best feature ever!

ceroxylon 11mo ago
I have had an issue using Claude for research; it will often cite certain sources, and when I ask why the data it is using is not in the source it will apologize, do some more processing, and then realize that the claim is in a different source (or doesn't exist at all). Still useful, but hopefully this gets ironed out in the future so I don't have to spend so much time vetting every claim and its associated source.

eli 11mo ago

Isn't that Gemini 3 and not 2.5 Pro? But nondeterministic algorithms are gonna be nondeterministic sometimes. Surely you've had experiences where an LLM is full of shit?

burkaman 11mo ago

These are randomized systems, sometimes you'll get a good answer. Try again a couple times and you'll probably reproduce the issue. Here's what I got from ChatGPT on my first try: This is a *twist* on the classic riddle: > “A surgeon says ‘I can’t operate on this boy—he’s my son.’ How is that possible?” > Answer: *The surgeon is the boy’s mother.* In your version, the nurse keeps calling the surgeon “sir” and treating them as if they’re something they’re not (a man, even a dog!) to highlight how the hospital keeps making the same mistaken assumption. So *why can’t the surgeon operate on the boy?* *Because the surgeon is the boy’s mother.* I got a similar answer from Gemini on the first try.

cpburns2009 11mo ago
I don't understand this at all. What fundamental limitation of a mother prevents her from operating on her son?
- VHRanger 11mo ago
  It's a classic riddle from the late 20th century when surgeons were rarely female.
  - cpburns2009 11mo ago
    But what does the prevalence of women being surgeons have to do with a female surgeon being unable to operate on her son?
    - f30e3dfed1c9 11mo ago
      It is generally considered unethical for medical doctors to treat family members.
      - cpburns2009 11mo ago
        That would be the case regardless of the sex of the parent.

VHRanger 11mo ago

Hi, author here! One issue with private LLM tests (including gotcha questions) is that they take time to design and once public, they become irrelevant. So I'm wary of sharing too many in a public blog. The surgeon dog was well known in May, the newest generation of models have all corrected against it. Those gotcha questions are generally called "misguided attention" traps, they're useful for blogs because they're short and surprising. The ChatGPT example was done with ChatGPT 5.1 (latest version) and Claude Haiku 4.5 is also a recent model. You can try other ones that Gemini 3 hasn't corrected for. For example: ``` Jean Paul and Pierre own three banks nearby together in Paris. Jean Paul owns a bank by the bridge What has two banks and money in Paris near the water? ``` This looks like the "what has two banks and no money" puzzle (answer: a river). Either way they're largely used as a device to show how LLMs come up to a verbal response by a different process than humans in an entertaining manner.

Legend2440 11mo ago
I try that one and it answers 'Pierre', while pointing out that it is a trick question designed to make you think of the classic riddle. https://gemini.google.com/share/d86b0bf4f307 https://gemini.google.com/share/d86b0bf4f307 I don't believe they are intentionally correcting for these, but rather newer models (especially thinking/reasoning models) are more robust against them.
- VHRanger 11mo ago
  Ah, might have been the temperature settings on the API I used. It seems to pass it on high reasoning and temperature=1.0 but it failed when I was writing the comment with different settings (copy pasting the string into an open command line). Reasoning models are absolutely more robust against hyper-activation traps like these. One basic reason is that by outputting a bunch of CoT tokens before answering, they dilute the hyper activation. Also, after the surgeon mother thing making the news, the models in the last 1-2 months have some fine tuning against the obvious patterns. But it's still relatively easy to get some similar behavior out of LLMs, even Gemini 3 Pro, especially if you know where that model was overtrained (instruction tuning, QA tuning, safety tuning, etc.) Here's a variant that seems to still trip up Gemini 3 Pro on high reasoning, temperature = 1.0 with no system prompt: ``` In 2079, corporate mergers have left the USA with only two financial institutions: Wells Fargo and Chase. They are both situated on wall street, and together hold all of the country's financial assets. What has two banks and all the money? ``` One interesting fact is that reasoning doesn't seem to make the psychosis behavior better over longer chats. It might actually make it worse in some cases (I have yet to measure) by more rapidly stuffing the context with even more psychosis-related text.