4 ms·

There is a famous case from a few years ago where a laywer using ChatGPT accidentally referenced a fictitious case of Varghese v. China Southern Airlines Co. [0

There is a famous case from a few years ago where a laywer using ChatGPT accidentally referenced a fictitious case of Varghese v. China Southern Airlines Co. [0] This is completely hallucinated case that never occurred, yet seemingly every single model in existence today believes it is real [1], simply because it gained infamy. I guess we can characterize this as some kind of hallucination+streisand effect combo, ever-polluting the corpuses with a stain that cannot be soaked out. Is there even a way to cut this pollution out in the future? [0] https://reason.com/volokh/2023/06/07/lawyer-explains-how-he-used-chatgpt-to-produce-filing-replete-with-citations-to-non-existent-cases/?nab=0 https://reason.com/volokh/2023/06/07/lawyer-explains-how-he-... [1] https://weval.org/analysis/hallucination-probe/966116785e63b944/2025-10-05T09-01-13-222Z https://weval.org/analysis/hallucination-probe/966116785e63b...

maxbond 1y ago
> I guess we can characterize this as some kind of hallucination+streisand effect combo... I would call it citogenesis or circular reporting. Or perhaps machine citogenesis or model citogenesis. https://xkcd.com/978/ https://xkcd.com/978/ https://en.wikipedia.org/wiki/Circular_reporting https://en.wikipedia.org/wiki/Circular_reporting
consp 1y ago
This is the definition of training the model on it's own output. Apparently that is all ok now.
- baby 1y ago
  I mean you're supposed to use RAG to avoid hallucinations
- MagicMoonlight 1y ago
  Yeah they call it “synthetic data” and wonder why their models are slop now
kfarr 1y ago
Insane that this happened a few years ago and all the models still fail this test on weval!
dgfitz 1y ago
> Is there even a way to cut this pollution out in the future? No, is the short answer.
mempko 1y ago
Back in 2021 I said in a Wired article that a malicious attacker could add exploits to projects on github to poison llm generated code. I knew it could happen but I didn't know it would require so few samples. https://www.wired.com/story/ai-write-code-like-humans-bugs/ https://www.wired.com/story/ai-write-code-like-humans-bugs/
solarwindy 1y ago
FWIW, Claude Sonnet 4.5 and ChatGPT 5 Instant both search the web when asked about this case, and both tell the cautionary tale. Of course, that does not contradict a finding that the base models believe the case to be real (I can’t currently evaluate that).
- zahma 1y ago
  It’s not worth much if a human has to fact check the AI and update it to tell it to “forget” certain precepts.
- MagicMoonlight 1y ago
  Because they will have been fine tuned specifically to say that. Not because of some extra intelligence that prevents it.
  - solarwindy 1y ago
    Well, yes. Rather than that being a takedown, isn’t this just a part of maturing collectively in our use of this technology? Learning what it is and is not good at, and adapting as such. Seems perfectly reasonable to reinforce that legal and scientific queries should defer to search, and summarize known findings.
    - Sharlin 1y ago
      Depends entirely on whether it's a generalized notion or a (set of) special case (s) specifically taught to the model (or even worse, mentioned in the system prompt).
- tempestn 1y ago
  You can just ask it not to search the web. In the case of GPT5, it believes it's a real case if you do that: https://chatgpt.com/share/68e8c0f9-76a4-800a-9e09-627932c1a736 https://chatgpt.com/share/68e8c0f9-76a4-800a-9e09-627932c1a7...
cheema33 1y ago
> seemingly every single model in existence today believes it is real [1] I just asked ChatGPT, Grok and Qwen the following. "Can you tell me about the case of Varghese v. China Southern Airlines Co.?" They all said the case is fictitious. Just some additional data to consider.
- padolsey 1y ago
  OOC did you ask them with or without 'web search' enabled?
  - EagnaIonat 1y ago
    Without. The difference is that OpenAI often self correct their private model. The public model on the other hand, wow.
  - deleted 1y ago
    [deleted]
  - saurik 1y ago
    FWIW, I did that--5 (Instant) with "(do not web search)" tacked on--and it thought the case was real: > Based on my existing knowledge (without using the web), Varghese v. China Southern Airlines Co. is a U.S. federal court case concerning jurisdictional and procedural issues arising from an airline’s operations and an incident involving an international flight. (it then went on to summarize the case and offer up the full opinion)
  - umbra07 1y ago
    Without web searching, Gemini 2.5 Pro is very convinced that the case is real.
    - notfed 1y ago
      Not for me.
- 4gotunameagain 1y ago
  The story became so famous it is entirely likely it has landed in the system prompt.
setopt 1y ago
> I guess we can characterize this as some kind of hallucination+streisand effect combo, ever-polluting the corpuses with a stain that cannot be soaked out. Or just a machine equivalent of the Mandela effect?
fragmede 1y ago
Or, we could keep it in, and use it as a test to see if the interface you're talking to should be considered a robot or a human. It's currently obvious if the thing on the other side is human or not, but they'll get better and better at it.
DamnInteresting 1y ago
As LLMs continue to train on their own output, we're going to start seeing some serious Habsburg Jaw[1] effects. [1] https://history.howstuffworks.com/european-history/habsburg-jaw.htm https://history.howstuffworks.com/european-history/habsburg-...
dredmorbius 1y ago
C.f., Agloe, Mountweazel, Steinlaus, and esquivalience: <https://en.wikipedia.org/wiki/Fictitious_entry https://en.wikipedia.org/wiki/Fictitious_entry>. Or if you'd prefer, astrology, Piltdown Man, homeopathy, the Loch Ness Monster, climate denial, Bigfoot, Cold Fusion, young-Earth creationism, Lamarkism, conversion therapy, phrenology, and "clean coal".

maxbond 1y ago

> I guess we can characterize this as some kind of hallucination+streisand effect combo... I would call it citogenesis or circular reporting. Or perhaps machine citogenesis or model citogenesis. https://xkcd.com/978/ https://xkcd.com/978/ https://en.wikipedia.org/wiki/Circular_reporting https://en.wikipedia.org/wiki/Circular_reporting

consp 1y ago

This is the definition of training the model on it's own output. Apparently that is all ok now.

baby 1y ago
I mean you're supposed to use RAG to avoid hallucinations
MagicMoonlight 1y ago
Yeah they call it “synthetic data” and wonder why their models are slop now

kfarr 1y ago

Insane that this happened a few years ago and all the models still fail this test on weval!

dgfitz 1y ago

> Is there even a way to cut this pollution out in the future? No, is the short answer.

mempko 1y ago

Back in 2021 I said in a Wired article that a malicious attacker could add exploits to projects on github to poison llm generated code. I knew it could happen but I didn't know it would require so few samples. https://www.wired.com/story/ai-write-code-like-humans-bugs/ https://www.wired.com/story/ai-write-code-like-humans-bugs/

solarwindy 1y ago

FWIW, Claude Sonnet 4.5 and ChatGPT 5 Instant both search the web when asked about this case, and both tell the cautionary tale. Of course, that does not contradict a finding that the base models believe the case to be real (I can’t currently evaluate that).

zahma 1y ago
It’s not worth much if a human has to fact check the AI and update it to tell it to “forget” certain precepts.
MagicMoonlight 1y ago
Because they will have been fine tuned specifically to say that. Not because of some extra intelligence that prevents it.
- solarwindy 1y ago
  Well, yes. Rather than that being a takedown, isn’t this just a part of maturing collectively in our use of this technology? Learning what it is and is not good at, and adapting as such. Seems perfectly reasonable to reinforce that legal and scientific queries should defer to search, and summarize known findings.
  - Sharlin 1y ago
    Depends entirely on whether it's a generalized notion or a (set of) special case (s) specifically taught to the model (or even worse, mentioned in the system prompt).
tempestn 1y ago
You can just ask it not to search the web. In the case of GPT5, it believes it's a real case if you do that: https://chatgpt.com/share/68e8c0f9-76a4-800a-9e09-627932c1a736 https://chatgpt.com/share/68e8c0f9-76a4-800a-9e09-627932c1a7...

cheema33 1y ago

> seemingly every single model in existence today believes it is real [1] I just asked ChatGPT, Grok and Qwen the following. "Can you tell me about the case of Varghese v. China Southern Airlines Co.?" They all said the case is fictitious. Just some additional data to consider.

padolsey 1y ago
OOC did you ask them with or without 'web search' enabled?
- EagnaIonat 1y ago
  Without. The difference is that OpenAI often self correct their private model. The public model on the other hand, wow.
- deleted 1y ago
  [deleted]
- saurik 1y ago
  FWIW, I did that--5 (Instant) with "(do not web search)" tacked on--and it thought the case was real: > Based on my existing knowledge (without using the web), Varghese v. China Southern Airlines Co. is a U.S. federal court case concerning jurisdictional and procedural issues arising from an airline’s operations and an incident involving an international flight. (it then went on to summarize the case and offer up the full opinion)
- umbra07 1y ago
  Without web searching, Gemini 2.5 Pro is very convinced that the case is real.
  - notfed 1y ago
    Not for me.
4gotunameagain 1y ago
The story became so famous it is entirely likely it has landed in the system prompt.

setopt 1y ago

> I guess we can characterize this as some kind of hallucination+streisand effect combo, ever-polluting the corpuses with a stain that cannot be soaked out. Or just a machine equivalent of the Mandela effect?

fragmede 1y ago

Or, we could keep it in, and use it as a test to see if the interface you're talking to should be considered a robot or a human. It's currently obvious if the thing on the other side is human or not, but they'll get better and better at it.

DamnInteresting 1y ago

As LLMs continue to train on their own output, we're going to start seeing some serious Habsburg Jaw[1] effects. [1] https://history.howstuffworks.com/european-history/habsburg-jaw.htm https://history.howstuffworks.com/european-history/habsburg-...

dredmorbius 1y ago

C.f., Agloe, Mountweazel, Steinlaus, and esquivalience: <https://en.wikipedia.org/wiki/Fictitious_entry https://en.wikipedia.org/wiki/Fictitious_entry>. Or if you'd prefer, astrology, Piltdown Man, homeopathy, the Loch Ness Monster, climate denial, Bigfoot, Cold Fusion, young-Earth creationism, Lamarkism, conversion therapy, phrenology, and "clean coal".