4 ms·

Well it's not only this, or protection from LLMs training on LLM output. LLMs training on human output is also problematic. I was traveling to an obscure small

by mstaoru 26d ago

Well it's not only this, or protection from LLMs training on LLM output. LLMs training on human output is also problematic. I was traveling to an obscure small town, doing some "research" with LLMs beforehand. Every and each one told me enthusiastically to go to "Foobar square" (name changed) for the "best street food in XYZ town", some added a lot of colorful details. There was no Foobar square in XYZ town. There was no Foobar square anywhere in the world. There was a SINGLE old Reddit comment, with no upvotes, to a unpopular post in an unpopular subreddit, where someone clearly badly misspelled the name of the square, and said something like "for street food go to Foobar square". Nothing about "the best" even. It's all a lie.

SoftTalker 26d ago
I think this was a common game on city/town subs. It happened here, there was a post asking for a good restaurant and someone just made up a name. It went viral and people started posting made-up menus for the place, reviews, and for a couple of months any time someone asked about a restaurant this fictional place would get mentioned. It was all done as a joke to see if they could get Gemini or ChatGPT to start recommending it.
- morkalork 26d ago
  The Montréal subreddit has been doing this for ages before LLMs were a thing because every summer and fall there's endless threads from tourists and students asking the same questions that recommending a local gay bathhouse became the meme answer.
- fer 25d ago
  A friend did some vandalism on Wikipedia 20 years ago (!), and yet, LLMs quote his "original research".
- zerd 23d ago
  Time to start some new restaurants matching those. Like Bubba Gump.
consp 26d ago
I've had gemini claiming code would compile and run while also outputting the same variable in the same sniplet with "fork" "frok" and "fokr" in the name. I'm not surprized it's trained on garbadge.
thephyber 26d ago
Your one example doesn't make all of LLMs a lie. It's like someone reading a National Enquirer article about "Hillary Clinton being an alien from outer space" (a real headline topic from decades ago) and drawling the conclusion that all journalism is "a lie". The user has to understand media literacy and be at least a little skeptical of the claims that are made, then cross reference with another source.
- fluoridation 25d ago
  Okay, but the anecdote states that every model repeated the pseudo-factoid about Foobar square, not just the 4 GB open source model equivalent of a tabloid.
  - CamperBob2 25d ago
    Without disclosing what you were prompting for, it's impossible to evaluate your claim.
    - fluoridation 25d ago
      Correct. We can either accept the claim or disregard it. The comment I replied to opted to accept it and then committed a fallacy, hence my response.
  - nulbyte 25d ago
    I think the key phrase here is, "an obscure small town." There may only be a single mention of this place, hence the only one on which a response can be based. This says more about the user's understanding of LLMs than it does about LLMs.
    - CamperBob2 25d ago
      This says more about the user's understanding of LLMs than it does about LLMs. "Tell me everything you know about (obscure small town), (state). Only what's unique to (town), not commonly-known facts" is an excellent way to test for hallucinatory tendencies in a new model, in my experience. Likely the best I've found. Quality of results is almost linearly proportional to the size of the model in many cases. The largest models like K3 and GLM 5.3 will either confine their responses to known true facts about the town and its surroundings, or admit they don't have enough information to answer. Smaller ones will reliably make up hilarious or downright-strange things. Another good test is https://whatever.scalzi.com/2025/12/13/ai-a-dedicated-fact-failing-machine-or-yet-another-reason-not-to-trust-it-for-anything/ https://whatever.scalzi.com/2025/12/13/ai-a-dedicated-fact-f... , which still works on the newest models. Of the open-weight models available, only Kimi K3 will consistently admit it has no idea who Scalzi's novel is dedicated to. The rest still make up random stuff and present it confidently. TL,DR: progress is possible, and it has been made, but it's happening slower than many people think.
evilfred 25d ago
LLMs have no sense of truth, they are just next token generators

by mstaoru 26d ago

SoftTalker 26d ago

I think this was a common game on city/town subs. It happened here, there was a post asking for a good restaurant and someone just made up a name. It went viral and people started posting made-up menus for the place, reviews, and for a couple of months any time someone asked about a restaurant this fictional place would get mentioned. It was all done as a joke to see if they could get Gemini or ChatGPT to start recommending it.

morkalork 26d ago
The Montréal subreddit has been doing this for ages before LLMs were a thing because every summer and fall there's endless threads from tourists and students asking the same questions that recommending a local gay bathhouse became the meme answer.
fer 25d ago
A friend did some vandalism on Wikipedia 20 years ago (!), and yet, LLMs quote his "original research".
zerd 23d ago
Time to start some new restaurants matching those. Like Bubba Gump.

consp 26d ago

I've had gemini claiming code would compile and run while also outputting the same variable in the same sniplet with "fork" "frok" and "fokr" in the name. I'm not surprized it's trained on garbadge.

thephyber 26d ago

Your one example doesn't make all of LLMs a lie. It's like someone reading a National Enquirer article about "Hillary Clinton being an alien from outer space" (a real headline topic from decades ago) and drawling the conclusion that all journalism is "a lie". The user has to understand media literacy and be at least a little skeptical of the claims that are made, then cross reference with another source.

fluoridation 25d ago
Okay, but the anecdote states that every model repeated the pseudo-factoid about Foobar square, not just the 4 GB open source model equivalent of a tabloid.
- CamperBob2 25d ago
  Without disclosing what you were prompting for, it's impossible to evaluate your claim.
  - fluoridation 25d ago
    Correct. We can either accept the claim or disregard it. The comment I replied to opted to accept it and then committed a fallacy, hence my response.
- nulbyte 25d ago
  I think the key phrase here is, "an obscure small town." There may only be a single mention of this place, hence the only one on which a response can be based. This says more about the user's understanding of LLMs than it does about LLMs.
  - CamperBob2 25d ago
    This says more about the user's understanding of LLMs than it does about LLMs. "Tell me everything you know about (obscure small town), (state). Only what's unique to (town), not commonly-known facts" is an excellent way to test for hallucinatory tendencies in a new model, in my experience. Likely the best I've found. Quality of results is almost linearly proportional to the size of the model in many cases. The largest models like K3 and GLM 5.3 will either confine their responses to known true facts about the town and its surroundings, or admit they don't have enough information to answer. Smaller ones will reliably make up hilarious or downright-strange things. Another good test is https://whatever.scalzi.com/2025/12/13/ai-a-dedicated-fact-failing-machine-or-yet-another-reason-not-to-trust-it-for-anything/ https://whatever.scalzi.com/2025/12/13/ai-a-dedicated-fact-f... , which still works on the newest models. Of the open-weight models available, only Kimi K3 will consistently admit it has no idea who Scalzi's novel is dedicated to. The rest still make up random stuff and present it confidently. TL,DR: progress is possible, and it has been made, but it's happening slower than many people think.

evilfred 25d ago

LLMs have no sense of truth, they are just next token generators