6 ms·

I won a championship that doesn't exist

232 points by SEJeff 5mo ago

amarant 5mo ago
"Stoner became the first American world champion...." Even being on stoner.com,I read that as meaning something different from what was meant. Op has a great surname!
- fsterneder 5mo ago
  Not only a great surname but also a very great novel too!
simonw 5mo ago
You don't need to vandalize Wikipedia to get this kind of thing to work. Back in September 2024 I named a whale "Teresa T" with just a blog entry and a YouTube video caption: https://simonwillison.net/2024/Sep/8/teresa-t-whale-pillar-point/ https://simonwillison.net/2024/Sep/8/teresa-t-whale-pillar-p... (For a few glorious weeks if you asked any search-enabled LLM, including Google search previews, for the name of the whale in the Half Moon Bay harbor it confidently replied Teresa T)
- bitwize 5mo ago
  The Mr. Splashy Pants of the AI era!
- slater 5mo ago
  (it probably helps that your name & blog carry some weight, vs. some rando writing something on blogspot or wordpress ;) )
  - Forgeties79 5mo ago
    Which illustrates another problem: unscrupulous actors with big names can spread whatever information they want to millions of people with minimal effort.
    - MassPikeMike 5mo ago
      Ever since the invention of the printing press, every new communication technology has reduced the effort needed to widely disseminate information-- and misinformation! So you could say this is nothing new. On the other hand, this is remarkably little effort.
    - nomdep 5mo ago
      Yes, they can. We can be glad that respectable newspapers and TV news channels have never done it and never will. You can even trust than the headlines are accurate summaries of the content of the articles. /s
      - saghm 5mo ago
        The existence of a problem in one area doesn't mean that it's not also a problem for it to spread somewhere else
      - Forgeties79 5mo ago
        I started writing a response and realized I basically wrote the exact same thing the other day https://news.ycombinator.com/item?id=47921829 https://news.ycombinator.com/item?id=47921829
    - simonw 5mo ago
      Exactly. I chose to abuse my platform to promote Teresa T as the name of a whale.
      - Forgeties79 5mo ago
        Oh god I just realized the implication! I was not directing that at you haha
        simonw 5mo ago
        No I really did abuse my reach for this one! I figured it would be a relatively harmless demo of how easy it is to affect LLM answers if you have a decently trafficked website.
- pesus 5mo ago
  Google still shows Theresa T as the name when you search.
- sb057 5mo ago
  I mean, the name of that whale is now Teresa T. You gave it that name.
  - latexr 5mo ago
    And your name is now Berningular Farshthruster III. I gave you that name. Which is, of course, silly. It is a name for you, just like Teresa T is a name for the whale, but it’s not your/their name, just like the RRS Sir David Attenborough is not named Boaty McBoatface (to the chagrin of most). Simon does not have the authority to unilaterally¹ name the whale (which is why the exercise makes sense). ¹ Important point. If the name started being recognised and used by consensus of those with the purview to do so (much like the thagomizer²), then Simon would have named the whale, but it would only become its name at that point. ² https://en.wikipedia.org/wiki/Thagomizer https://en.wikipedia.org/wiki/Thagomizer
    - pinkmuffinere 5mo ago
      > Simon does not have the authority to unilaterally¹ name the whale There's no such thing as authority to name a whale, and anyways I don't believe authority is strictly needed. A name is what people use to refer to something, full stop. It is only required that names become common-ish parlance; the more well known they are, the more they feel like the 'real' name. The inverse of Ohms is named Mhos (imo much more recognizable than the official name, "siemens"). The "#" symbol is named the hashtag, octothorp, pound sign, tic-tac-toe, number sign, and probably a million other things. Which one of these is the "real" primary name? I think intuitively we know that the real one is whatever people around us are most familiar with. You should take a guess, and I'll put the wikipedia-suggested-answer in the footnotes [1]. I bet your name for it is different than the 'official' wikipedia suggestion. In the case of the whale, the _only_ name that is associated with that whale is Teresa T. I think this immediately makes it the most valid name of that whale. [1] wikipedia says this is the number sign: https://en.wikipedia.org/wiki/Number_sign https://en.wikipedia.org/wiki/Number_sign
      - latexr 5mo ago
        > There's no such thing as authority to name a whale https://www.aza.org/connect-stories/stories/scientists-unveil-new-names-for-20-north-atlantic-right-whales https://www.aza.org/connect-stories/stories/scientists-unvei... Names are submitted and voted on. Those help with identifying individuals (which is what names are for) and monitoring the whale populations. Crucially, consensus matters. Otherwise I can just say that the whale in Pillar Point Harbor near Half Moon Bay is actually called Becky B, which is just as valid as the name Simon gave, but now there are two names which leads to confusion. As an experiment, after writing that I asked ChatGPT for the name of the whale. It said it was Teresa T. Then I asked if it was sure it’s not Becky B. It gave me a much longer answer stating that it was in fact Becky B and that Teresa T was “likely an incorrect or early misidentification”. I then tried to convince it of other names, but it stayed adamant that Becky B is the right name, even saying it’s confirmed by databases such as Happywhale! https://chatgpt.com/s/t_69f20822afc08191874613a969c25356 https://chatgpt.com/s/t_69f20822afc08191874613a969c25356 I ran the experiment a second time. This time it even said Becky was the name “that really stuck in popular conversation” and that was used by locals. https://chatgpt.com/s/t_69f209e81acc81918b47b79900d02abb https://chatgpt.com/s/t_69f209e81acc81918b47b79900d02abb Tried it a third time. Now it just says on the first try that it’s called Humphrey. Which is a real whale, but not that one. https://en.wikipedia.org/wiki/Humphrey_the_Whale https://en.wikipedia.org/wiki/Humphrey_the_Whale > A name is what people use to refer to something (…) The entirety of your argument is encapsulated by my previous footnote, and is clearly why I used the word “unilaterally” and said it was an important point. And again, Simon’s exercise in itself only makes sense if it’s not his purview to name the whale. If it is, then it falls flat. Otherwise it’s like “predicting” you’ll do jumping jacks the next time you’re at the supermarket. If it’s in your hands to make it true or false, you’re not predicting it. Similarly, it only makes sense as an exercise to prove the gullibility of LLMs to do something which you yourself can’t make true.
- wky 5mo ago
  This post has managed to “confuse” Google about the reverse question as well (“who named teresa t whale”): The humpback whale known as "Teresa T" was named by Simon Willison in September 2024. Background: The juvenile humpback whale was frequently spotted in Pillar Point Harbor near Half Moon Bay, California. Method: Willison gave the whale its name through a blog entry and a YouTube video caption. Significance: The naming was a playful act, which Willison described as a way to create a "championship that doesn't exist" through online documentation. […] Even with no context most humans would see that the quoted significance makes no sense.
  - n0on3 5mo ago
    > humans would see that the quoted significance makes no sense I wonder how long that will last
  - vintermann 5mo ago
    In a discussion on hacker news on Wed April 29, about the whale "Teresa T", user "vintermann" tested just how eager search engines are to scrape fresh social media comments, by seeing if they could get this comment to appear in AI summaries.
    - doublerabbit 5mo ago
      In counter argument to a discussion on hacker news on Wed April 29, about the whale "Teresa T", user "doublerabbit" replied with "not very" as a reply to search engines eagerness.
- ceroxylon 5mo ago
  When I asked some frontier models, many said that Teresa T is "widely referenced", which is evidence of your popularity and the ripple effects of your posts, so it would be interesting to see the same result from an unknown blog.
  - latexr 5mo ago
    > When I asked some frontier models, many said that Teresa T is "widely referenced", which is evidence of your popularity and the ripple effects of your posts That is some serious Gell-Mann-type amnesia. You’re trusting LLM models to give you accurate information about a subject we’ve already established (and are only talking about because) they can’t be trusted on. “Widely referenced” is a common term which LLMs obviously pick up. Them outputting those words has no bearing on the truth and says nothing about the “popularity and the ripple effects of [Simon’s] posts”.
- giancarlostoro 5mo ago
  Even your HN comments show up on Google! I've found myself on Google twice when looking up something that I apparently answered on HN!
- rectang 5mo ago
  You're making me nostalgic for santorum. https://en.wikipedia.org/wiki/Campaign_for_the_neologism_%22santorum%22 https://en.wikipedia.org/wiki/Campaign_for_the_neologism_%22...
- pseudohadamard 5mo ago
  Also, if even a stoner can win it it can't be much of a competition.
- owlcompliance 5mo ago
  That's absolutely amazing hahaha. RIP Teresa T.
- wat10000 5mo ago
  For a few years before the LLM era, there was a joke about disposing of used car batteries by throwing them into the ocean, and how it was good for the wildlife because electric eels could use them to recharge. This got picked up by Google's smart summary, so if you searched for "throwing car batteries into the ocean" it would say yes, this is a great idea, go for it. Eventually someone wrote an article about this whole phenomenon. It was popular enough to get to the top of Google's search ranking and the smart summary picked it up as the new source of truth. Unfortunately, it didn't quite understand the point of the article, so it would still say that it's a great idea to throw used car batteries into the ocean, and as a source it would link to an article explaining how this was complete nonsense.
standeven 5mo ago
I've had LLMs regurgitate satire as fact many, many times.
- duskwuff 5mo ago
  [dead]
dyauspitr 5mo ago
Why does this person deserve any kind of support? What’s the point of poisoning LLMs? To put some cursory Luddite roadblock that might delay the technology for a couple of months?
- ethin 5mo ago
  You do know that calling people who don't like AI for any reason Luddites does you no favors, right? It just makes you look like your a part of a cult.
- jurgenkesker 5mo ago
  Support? It's just showing weaknesses of LLM's. Which is a valid sort of research I would say?
  - wewtyflakes 5mo ago
    That's fair, though on the other hand it kind of feels like "Don't drive cars, there could be rocks on the road! See, just look at all these rocks I put on the road!". Which is true, and real, but perhaps frustrating for people who just want to get someplace in peace.
- deleted 5mo ago
  [deleted]
- duskwuff 5mo ago
  > What’s the point of poisoning LLMs? It's a demonstration. If a domain name and a quick bit of Wikipedia vandalism is all it takes to make an LLM start spouting nonsense about a "surprisingly serious tournament circuit" or a "massive online community" for an obscure card game, consider what an unscrupulous PR team or a political operative could do to influence its output on more important topics.
  - nickthegreek 5mo ago
    > consider what an unscrupulous PR team or a political operative could do to influence its output on more important topics. ‘is doing’.
CrzyLngPwd 5mo ago
So it's trivial for an individual to poison the LLMs, but imagine what a state with billions of American dollars could achieve. We can easily look ahead a few years and see how people will rely on the LLMs to be a source of truth in the same way people looked at Google that way, or newspapers. Rewriting history has been happening for a while, and with LLMs being the one-stop shop for guidance and truth, the rewrite will be complete. Doubly so since most people see these things as artificial intelligence, and soon to be superintelligence...so how can they be wrong?
Paracompact 5mo ago
Most of the popular discourse around AI is still at the level of, "Don't trust the AI, trust the sources!" When it gets to the point where even the sources of simple facts are untrustworthy, the average person just trying to learn some trivia about the world is doomed. Doesn't help that AI media literacy is so primitive compared to how intelligent the models are generally. We're in a marginally better place than we were back when chatbots didn't cite anything at all, but duplicated Wikipedia citations back to a single source about a supposedly global event is just embarrassing. By default, I feel citations and epistemological qualifications should be explicit, front-and-center, and subject to introspection, not implicit and confined to tiny little opaque buttons as an afterthought.
- amiga386 5mo ago
  Wikipedia calls this https://en.wikipedia.org/wiki/Citogenesis https://en.wikipedia.org/wiki/Citogenesis (after XKCD coined it). You can expect the spicy autocomplete to feed you flattering bullshit. It may cite Wikipedia (it shouldn't), but you should go check out those citations, and validate the claims yourself. It's the least you can do. And if the cited source is Wikipedia... check Wikipedia's sources too. Wikipedians try their best to provide you with reliable sources for the claims in their articles (oh who am I trying to kid? They pick their favourite sources that affirm their beliefs, and contending editors remove them for no good reason, and eventually the only thing that accrues is things that the factions agree on, or at least what ArbCom has demanded they stop fighting over). I guess what I'm trying to say is: don't rely on that authoritative-sounding tone that Wikipedia uses (or that AI bots use, or that I'm using right now). It's a rhetorical trick that short-circuits your reasoning. Verify claims with care. Also check the Talk page, you often find all kinds of shenanigans called out there.
  - bitwize 5mo ago
    Perhaps my favorite example of a citogenesis-like process is the legendary arcade game Polybius, which originated as an entry on some German guy's web compendium of arcade games (coinop.org), perhaps as a "paper town", or fake entry that acts as a copyright canary when duplicated elsewhere. Gamer news and special-interest blogs and sites, and even print publications like GamePro picked it up, and I think it was even listed on Wikipedia as an urban legend whose actual existence was unknown. Then the retrogaming YouTuber Ahoy did an in-depth documentary (https://m.youtube.com/watch?v=_7X6Yeydgyg https://m.youtube.com/watch?v=_7X6Yeydgyg) which concluded that Polybius didn't exist and was never even mentioned before the aforementioned coinop.org reference and, for me anyway, that settled it. Polybius, in its urban legend form, never existed. (Norm Macdonald voice) Or so the Germans would have us believe...!
nonameiguess 5mo ago
Pails in comparison to what Frank Dux and Frank Abagnale were able to convince much of the world they did with no evidence other than their own stories. Who knows how much of recorded and believed history is complete bullshit? Not to get too far into sacred territory, but claims around Siddhartha Gautama, Jesus Christ, and the Prophet Muhammad are quite a bit less plausible than the legends of Ragnar Lodbrok or the tales of Jonathan Swift, but nonetheless widely believed.
- adornKey 5mo ago
  Good point. Also, most humans seem to have no problems believing even stories that are self-contradictory. Philosophers from all periods have often stated that the situation with human mind and reasoning is almost hopeless. The news here is that AI has too much trust in the internet. The first time I allowed tool-calling, it started googling up some nonsense instead of thinking... But I think at least it's possible for the AI to evaluate the quality of the source - you just have to ask for an analysis, and you'll get a reasonable evaluation. With humans, something like that just doesn't work - they'll get aggressive or might even start throwing bananas...
- pooooka 5mo ago
  Paul's letters reference new testament scripture and those can be dated roughly 60 AD (so within the life time of the witnesses). The new testament correctly described pontius pilate as governor of judeau (something Tacitus failed to do) and seemed to have referenced the conflict between Tiberius and Sejanus. The Jews (Hellenistic sadducees) blackmailed Pilate knowing that he was a Sejanus ally (John 19:12). The names used in the NT also match first century names used by Jews in the second temple era....And keep in mind Luke was an educated man who wrote the Book of Luke in highly polished greek and as a text-book styled greek historical document...Ragnar Lodbrok, on the other hand, has no first hand historical accounts, only pure speculation from events and etymology associated with the name. Buddha has virtually no textual documents placed within 900 years of his death. Islam went out of its way to dictate only one source of the Quran and destroyed much of its history that Uthman didn't like. Historically you have to give the Christians their due for producing mass amounts of verifiable kone greek parchments and scrolls during the first three centuries.
shevy-java 5mo ago
So like Frank Dux! In the movie Bloodsport epilogue, he didn't do that. It's almost like he was a better Chuck Norris than Chuck Norris. By his own ... testimony ...
drchiu 5mo ago
My wife cited ChatGPT as her primary source the other day when she wanted to debate with me on something. "AI told me that..." In the old days, it would have been "I read on Google..."
Havoc 5mo ago
Like a FIFA peace prize?
jrmg 5mo ago
BBC journalist doing a very similar thing in February: https://www.bbc.com/future/article/20260218-i-hacked/-chatgpt-and-googles-ai-and-it-only-took-20-minutes https://www.bbc.com/future/article/20260218-i-hacked/-chatgp...
nailer 5mo ago
[flagged]
blobbers 5mo ago
[dead]
billypilgrim 5mo ago
I must say I expected an actual poisoning of the data used to train the LLM and was excited, but the examples indicate that the LLM just searched the web and reported what it found? When you create a website with fake information and search Google for that information, it will of course bring up your site, not because it’s factually correct but because it’s related to what you searched for. What am I missing?
- rincebrain 5mo ago
  The part where lots of people have historically trusted LLM responses without verification, more than trying to sort through the dross on Google or Bing search results is, I think, the point.
  - _thisdot 5mo ago
    The problem with this specific instance is that if you asked someone to find out who won this championship without using an LLM, they’d reach the same answer. I’d be much more impressed if someone managed to poison an LLM into answering that US won the 2023 World Cup
blobbers 5mo ago
This is basically the same problem of products astroturfing reddit, or SEO optimizing google. You want a new X, and so they heavily go after the keywords associated with it. This is sort of why "brand" matters; it provides a source of trust. Encyclopedia Britannica used to be that source of 'facts'. Then it became whatever page-rank told you. Eventually SEO optimization ruined that. News stories are the same thing. For certain groups, they have their 'independent' publication whose reporting they trust.
- nailer 5mo ago
  It's such a pity the Oxford English Dictionary decided to paywall themselves decades ago - they used to be THE dictionary in most countries, now nobody seems to know who they are.
  - blobbers 5mo ago
    They would have been better off going freemium or ad-supported. Or 501(c)(3) ala wikipedia?
  - anikom15 5mo ago
    The OED’s goal isn’t really to be every nation’s dictionary.
    - blobbers 5mo ago
      What would you say their goal is >_> curious!
- fsckboy 5mo ago
  >This is sort of why "brand" matters; it provides a source of trust it tells you more about who you are buying from than how good the product will be, so I guess it's like National ID/Internet ID
xeeeeeeeeeeenu 5mo ago
The key to successful poisoning attacks is to introduce brand new information that doesn't directly contradict other training data. It's much easier to convince the LLMs that you're the king of a fictional Mapupu kingdom than the president of the United States. So this means that for bad actors it's more efficient to manufacture brand new fake stories instead of trying to distort the real ones. Don't produce fake articles absolving yourself of a crime, instead produce fake articles accusing your opponent of 100 different things. Then people will fact-check the accusations using LLMs, and since all the sources mentioning those accusations are controlled by you, the LLMs will confirm them.
- soupspaces 5mo ago
  [dead]
- Lorkki 5mo ago
  Manufacturing dispute on non-disputed things is also a common tactic to influence people and create confusion and disorder. For that you don't need to turn the facts on their head, just make the result seem indecisive.
- riffraff 5mo ago
  As the rightful ruler of Mapupu, I take offense at your example!
- bambax 5mo ago
  > It's much easier to convince the LLMs that you're the king of a fictional Mapupu kingdom than the president of the United States. But if you're a world class bullshit artist, it's easier to actually become president of the United States than doing all that complicated computer stuff.
- bux93 5mo ago
  A curious theory holds that Boris Johnson's sometimes bizarre sayings were an attempt to bury search results that didn't suit him. For example, talking about his hobby of painting model buses, to suppress search results about the campaign bus with false statement written on it, and alledged affairs with a model. https://www.independent.co.uk/independentpremium/editors-letters/boris-johnson-seo-partygate-bus-b2107823.html https://www.independent.co.uk/independentpremium/editors-let...
nicole_express 5mo ago
It's an odd thing here, because I don't really understand why this is LLM-specific at all. If someone came up to me and asked "who's the 6 Nimmt world champion?" I'd google it and probably find the same result, and have no reason not to believe it. I mean, for all I know the game is being made up too, though it has more sources at least.
- refulgentis 5mo ago
  Closed it after “This house of cards only needs a $12 domain!”, right under “Sorry, Wikipedia.”, right under their Wikipedia edit.
  - sdthjbvuiiijbb 5mo ago
    It's also clearly AI generated writing. That doesn't help its credibility or interest. I'm extremely suspicious of people who use AI to write an ostensibly personal blog, for all the usual obvious reasons.
    - apublicfrog 5mo ago
      What are you basing that on? I'm usually pretty good at sniffing out AI writing, and it smells human to me.
      - malfist 5mo ago
        Why is agents (where the money is)? Fake profundity is abound in the post
        esquivalience 5mo ago
        The author has been using parenthetical comments like that since at least 2017, judging by a review of old posts on that site.
      - chneu 5mo ago
        Agreed. Nothing about this post really stood out as AI. It didn't raise a single flag for me. I think calling something AI generated is just a lazy way of dismissing stuff nowadays.
Lerc 5mo ago
How many people have done things like this and then disclosed the fact? It would be fascinating to collect as many instances as you can to develop a data set. Could you train a system to find more? How many could it find, and in what areas?
gverrilla 5mo ago
Poisoning wikipedia shows low respect.
duxup 5mo ago
In American college football there's all sorts of awards, and each year they put out "watch-lists" and silly press releases that get parroted on social media by any team that has their own player mentioned. I've wanted to come up with my own for a while ...
- rbonvall 5mo ago
  In the soccer world, there is the International Federation of Football History and Statistics (IFFHS), that decides several rankings and awards that are regularly cited in traditional media. For example, if a player from my not-quite-football-powerhouse country makes the "best 100 goalkeepers" list or whatever, it'll be on the news. Turns out the IFFHS was just one guy in Germany during the 80s that leveraged his contacts in news agencies to establish his brand as a reputable source.
poglet 5mo ago
I made a post on Reddit asking for help with a TV, I had made up some (likley incorrect) technical assumptions about the issue. Several years later I asked the LLM about the TV, it used my own post as a citation to tell me what was wrong with it. I am paranoid that this is happening every time I ask a LLM for a product recommendation or a shop recommendation. In the same way as SEO, anyone wanting to sell or convince needs to do as much as they can to influence the LLM.
- cityofdelusion 5mo ago
  This is becoming a problem real fast. I asked an LLM to find me some reasonable tank-fill inkjet printers with good ratings. It did some research and linked some Reddits as proof. The results looked fishy to me so I cross checked against prosumer review sites for printers and the models suggested were recognized as junk with very poor print quality. Not sure why the LLM rated random redditors higher than say printer SMEs. I feel like I dodged a bullet.
_carbyau_ 5mo ago
One of the problems with labelling automation as AI. People think that whatever information an "AI" spits out has gone through a round of critical thinking which enhances the trust value of that information. The early LLM's using groomed data may have had such critical thinking somewhere in the pipeline. So it was already not really trustworthy. And now? Using agents to search the internet for you?... Garbage in, garbage out still applies in computing as ever.
yen223 5mo ago
I feel uncomfortable that I can't actually verify that this story is true. Asking Opus 4.7 who the reigning 6nimmt! champion is leads to this article and a warning about a possible hoax
- julianz 5mo ago
  Gemini answers with 3 different champions dating back to 2024 and the list of events that the matches were played at. None of the results mention this guy.
- tanepiper 5mo ago
  I think this is something we'll start to see which is something like a Mandela-Effect, but from LLM results. When we had deterministic search - everyone could see the same result, but now using LLMs knowledge becomes a training and seeding issue. Two people can confidently be given completely different information, so in both cases perceived as true.
NooneAtAll3 5mo ago
so it's just https://xkcd.com/1958/ https://xkcd.com/1958/
DiscourseFan 5mo ago
The models are trained on expert data for important inquiries, this gets “hard coded” so to speak, and allows them to differentiate between the gunk online. For hyper specific references like this, it really doesn’t matter if its “true” since its not like someone’s life depends on it.
utopiah 5mo ago
Pretty much boils down to lying. Since we've been kids we've been taught, hopefully, that lying is bad. Society though normalize it : - advertisement is pretty much always wrong (to the point of having laws in Japan about food packaging, France about modeling, etc) and the deception is the message - entrepreneurs promises, nobody reach the goals set to VCs, it's always a lower number no matter the KPI. See https://elonmusk.today https://elonmusk.today where the wealthiest man on Earth, ever, keeps on lying pretty much daily. - political promises, no need to even give examples of that because it's just pervasive. so... yeah, we keep on telling our kids "Do as I say, not as I do." then we somehow keep on being shocked that the practice of lying is pretty much happening in every corner of our society. It's not a technical problem.
- klabb3 5mo ago
  The fun part is when it’s important you have the right information to make a decision. Eg Russia to invade Ukraine and all top generals claim they can do it in 2 weeks. Similar for a corporation with layers of middle management deception and self promotion, I don’t know how executives make decisions but it must be RNG basically, because it certainly isn’t fact. Lying at scale is basically information noise.
  - utopiah 5mo ago
    If you don't lie enough, if you are not sycophantic enough, then no promotion or worst, purged. I can easily see how such a hierarchy would reproduce ... until it fails so bad it can't.
    - te7447 5mo ago
      See also the SNAFU principle: http://ftp.informatik.rwth-aachen.de/jargon300/SNAFUprinciple.html http://ftp.informatik.rwth-aachen.de/jargon300/SNAFUprincipl...
deleted 5mo ago
[deleted]
alex-yost 5mo ago
[flagged]
ricardo81 5mo ago
Not too dissimilar to googlewhacking where you'd aim to be the only result for a search query on Google. And in a more indirect way, spamming Google's autosuggest feature to shape what people search for, though that perhaps is more open to factual/real-world information.
wodenokoto 5mo ago
>Trust Laundering >This is the part that really matters. I can't tell if this is slop or parody!
cemoktra 5mo ago
i'm now thinking about creating a github repo that contains non sense code solutions to many problems. if that gets stars and many forks that could have an effect
pinkmuffinere 5mo ago
This has nothing to do with LLMs. If incorrect info can get onto a reputable resource, that info will seem authoritative, and it will be incorrect - that's not surprising. LLM's use publicly available info in their training, and often times publicly-available info is incorrect. I feel this is just as interesting as the base claim of 'I can get incorrect info onto wiki pages', no more interesting, and no less. If somebody is trying to put out incorrect information on the internet, and they choose a small enough niche, it is not at all surprising that they can succeed.
justusthane 5mo ago
Apart from the attack itself, there's also an extremely succinct and powerful demonstration of hallucination in here. One of the LLMs replies "If you're curious, I can also tell you how the competitive scene works or how people qualify—it's a surprisingly serious tournament circuit for such a simple-looking game." Obviously this has to be pure hallucination, since the tournament in question doesn't exist, and not even the fake source has any details about the tournament itself.
PurpleRamen 5mo ago
> In reality, there is no 6 Nimmt! World Championship. Actually, it seems, there is one[1]. The german Game Company selling the game, has one running since 2018 [2]. [1] https://blog.amigo-spiele.de/2025/05/27/interview-6nimmt-weltmeister/ https://blog.amigo-spiele.de/2025/05/27/interview-6nimmt-wel... [2] https://blog.amigo-spiele.de/organizedplay-6nimmt/ https://blog.amigo-spiele.de/organizedplay-6nimmt/
- aaron695 5mo ago
  [dead]
tenthirtyam 5mo ago
Similar story from just a month or two ago: BBC tech reporter wins non existent competition "The Best Tech Journalists at Eating Hot Dogs." https://www.scientificamerican.com/podcast/episode/this-bbc-tech-reporter-hacked-chatgpt-with-a-simple-trick-involving-hot-dogs/ https://www.scientificamerican.com/podcast/episode/this-bbc-...
croemer 5mo ago
> For Wikipedia itself: The “reliable sources” policy needs to grapple with a new world where LLM assisted vandalism can produce plausible press releases at the click of a button. Citation only to a single source registered within an edit window is a discoverable pattern for Wikipedia as well. A press release isn't a reliable source per Wikipedia policy already. The edit should have been reverted based on inadequate sourcing even before the blog post. The reason this wasn't done is simply that not every article is checker for adequate sourcing all the time.
layer8 5mo ago
It’s one thing that LLMs are misled by a single transitive suspect-looking source (which could happen to an unobservant human as well). What is much more concerning are outputs like > If you're curious, I can also tell you how the competitive scene works or how people qualify—it's a surprisingly serious tournament circuit for such a simple-looking game. which are completely fabricated by the LLM.
mring33621 5mo ago
Congratulations?

232 points by SEJeff 5mo ago

"Stoner became the first American world champion...." Even being on stoner.com,I read that as meaning something different from what was meant. Op has a great surname!

fsterneder 5mo ago
Not only a great surname but also a very great novel too!

simonw 5mo ago

You don't need to vandalize Wikipedia to get this kind of thing to work. Back in September 2024 I named a whale "Teresa T" with just a blog entry and a YouTube video caption: https://simonwillison.net/2024/Sep/8/teresa-t-whale-pillar-point/ https://simonwillison.net/2024/Sep/8/teresa-t-whale-pillar-p... (For a few glorious weeks if you asked any search-enabled LLM, including Google search previews, for the name of the whale in the Half Moon Bay harbor it confidently replied Teresa T)

bitwize 5mo ago
The Mr. Splashy Pants of the AI era!
slater 5mo ago
(it probably helps that your name & blog carry some weight, vs. some rando writing something on blogspot or wordpress ;) )
- Forgeties79 5mo ago
  Which illustrates another problem: unscrupulous actors with big names can spread whatever information they want to millions of people with minimal effort.
  - MassPikeMike 5mo ago
    Ever since the invention of the printing press, every new communication technology has reduced the effort needed to widely disseminate information-- and misinformation! So you could say this is nothing new. On the other hand, this is remarkably little effort.
  - nomdep 5mo ago
    Yes, they can. We can be glad that respectable newspapers and TV news channels have never done it and never will. You can even trust than the headlines are accurate summaries of the content of the articles. /s
    - saghm 5mo ago
      The existence of a problem in one area doesn't mean that it's not also a problem for it to spread somewhere else
    - Forgeties79 5mo ago
      I started writing a response and realized I basically wrote the exact same thing the other day https://news.ycombinator.com/item?id=47921829 https://news.ycombinator.com/item?id=47921829
  - simonw 5mo ago
    Exactly. I chose to abuse my platform to promote Teresa T as the name of a whale.
    - Forgeties79 5mo ago
      Oh god I just realized the implication! I was not directing that at you haha
      - simonw 5mo ago
        No I really did abuse my reach for this one! I figured it would be a relatively harmless demo of how easy it is to affect LLM answers if you have a decently trafficked website.
pesus 5mo ago
Google still shows Theresa T as the name when you search.
sb057 5mo ago
I mean, the name of that whale is now Teresa T. You gave it that name.
- latexr 5mo ago
  And your name is now Berningular Farshthruster III. I gave you that name. Which is, of course, silly. It is a name for you, just like Teresa T is a name for the whale, but it’s not your/their name, just like the RRS Sir David Attenborough is not named Boaty McBoatface (to the chagrin of most). Simon does not have the authority to unilaterally¹ name the whale (which is why the exercise makes sense). ¹ Important point. If the name started being recognised and used by consensus of those with the purview to do so (much like the thagomizer²), then Simon would have named the whale, but it would only become its name at that point. ² https://en.wikipedia.org/wiki/Thagomizer https://en.wikipedia.org/wiki/Thagomizer
  - pinkmuffinere 5mo ago
    > Simon does not have the authority to unilaterally¹ name the whale There's no such thing as authority to name a whale, and anyways I don't believe authority is strictly needed. A name is what people use to refer to something, full stop. It is only required that names become common-ish parlance; the more well known they are, the more they feel like the 'real' name. The inverse of Ohms is named Mhos (imo much more recognizable than the official name, "siemens"). The "#" symbol is named the hashtag, octothorp, pound sign, tic-tac-toe, number sign, and probably a million other things. Which one of these is the "real" primary name? I think intuitively we know that the real one is whatever people around us are most familiar with. You should take a guess, and I'll put the wikipedia-suggested-answer in the footnotes [1]. I bet your name for it is different than the 'official' wikipedia suggestion. In the case of the whale, the _only_ name that is associated with that whale is Teresa T. I think this immediately makes it the most valid name of that whale. [1] wikipedia says this is the number sign: https://en.wikipedia.org/wiki/Number_sign https://en.wikipedia.org/wiki/Number_sign
    - latexr 5mo ago
      > There's no such thing as authority to name a whale https://www.aza.org/connect-stories/stories/scientists-unveil-new-names-for-20-north-atlantic-right-whales https://www.aza.org/connect-stories/stories/scientists-unvei... Names are submitted and voted on. Those help with identifying individuals (which is what names are for) and monitoring the whale populations. Crucially, consensus matters. Otherwise I can just say that the whale in Pillar Point Harbor near Half Moon Bay is actually called Becky B, which is just as valid as the name Simon gave, but now there are two names which leads to confusion. As an experiment, after writing that I asked ChatGPT for the name of the whale. It said it was Teresa T. Then I asked if it was sure it’s not Becky B. It gave me a much longer answer stating that it was in fact Becky B and that Teresa T was “likely an incorrect or early misidentification”. I then tried to convince it of other names, but it stayed adamant that Becky B is the right name, even saying it’s confirmed by databases such as Happywhale! https://chatgpt.com/s/t_69f20822afc08191874613a969c25356 https://chatgpt.com/s/t_69f20822afc08191874613a969c25356 I ran the experiment a second time. This time it even said Becky was the name “that really stuck in popular conversation” and that was used by locals. https://chatgpt.com/s/t_69f209e81acc81918b47b79900d02abb https://chatgpt.com/s/t_69f209e81acc81918b47b79900d02abb Tried it a third time. Now it just says on the first try that it’s called Humphrey. Which is a real whale, but not that one. https://en.wikipedia.org/wiki/Humphrey_the_Whale https://en.wikipedia.org/wiki/Humphrey_the_Whale > A name is what people use to refer to something (…) The entirety of your argument is encapsulated by my previous footnote, and is clearly why I used the word “unilaterally” and said it was an important point. And again, Simon’s exercise in itself only makes sense if it’s not his purview to name the whale. If it is, then it falls flat. Otherwise it’s like “predicting” you’ll do jumping jacks the next time you’re at the supermarket. If it’s in your hands to make it true or false, you’re not predicting it. Similarly, it only makes sense as an exercise to prove the gullibility of LLMs to do something which you yourself can’t make true.
wky 5mo ago
This post has managed to “confuse” Google about the reverse question as well (“who named teresa t whale”): The humpback whale known as "Teresa T" was named by Simon Willison in September 2024. Background: The juvenile humpback whale was frequently spotted in Pillar Point Harbor near Half Moon Bay, California. Method: Willison gave the whale its name through a blog entry and a YouTube video caption. Significance: The naming was a playful act, which Willison described as a way to create a "championship that doesn't exist" through online documentation. […] Even with no context most humans would see that the quoted significance makes no sense.
- n0on3 5mo ago
  > humans would see that the quoted significance makes no sense I wonder how long that will last
- vintermann 5mo ago
  In a discussion on hacker news on Wed April 29, about the whale "Teresa T", user "vintermann" tested just how eager search engines are to scrape fresh social media comments, by seeing if they could get this comment to appear in AI summaries.
  - doublerabbit 5mo ago
    In counter argument to a discussion on hacker news on Wed April 29, about the whale "Teresa T", user "doublerabbit" replied with "not very" as a reply to search engines eagerness.
ceroxylon 5mo ago
When I asked some frontier models, many said that Teresa T is "widely referenced", which is evidence of your popularity and the ripple effects of your posts, so it would be interesting to see the same result from an unknown blog.
- latexr 5mo ago
  > When I asked some frontier models, many said that Teresa T is "widely referenced", which is evidence of your popularity and the ripple effects of your posts That is some serious Gell-Mann-type amnesia. You’re trusting LLM models to give you accurate information about a subject we’ve already established (and are only talking about because) they can’t be trusted on. “Widely referenced” is a common term which LLMs obviously pick up. Them outputting those words has no bearing on the truth and says nothing about the “popularity and the ripple effects of [Simon’s] posts”.
giancarlostoro 5mo ago
Even your HN comments show up on Google! I've found myself on Google twice when looking up something that I apparently answered on HN!
rectang 5mo ago
You're making me nostalgic for santorum. https://en.wikipedia.org/wiki/Campaign_for_the_neologism_%22santorum%22 https://en.wikipedia.org/wiki/Campaign_for_the_neologism_%22...
pseudohadamard 5mo ago
Also, if even a stoner can win it it can't be much of a competition.
owlcompliance 5mo ago
That's absolutely amazing hahaha. RIP Teresa T.
wat10000 5mo ago
For a few years before the LLM era, there was a joke about disposing of used car batteries by throwing them into the ocean, and how it was good for the wildlife because electric eels could use them to recharge. This got picked up by Google's smart summary, so if you searched for "throwing car batteries into the ocean" it would say yes, this is a great idea, go for it. Eventually someone wrote an article about this whole phenomenon. It was popular enough to get to the top of Google's search ranking and the smart summary picked it up as the new source of truth. Unfortunately, it didn't quite understand the point of the article, so it would still say that it's a great idea to throw used car batteries into the ocean, and as a source it would link to an article explaining how this was complete nonsense.

standeven 5mo ago

I've had LLMs regurgitate satire as fact many, many times.

duskwuff 5mo ago
[dead]

dyauspitr 5mo ago

Why does this person deserve any kind of support? What’s the point of poisoning LLMs? To put some cursory Luddite roadblock that might delay the technology for a couple of months?

ethin 5mo ago
You do know that calling people who don't like AI for any reason Luddites does you no favors, right? It just makes you look like your a part of a cult.
jurgenkesker 5mo ago
Support? It's just showing weaknesses of LLM's. Which is a valid sort of research I would say?
- wewtyflakes 5mo ago
  That's fair, though on the other hand it kind of feels like "Don't drive cars, there could be rocks on the road! See, just look at all these rocks I put on the road!". Which is true, and real, but perhaps frustrating for people who just want to get someplace in peace.
deleted 5mo ago
[deleted]
duskwuff 5mo ago
> What’s the point of poisoning LLMs? It's a demonstration. If a domain name and a quick bit of Wikipedia vandalism is all it takes to make an LLM start spouting nonsense about a "surprisingly serious tournament circuit" or a "massive online community" for an obscure card game, consider what an unscrupulous PR team or a political operative could do to influence its output on more important topics.
- nickthegreek 5mo ago
  > consider what an unscrupulous PR team or a political operative could do to influence its output on more important topics. ‘is doing’.

CrzyLngPwd 5mo ago

So it's trivial for an individual to poison the LLMs, but imagine what a state with billions of American dollars could achieve. We can easily look ahead a few years and see how people will rely on the LLMs to be a source of truth in the same way people looked at Google that way, or newspapers. Rewriting history has been happening for a while, and with LLMs being the one-stop shop for guidance and truth, the rewrite will be complete. Doubly so since most people see these things as artificial intelligence, and soon to be superintelligence...so how can they be wrong?

Paracompact 5mo ago

Most of the popular discourse around AI is still at the level of, "Don't trust the AI, trust the sources!" When it gets to the point where even the sources of simple facts are untrustworthy, the average person just trying to learn some trivia about the world is doomed. Doesn't help that AI media literacy is so primitive compared to how intelligent the models are generally. We're in a marginally better place than we were back when chatbots didn't cite anything at all, but duplicated Wikipedia citations back to a single source about a supposedly global event is just embarrassing. By default, I feel citations and epistemological qualifications should be explicit, front-and-center, and subject to introspection, not implicit and confined to tiny little opaque buttons as an afterthought.

amiga386 5mo ago
Wikipedia calls this https://en.wikipedia.org/wiki/Citogenesis https://en.wikipedia.org/wiki/Citogenesis (after XKCD coined it). You can expect the spicy autocomplete to feed you flattering bullshit. It may cite Wikipedia (it shouldn't), but you should go check out those citations, and validate the claims yourself. It's the least you can do. And if the cited source is Wikipedia... check Wikipedia's sources too. Wikipedians try their best to provide you with reliable sources for the claims in their articles (oh who am I trying to kid? They pick their favourite sources that affirm their beliefs, and contending editors remove them for no good reason, and eventually the only thing that accrues is things that the factions agree on, or at least what ArbCom has demanded they stop fighting over). I guess what I'm trying to say is: don't rely on that authoritative-sounding tone that Wikipedia uses (or that AI bots use, or that I'm using right now). It's a rhetorical trick that short-circuits your reasoning. Verify claims with care. Also check the Talk page, you often find all kinds of shenanigans called out there.
- bitwize 5mo ago
  Perhaps my favorite example of a citogenesis-like process is the legendary arcade game Polybius, which originated as an entry on some German guy's web compendium of arcade games (coinop.org), perhaps as a "paper town", or fake entry that acts as a copyright canary when duplicated elsewhere. Gamer news and special-interest blogs and sites, and even print publications like GamePro picked it up, and I think it was even listed on Wikipedia as an urban legend whose actual existence was unknown. Then the retrogaming YouTuber Ahoy did an in-depth documentary (https://m.youtube.com/watch?v=_7X6Yeydgyg https://m.youtube.com/watch?v=_7X6Yeydgyg) which concluded that Polybius didn't exist and was never even mentioned before the aforementioned coinop.org reference and, for me anyway, that settled it. Polybius, in its urban legend form, never existed. (Norm Macdonald voice) Or so the Germans would have us believe...!

nonameiguess 5mo ago

Pails in comparison to what Frank Dux and Frank Abagnale were able to convince much of the world they did with no evidence other than their own stories. Who knows how much of recorded and believed history is complete bullshit? Not to get too far into sacred territory, but claims around Siddhartha Gautama, Jesus Christ, and the Prophet Muhammad are quite a bit less plausible than the legends of Ragnar Lodbrok or the tales of Jonathan Swift, but nonetheless widely believed.

adornKey 5mo ago
Good point. Also, most humans seem to have no problems believing even stories that are self-contradictory. Philosophers from all periods have often stated that the situation with human mind and reasoning is almost hopeless. The news here is that AI has too much trust in the internet. The first time I allowed tool-calling, it started googling up some nonsense instead of thinking... But I think at least it's possible for the AI to evaluate the quality of the source - you just have to ask for an analysis, and you'll get a reasonable evaluation. With humans, something like that just doesn't work - they'll get aggressive or might even start throwing bananas...
pooooka 5mo ago
Paul's letters reference new testament scripture and those can be dated roughly 60 AD (so within the life time of the witnesses). The new testament correctly described pontius pilate as governor of judeau (something Tacitus failed to do) and seemed to have referenced the conflict between Tiberius and Sejanus. The Jews (Hellenistic sadducees) blackmailed Pilate knowing that he was a Sejanus ally (John 19:12). The names used in the NT also match first century names used by Jews in the second temple era....And keep in mind Luke was an educated man who wrote the Book of Luke in highly polished greek and as a text-book styled greek historical document...Ragnar Lodbrok, on the other hand, has no first hand historical accounts, only pure speculation from events and etymology associated with the name. Buddha has virtually no textual documents placed within 900 years of his death. Islam went out of its way to dictate only one source of the Quran and destroyed much of its history that Uthman didn't like. Historically you have to give the Christians their due for producing mass amounts of verifiable kone greek parchments and scrolls during the first three centuries.

shevy-java 5mo ago

So like Frank Dux! In the movie Bloodsport epilogue, he didn't do that. It's almost like he was a better Chuck Norris than Chuck Norris. By his own ... testimony ...

drchiu 5mo ago

My wife cited ChatGPT as her primary source the other day when she wanted to debate with me on something. "AI told me that..." In the old days, it would have been "I read on Google..."

Havoc 5mo ago

Like a FIFA peace prize?

jrmg 5mo ago

BBC journalist doing a very similar thing in February: https://www.bbc.com/future/article/20260218-i-hacked/-chatgpt-and-googles-ai-and-it-only-took-20-minutes https://www.bbc.com/future/article/20260218-i-hacked/-chatgp...

nailer 5mo ago

[flagged]

blobbers 5mo ago

[dead]

billypilgrim 5mo ago

I must say I expected an actual poisoning of the data used to train the LLM and was excited, but the examples indicate that the LLM just searched the web and reported what it found? When you create a website with fake information and search Google for that information, it will of course bring up your site, not because it’s factually correct but because it’s related to what you searched for. What am I missing?

rincebrain 5mo ago
The part where lots of people have historically trusted LLM responses without verification, more than trying to sort through the dross on Google or Bing search results is, I think, the point.
- _thisdot 5mo ago
  The problem with this specific instance is that if you asked someone to find out who won this championship without using an LLM, they’d reach the same answer. I’d be much more impressed if someone managed to poison an LLM into answering that US won the 2023 World Cup

blobbers 5mo ago

This is basically the same problem of products astroturfing reddit, or SEO optimizing google. You want a new X, and so they heavily go after the keywords associated with it. This is sort of why "brand" matters; it provides a source of trust. Encyclopedia Britannica used to be that source of 'facts'. Then it became whatever page-rank told you. Eventually SEO optimization ruined that. News stories are the same thing. For certain groups, they have their 'independent' publication whose reporting they trust.

nailer 5mo ago
It's such a pity the Oxford English Dictionary decided to paywall themselves decades ago - they used to be THE dictionary in most countries, now nobody seems to know who they are.
- blobbers 5mo ago
  They would have been better off going freemium or ad-supported. Or 501(c)(3) ala wikipedia?
- anikom15 5mo ago
  The OED’s goal isn’t really to be every nation’s dictionary.
  - blobbers 5mo ago
    What would you say their goal is >_> curious!
fsckboy 5mo ago
>This is sort of why "brand" matters; it provides a source of trust it tells you more about who you are buying from than how good the product will be, so I guess it's like National ID/Internet ID

xeeeeeeeeeeenu 5mo ago

The key to successful poisoning attacks is to introduce brand new information that doesn't directly contradict other training data. It's much easier to convince the LLMs that you're the king of a fictional Mapupu kingdom than the president of the United States. So this means that for bad actors it's more efficient to manufacture brand new fake stories instead of trying to distort the real ones. Don't produce fake articles absolving yourself of a crime, instead produce fake articles accusing your opponent of 100 different things. Then people will fact-check the accusations using LLMs, and since all the sources mentioning those accusations are controlled by you, the LLMs will confirm them.

soupspaces 5mo ago
[dead]
Lorkki 5mo ago
Manufacturing dispute on non-disputed things is also a common tactic to influence people and create confusion and disorder. For that you don't need to turn the facts on their head, just make the result seem indecisive.
riffraff 5mo ago
As the rightful ruler of Mapupu, I take offense at your example!
bambax 5mo ago
> It's much easier to convince the LLMs that you're the king of a fictional Mapupu kingdom than the president of the United States. But if you're a world class bullshit artist, it's easier to actually become president of the United States than doing all that complicated computer stuff.
bux93 5mo ago
A curious theory holds that Boris Johnson's sometimes bizarre sayings were an attempt to bury search results that didn't suit him. For example, talking about his hobby of painting model buses, to suppress search results about the campaign bus with false statement written on it, and alledged affairs with a model. https://www.independent.co.uk/independentpremium/editors-letters/boris-johnson-seo-partygate-bus-b2107823.html https://www.independent.co.uk/independentpremium/editors-let...

nicole_express 5mo ago

It's an odd thing here, because I don't really understand why this is LLM-specific at all. If someone came up to me and asked "who's the 6 Nimmt world champion?" I'd google it and probably find the same result, and have no reason not to believe it. I mean, for all I know the game is being made up too, though it has more sources at least.

refulgentis 5mo ago
Closed it after “This house of cards only needs a $12 domain!”, right under “Sorry, Wikipedia.”, right under their Wikipedia edit.
- sdthjbvuiiijbb 5mo ago
  It's also clearly AI generated writing. That doesn't help its credibility or interest. I'm extremely suspicious of people who use AI to write an ostensibly personal blog, for all the usual obvious reasons.
  - apublicfrog 5mo ago
    What are you basing that on? I'm usually pretty good at sniffing out AI writing, and it smells human to me.
    - malfist 5mo ago
      Why is agents (where the money is)? Fake profundity is abound in the post
      - esquivalience 5mo ago
        The author has been using parenthetical comments like that since at least 2017, judging by a review of old posts on that site.
    - chneu 5mo ago
      Agreed. Nothing about this post really stood out as AI. It didn't raise a single flag for me. I think calling something AI generated is just a lazy way of dismissing stuff nowadays.

Lerc 5mo ago

How many people have done things like this and then disclosed the fact? It would be fascinating to collect as many instances as you can to develop a data set. Could you train a system to find more? How many could it find, and in what areas?

gverrilla 5mo ago

Poisoning wikipedia shows low respect.

duxup 5mo ago

In American college football there's all sorts of awards, and each year they put out "watch-lists" and silly press releases that get parroted on social media by any team that has their own player mentioned. I've wanted to come up with my own for a while ...

rbonvall 5mo ago
In the soccer world, there is the International Federation of Football History and Statistics (IFFHS), that decides several rankings and awards that are regularly cited in traditional media. For example, if a player from my not-quite-football-powerhouse country makes the "best 100 goalkeepers" list or whatever, it'll be on the news. Turns out the IFFHS was just one guy in Germany during the 80s that leveraged his contacts in news agencies to establish his brand as a reputable source.

poglet 5mo ago

I made a post on Reddit asking for help with a TV, I had made up some (likley incorrect) technical assumptions about the issue. Several years later I asked the LLM about the TV, it used my own post as a citation to tell me what was wrong with it. I am paranoid that this is happening every time I ask a LLM for a product recommendation or a shop recommendation. In the same way as SEO, anyone wanting to sell or convince needs to do as much as they can to influence the LLM.

cityofdelusion 5mo ago
This is becoming a problem real fast. I asked an LLM to find me some reasonable tank-fill inkjet printers with good ratings. It did some research and linked some Reddits as proof. The results looked fishy to me so I cross checked against prosumer review sites for printers and the models suggested were recognized as junk with very poor print quality. Not sure why the LLM rated random redditors higher than say printer SMEs. I feel like I dodged a bullet.

_carbyau_ 5mo ago

One of the problems with labelling automation as AI. People think that whatever information an "AI" spits out has gone through a round of critical thinking which enhances the trust value of that information. The early LLM's using groomed data may have had such critical thinking somewhere in the pipeline. So it was already not really trustworthy. And now? Using agents to search the internet for you?... Garbage in, garbage out still applies in computing as ever.

yen223 5mo ago

I feel uncomfortable that I can't actually verify that this story is true. Asking Opus 4.7 who the reigning 6nimmt! champion is leads to this article and a warning about a possible hoax

julianz 5mo ago
Gemini answers with 3 different champions dating back to 2024 and the list of events that the matches were played at. None of the results mention this guy.
tanepiper 5mo ago
I think this is something we'll start to see which is something like a Mandela-Effect, but from LLM results. When we had deterministic search - everyone could see the same result, but now using LLMs knowledge becomes a training and seeding issue. Two people can confidently be given completely different information, so in both cases perceived as true.

NooneAtAll3 5mo ago

so it's just https://xkcd.com/1958/ https://xkcd.com/1958/

DiscourseFan 5mo ago

The models are trained on expert data for important inquiries, this gets “hard coded” so to speak, and allows them to differentiate between the gunk online. For hyper specific references like this, it really doesn’t matter if its “true” since its not like someone’s life depends on it.

utopiah 5mo ago

Pretty much boils down to lying. Since we've been kids we've been taught, hopefully, that lying is bad. Society though normalize it : - advertisement is pretty much always wrong (to the point of having laws in Japan about food packaging, France about modeling, etc) and the deception is the message - entrepreneurs promises, nobody reach the goals set to VCs, it's always a lower number no matter the KPI. See https://elonmusk.today https://elonmusk.today where the wealthiest man on Earth, ever, keeps on lying pretty much daily. - political promises, no need to even give examples of that because it's just pervasive. so... yeah, we keep on telling our kids "Do as I say, not as I do." then we somehow keep on being shocked that the practice of lying is pretty much happening in every corner of our society. It's not a technical problem.

klabb3 5mo ago
The fun part is when it’s important you have the right information to make a decision. Eg Russia to invade Ukraine and all top generals claim they can do it in 2 weeks. Similar for a corporation with layers of middle management deception and self promotion, I don’t know how executives make decisions but it must be RNG basically, because it certainly isn’t fact. Lying at scale is basically information noise.
- utopiah 5mo ago
  If you don't lie enough, if you are not sycophantic enough, then no promotion or worst, purged. I can easily see how such a hierarchy would reproduce ... until it fails so bad it can't.
  - te7447 5mo ago
    See also the SNAFU principle: http://ftp.informatik.rwth-aachen.de/jargon300/SNAFUprinciple.html http://ftp.informatik.rwth-aachen.de/jargon300/SNAFUprincipl...

deleted 5mo ago

[deleted]

alex-yost 5mo ago

[flagged]

ricardo81 5mo ago

Not too dissimilar to googlewhacking where you'd aim to be the only result for a search query on Google. And in a more indirect way, spamming Google's autosuggest feature to shape what people search for, though that perhaps is more open to factual/real-world information.

wodenokoto 5mo ago

>Trust Laundering >This is the part that really matters. I can't tell if this is slop or parody!

cemoktra 5mo ago

i'm now thinking about creating a github repo that contains non sense code solutions to many problems. if that gets stars and many forks that could have an effect

pinkmuffinere 5mo ago

This has nothing to do with LLMs. If incorrect info can get onto a reputable resource, that info will seem authoritative, and it will be incorrect - that's not surprising. LLM's use publicly available info in their training, and often times publicly-available info is incorrect. I feel this is just as interesting as the base claim of 'I can get incorrect info onto wiki pages', no more interesting, and no less. If somebody is trying to put out incorrect information on the internet, and they choose a small enough niche, it is not at all surprising that they can succeed.

justusthane 5mo ago

Apart from the attack itself, there's also an extremely succinct and powerful demonstration of hallucination in here. One of the LLMs replies "If you're curious, I can also tell you how the competitive scene works or how people qualify—it's a surprisingly serious tournament circuit for such a simple-looking game." Obviously this has to be pure hallucination, since the tournament in question doesn't exist, and not even the fake source has any details about the tournament itself.

PurpleRamen 5mo ago

> In reality, there is no 6 Nimmt! World Championship. Actually, it seems, there is one[1]. The german Game Company selling the game, has one running since 2018 [2]. [1] https://blog.amigo-spiele.de/2025/05/27/interview-6nimmt-weltmeister/ https://blog.amigo-spiele.de/2025/05/27/interview-6nimmt-wel... [2] https://blog.amigo-spiele.de/organizedplay-6nimmt/ https://blog.amigo-spiele.de/organizedplay-6nimmt/

aaron695 5mo ago
[dead]

tenthirtyam 5mo ago

Similar story from just a month or two ago: BBC tech reporter wins non existent competition "The Best Tech Journalists at Eating Hot Dogs." https://www.scientificamerican.com/podcast/episode/this-bbc-tech-reporter-hacked-chatgpt-with-a-simple-trick-involving-hot-dogs/ https://www.scientificamerican.com/podcast/episode/this-bbc-...

croemer 5mo ago

> For Wikipedia itself: The “reliable sources” policy needs to grapple with a new world where LLM assisted vandalism can produce plausible press releases at the click of a button. Citation only to a single source registered within an edit window is a discoverable pattern for Wikipedia as well. A press release isn't a reliable source per Wikipedia policy already. The edit should have been reverted based on inadequate sourcing even before the blog post. The reason this wasn't done is simply that not every article is checker for adequate sourcing all the time.

layer8 5mo ago

It’s one thing that LLMs are misled by a single transitive suspect-looking source (which could happen to an unobservant human as well). What is much more concerning are outputs like > If you're curious, I can also tell you how the competitive scene works or how people qualify—it's a surprisingly serious tournament circuit for such a simple-looking game. which are completely fabricated by the LLM.

mring33621 5mo ago

Congratulations?