7 ms·

> or it (shocking) cites Wikipedia instead of the BBC. No... the problem is that it cites Wikipedia articles that don't exist. > ChatGPT linked to a non-exist

> or it (shocking) cites Wikipedia instead of the BBC. No... the problem is that it cites Wikipedia articles that don't exist. > ChatGPT linked to a non-existent Wikipedia article on the “European Union Enlargement Goals for 2040”. In fact, there is no official EU policy under that name. The response hallucinates a URL but also, indirectly, an EU goal and policy.

scarmig 1y ago
> Participating organizations raised concerns about responses that relied heavily or solely on Wikipedia content – Radio-Canada calculated that of 108 sources cited in responses from ChatGPT, 58% were from Wikipedia. CBC-Radio-Canada are amongst a number of Canadian media organisations suing ChatGPT’s creator, OpenAI, for copyright infringement. Although the impact of this on ChatGPT’s approach to sourcing is not explicitly known, it may explain the high use of Wikipedia sources. Also, is attributing, without any citation, ChatGPT's preference for Wikipedia to a reprisal to an active lawsuit a significant issue? Or do the authors get off scot-free because they caged it in "we don't know, but maybe it's the case"?
- deleted 1y ago
  [deleted]
- ffsm8 1y ago
  Literally constantly? It takes both careful prompting and throughout double-checking to really notice however. Because often the links also exist, just don't represent what the LLM made it sound like. And the worst part about the people unironically thinking they can use it for "research" is, that it essentially supercharges confirmation bias. The inefficient sidequests you do while researching is generally what actually gives you the ability to really reason about a topic. If you instead just laser focus on the tidbits you prompted with... Well, your opinion is a lot less grounded.
  - edavison1 1y ago
    Ran into this the other day researching a brewery. Google AI summary referenced a glowing NYT profile of its beers. The linked article was not in fact about that brewery, but an entirely different one. Brewery I was researching has never been mentioned in the NYT. Complete invention at that point and has 'stolen' the good press from a different place and just fed the user what they wanted to see, namely a recommendation for the thing I was googling.
- terminalshort 1y ago
  It's a huge issue. No wonder AI hallucinates when it trains on this kind of crap.
kenjackson 1y ago
Actually there was a Wikipedia article of this name, but it was deleted in June -- because it was AI generated. Unfortunately AI falls for this much like humans do. https://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletion/European_Union_Enlargement_Goals_for_2040 https://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletio...
- Workaccount2 1y ago
  This is likely because of the knowledge cutoff. I have seen a few cases before of "hallucinations" that turned out to be things that did exist, but no longer do.
  - 1980phipsi 1y ago
    The fix for this is for the AI to double-check all links before providing them to the user. I frequently ask ChatGPT to double check that references actually exist when it gives me them. It should be built in!
    - rideontime 1y ago
      But that would mean OpenAI would lose even more money on every query.
      - mdhb 1y ago
        Almost as though it’s not a sustainable business model and relies of tricking people in order to keep the lights on.
      - ModernMech 1y ago
        Better make each query count then.
    - blitzar 1y ago
      I have found my self doing the same "citation needed" loop - but with ai this is a dangerous game as it will now double down on whatever it made up and go looking for citations to justify its answer. Pre prompting to cite sources is obviously a better way of going about things.
menaerus 1y ago
> For the current research, a set of 30 “core” news questions was developed Right. Let's talk about statistics for a bit. Or let's put it differently: they found in their report that 45% of the answers for 30 questions they have "developed" had a significant issue, e.g. inexisting reference I'll give you 30 questions out of my sleeve where 95% of the answers will not have any significant issue.
- matthewmacleod 1y ago
  Yes, I'm sure you could hack together some bullshit questions to demonstrate whatever you want. Is there a specific reason that the reasonably straightforward methodology they did use is somehow flawed?
  - menaerus 1y ago
    Yes, and you answered it yourself.
    - darkwater 1y ago
      Err, no? Being _possible_ does not necessarily imply that's what happened.
      - menaerus 1y ago
        A bucket of 30 questions is not a statistically significant sample size which we can use to support the hypothesis which goes to say that all AI assistants they tested are 45% of the time wrong. That's not how science works. Neither is my bucket of 30 questions statistcally significant but it goes to say that I can disprove their hypothesis just by giving them my sample. I think that the report is being disingenious and I don't understand for what reasons. it's funny that they say "misrepresent" when that's exactly what they are doing.
        extrabajs 1y ago
        Statistically significant... sample size? Support the hypothesis?
hnuser123456 1y ago
Do we have any good research on how much less often larger, newer models will just make stuff up like this? As it is, it's pretty clear LLMs are categorically not a good idea for directly querying for information in any non-fiction-writing context. If you're using an LLM to research something that needs to be accurate, the LLM needs to be doing a tool call to a web search and only asked to summarize relevant facts from the existing information it can find, and have them be cited by hard-coding the UI to link the pages the LLM reviewed. The LLM itself cannot be trusted to generate its own citations. It will just generate something that looks like a relevant citation, along with whatever imaginary content it wants to attribute to this non-existent source.
- jacobolus 1y ago
  A further problem is that Wikipedia is chock full of nonsense, with a large proportion of articles that were never fact checked by an expert, and many that were written to promote various biased points of view, inadvertently uncritically repeat claims from slanted sources, or mischaracterize claims made in good sources. Many if not most articles have poor choice of emphasis of subtopics, omit important basic topics, and make routine factual errors. (This problem is not unique to Wikipedia by any means, and despite its flaws Wikipedia is an amazing achievement.) A critical human reader can go as deep as they like in examining claims there: can look at the source listed for a claim, can often click through to read the claim in the source, can examine the talk page and article history, can search through the research literature trying to figure out where the claim came from or how it mutated in passing from source to source, etc. But an AI "reader" is a predictive statistical model, not a critical consumer of information.
  - senderista 1y ago
    Just the other day, I clicked through to a Wikipedia reference (a news article) and discovered that the citing sentence grossly misrepresented the source. Probably not accidental since it was about a politically charged subject.
  - zahlman 1y ago
    > many that were written to promote various biased points of view, inadvertently uncritically repeat claims from slanted sources, or mischaracterize claims made in good sources. Yep. Including, if not especially, the ones actively worked on by the most active contributors. The process for vetting sources (both in terms of suitability for a particular article, and general "reliable sources" status) is also seriously problematic. Especially when it comes to any topic which fundamentally relates to the reliability of journalism and the media in general.
shinycode 1y ago
I used perplexity for searches and I clicked on all sources that were given. Depending on the model used from 100% to 20% of the urls I tested did not exist. I kept on querying the LLM about it and it finally told me that it generated « the most probable » urls for the topic in question based on the ones he knows exists. Useless.
- smrq 1y ago
  I share your opinion on the results, but why would you trust the LLM explanation for why it does what it does?
  - shinycode 1y ago
    I don’t trust it at all. I wanted to know if he would be able to explain its own results. Just because it was displaying sources and links made me trust it until I checked and was horrified. I wanted to know if it was old link that broke or changed but no apparently
    - macintux 1y ago
      You said: >...it finally told me that it generated « the most probable » urls for the topic in question based on the ones he knows exists. smrq is asking why you would believe that explanation. The LLM doesn't necessarily know why it's doing what it's doing, so that could be another hallucination. Your answer: > ...I wanted to know if it was old link that broke or changed but no apparently Leads me to believe that you misunderstood smrq's question.
      - shinycode 1y ago
        No I got the question, I said that I wanted to see what kind of explanation it would give me. Ofc it can hallucinate that explanation as well. The bottom line is I don’t trust it, and the source link are fake (and not broken or obsolete)
aflag 1y ago
Existing is just a point in time

scarmig 1y ago

> Participating organizations raised concerns about responses that relied heavily or solely on Wikipedia content – Radio-Canada calculated that of 108 sources cited in responses from ChatGPT, 58% were from Wikipedia. CBC-Radio-Canada are amongst a number of Canadian media organisations suing ChatGPT’s creator, OpenAI, for copyright infringement. Although the impact of this on ChatGPT’s approach to sourcing is not explicitly known, it may explain the high use of Wikipedia sources. Also, is attributing, without any citation, ChatGPT's preference for Wikipedia to a reprisal to an active lawsuit a significant issue? Or do the authors get off scot-free because they caged it in "we don't know, but maybe it's the case"?

deleted 1y ago
[deleted]
ffsm8 1y ago
Literally constantly? It takes both careful prompting and throughout double-checking to really notice however. Because often the links also exist, just don't represent what the LLM made it sound like. And the worst part about the people unironically thinking they can use it for "research" is, that it essentially supercharges confirmation bias. The inefficient sidequests you do while researching is generally what actually gives you the ability to really reason about a topic. If you instead just laser focus on the tidbits you prompted with... Well, your opinion is a lot less grounded.
- edavison1 1y ago
  Ran into this the other day researching a brewery. Google AI summary referenced a glowing NYT profile of its beers. The linked article was not in fact about that brewery, but an entirely different one. Brewery I was researching has never been mentioned in the NYT. Complete invention at that point and has 'stolen' the good press from a different place and just fed the user what they wanted to see, namely a recommendation for the thing I was googling.
terminalshort 1y ago
It's a huge issue. No wonder AI hallucinates when it trains on this kind of crap.

kenjackson 1y ago

Actually there was a Wikipedia article of this name, but it was deleted in June -- because it was AI generated. Unfortunately AI falls for this much like humans do. https://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletion/European_Union_Enlargement_Goals_for_2040 https://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletio...

Workaccount2 1y ago
This is likely because of the knowledge cutoff. I have seen a few cases before of "hallucinations" that turned out to be things that did exist, but no longer do.
- 1980phipsi 1y ago
  The fix for this is for the AI to double-check all links before providing them to the user. I frequently ask ChatGPT to double check that references actually exist when it gives me them. It should be built in!
  - rideontime 1y ago
    But that would mean OpenAI would lose even more money on every query.
    - mdhb 1y ago
      Almost as though it’s not a sustainable business model and relies of tricking people in order to keep the lights on.
    - ModernMech 1y ago
      Better make each query count then.
  - blitzar 1y ago
    I have found my self doing the same "citation needed" loop - but with ai this is a dangerous game as it will now double down on whatever it made up and go looking for citations to justify its answer. Pre prompting to cite sources is obviously a better way of going about things.

menaerus 1y ago

> For the current research, a set of 30 “core” news questions was developed Right. Let's talk about statistics for a bit. Or let's put it differently: they found in their report that 45% of the answers for 30 questions they have "developed" had a significant issue, e.g. inexisting reference I'll give you 30 questions out of my sleeve where 95% of the answers will not have any significant issue.

matthewmacleod 1y ago
Yes, I'm sure you could hack together some bullshit questions to demonstrate whatever you want. Is there a specific reason that the reasonably straightforward methodology they did use is somehow flawed?
- menaerus 1y ago
  Yes, and you answered it yourself.
  - darkwater 1y ago
    Err, no? Being _possible_ does not necessarily imply that's what happened.
    - menaerus 1y ago
      A bucket of 30 questions is not a statistically significant sample size which we can use to support the hypothesis which goes to say that all AI assistants they tested are 45% of the time wrong. That's not how science works. Neither is my bucket of 30 questions statistcally significant but it goes to say that I can disprove their hypothesis just by giving them my sample. I think that the report is being disingenious and I don't understand for what reasons. it's funny that they say "misrepresent" when that's exactly what they are doing.
      - extrabajs 1y ago
        Statistically significant... sample size? Support the hypothesis?

hnuser123456 1y ago

Do we have any good research on how much less often larger, newer models will just make stuff up like this? As it is, it's pretty clear LLMs are categorically not a good idea for directly querying for information in any non-fiction-writing context. If you're using an LLM to research something that needs to be accurate, the LLM needs to be doing a tool call to a web search and only asked to summarize relevant facts from the existing information it can find, and have them be cited by hard-coding the UI to link the pages the LLM reviewed. The LLM itself cannot be trusted to generate its own citations. It will just generate something that looks like a relevant citation, along with whatever imaginary content it wants to attribute to this non-existent source.

jacobolus 1y ago
A further problem is that Wikipedia is chock full of nonsense, with a large proportion of articles that were never fact checked by an expert, and many that were written to promote various biased points of view, inadvertently uncritically repeat claims from slanted sources, or mischaracterize claims made in good sources. Many if not most articles have poor choice of emphasis of subtopics, omit important basic topics, and make routine factual errors. (This problem is not unique to Wikipedia by any means, and despite its flaws Wikipedia is an amazing achievement.) A critical human reader can go as deep as they like in examining claims there: can look at the source listed for a claim, can often click through to read the claim in the source, can examine the talk page and article history, can search through the research literature trying to figure out where the claim came from or how it mutated in passing from source to source, etc. But an AI "reader" is a predictive statistical model, not a critical consumer of information.
- senderista 1y ago
  Just the other day, I clicked through to a Wikipedia reference (a news article) and discovered that the citing sentence grossly misrepresented the source. Probably not accidental since it was about a politically charged subject.
- zahlman 1y ago
  > many that were written to promote various biased points of view, inadvertently uncritically repeat claims from slanted sources, or mischaracterize claims made in good sources. Yep. Including, if not especially, the ones actively worked on by the most active contributors. The process for vetting sources (both in terms of suitability for a particular article, and general "reliable sources" status) is also seriously problematic. Especially when it comes to any topic which fundamentally relates to the reliability of journalism and the media in general.

shinycode 1y ago

I used perplexity for searches and I clicked on all sources that were given. Depending on the model used from 100% to 20% of the urls I tested did not exist. I kept on querying the LLM about it and it finally told me that it generated « the most probable » urls for the topic in question based on the ones he knows exists. Useless.

smrq 1y ago
I share your opinion on the results, but why would you trust the LLM explanation for why it does what it does?
- shinycode 1y ago
  I don’t trust it at all. I wanted to know if he would be able to explain its own results. Just because it was displaying sources and links made me trust it until I checked and was horrified. I wanted to know if it was old link that broke or changed but no apparently
  - macintux 1y ago
    You said: >...it finally told me that it generated « the most probable » urls for the topic in question based on the ones he knows exists. smrq is asking why you would believe that explanation. The LLM doesn't necessarily know why it's doing what it's doing, so that could be another hallucination. Your answer: > ...I wanted to know if it was old link that broke or changed but no apparently Leads me to believe that you misunderstood smrq's question.
    - shinycode 1y ago
      No I got the question, I said that I wanted to see what kind of explanation it would give me. Ofc it can hallucinate that explanation as well. The bottom line is I don’t trust it, and the source link are fake (and not broken or obsolete)

aflag 1y ago

Existing is just a point in time