3 ms·

A few weeks ago, I had a question and I couldn't find the answer with Wikipedia or with internet searches. So I asked ChatGPT. I got an answer. Even better, it

A few weeks ago, I had a question and I couldn't find the answer with Wikipedia or with internet searches. So I asked ChatGPT. I got an answer. Even better, it was sourced... Except that the source was a Wikipedia page that I had read and that didn't contain what I wanted. ChatGPT's answer was "fouled up beautifully", as Donald Knuth would say. A plain search is just a link and a quote, while the IA's answer is an educated guess. TL;DR ChatGPT can "lie" and its bad extrapolations are hard to debunk.

manmal 3y ago
Hallucinations have always been a problem with transformers. But with the replication crisis in many sciences and political meddling abound, I‘d be wary of any kind of critical information, especially on Wikipedia. Of course, transformer models have the same issues since they are trained on tainted info too.
- moritzwarhier 3y ago
  > But with the replication crisis in many sciences and political meddling abound, I‘d be wary of any kind of critical information, especially on Wikipedia. Of course, transformer models have the same issues I think this is a bad comparison. The issues OP described are not limited to tainted training data. Being critical of Wikipedia and critical of science is not equal to treating text as a black box looking for the most fitting continuation. LLMs are not fact machines. When you equate this kind of error with the scientific replication crisis or generally a crtique of scientific methods and political influence (what about commercial?), I don't think this demonstrates critical thinking.
  - manmal 3y ago
    Can you point to where in my comment I asserted that hallucinations are equal to other sources of errors, or even compared them? Because I tend to agree with you. I tried to point out that info found externally always has to be cross-checked, no matter where it originates.
    - moritzwarhier 3y ago
      I meant the sentence I quoted, or the part > [...] especially on Wikipedia. Of course, transformer models have the same issues I agree that one should be critical of all sources, including Wikipedia. I don't agree that GPT is an information source comparable to Wikipedia or scientific studies. Both can be wrong, biased or incorrect though. In other words, I would never consider the usage of any LLM worth to cross-check information found elsewhere, as opposed to e.g. Wikipedia. Which does not mean that I consider the info compiled on Wikipedia always as trsutworthy. Then again, I wouldn't use an LLM to cross-check Wikipedia. They're not equals. Sorry if I misunderstood anything in your comment.
      - manmal 3y ago
        Using Wikipedia to cross-check can be dangerous since it often is the source of the content you‘re trying to verify.
        moritzwarhier 3y ago
        This is surely true and can be a danger in many cases. Still, Wikipedia is curated by humans and in most cases I would argue that articles improve over time, errors get corrected etc. I'm not arguing that Wikipedia is a good primary source, I still think it is very suitable as a point of entry if you want to cross-check information from other sources. A user-editable encyclopedia is not perfect, but it is not comparable to automatically generated text that has no regard for correctness, that only tries to fit its training data and prompt. For example, I wouldn't trust GPT when asking for the height of some building in my city. I would consider it likely though that it confidently gives ke a wrong number. Without all the RLHF training to refuse "best guess" answers, situation would be even more bleak.
lkramer 3y ago
I had similar experience; a friend use chatGPT to answer a question we had (was Stalin's son involved in the crash of an airplane carrying a Hockey team?). And it combined two separate events, a crash in 1953 which was actually the one we were thinking of, and another from 2011. It didn't give any sources, though I guess it may have been able to, but it was very confident in its answer and I only realised it was wrong because I was curious and wanted to find out more, but that certainly made me look at chatGPT differently. I don't think LLMs are in any way the replacement for search engines, certainly not in their current form.
- danielbln 3y ago
  Yes they are, though not in isolation. Additional mechanisms are needed to inject ground truth reliable knowledge into the LLM context. Don't ask an isolated LLM about facts and expect no hallucinations. Instead, use ChatGPT with search plug-in, Bing search, Perplexity.ai or Phind.com (Im sure there are others). Once reliable truth available to the LLM via context, hallucinations go down _significantly_ to the point where (to me at least) it's no longer really much of an issue.
danielbln 3y ago
Not to take away from your observations, but ChatGPT has been around for close to a year now and LLM hallucinations have been talked about at length basically everywhere. That's far from a new or surprising thing at this point and in fact there is a plethora of mitigation strategies available (mostly centered around additional external mechanisms that find or validate the truth the LLM works with).

manmal 3y ago

Hallucinations have always been a problem with transformers. But with the replication crisis in many sciences and political meddling abound, I‘d be wary of any kind of critical information, especially on Wikipedia. Of course, transformer models have the same issues since they are trained on tainted info too.

moritzwarhier 3y ago
> But with the replication crisis in many sciences and political meddling abound, I‘d be wary of any kind of critical information, especially on Wikipedia. Of course, transformer models have the same issues I think this is a bad comparison. The issues OP described are not limited to tainted training data. Being critical of Wikipedia and critical of science is not equal to treating text as a black box looking for the most fitting continuation. LLMs are not fact machines. When you equate this kind of error with the scientific replication crisis or generally a crtique of scientific methods and political influence (what about commercial?), I don't think this demonstrates critical thinking.
- manmal 3y ago
  Can you point to where in my comment I asserted that hallucinations are equal to other sources of errors, or even compared them? Because I tend to agree with you. I tried to point out that info found externally always has to be cross-checked, no matter where it originates.
  - moritzwarhier 3y ago
    I meant the sentence I quoted, or the part > [...] especially on Wikipedia. Of course, transformer models have the same issues I agree that one should be critical of all sources, including Wikipedia. I don't agree that GPT is an information source comparable to Wikipedia or scientific studies. Both can be wrong, biased or incorrect though. In other words, I would never consider the usage of any LLM worth to cross-check information found elsewhere, as opposed to e.g. Wikipedia. Which does not mean that I consider the info compiled on Wikipedia always as trsutworthy. Then again, I wouldn't use an LLM to cross-check Wikipedia. They're not equals. Sorry if I misunderstood anything in your comment.
    - manmal 3y ago
      Using Wikipedia to cross-check can be dangerous since it often is the source of the content you‘re trying to verify.
      - moritzwarhier 3y ago
        This is surely true and can be a danger in many cases. Still, Wikipedia is curated by humans and in most cases I would argue that articles improve over time, errors get corrected etc. I'm not arguing that Wikipedia is a good primary source, I still think it is very suitable as a point of entry if you want to cross-check information from other sources. A user-editable encyclopedia is not perfect, but it is not comparable to automatically generated text that has no regard for correctness, that only tries to fit its training data and prompt. For example, I wouldn't trust GPT when asking for the height of some building in my city. I would consider it likely though that it confidently gives ke a wrong number. Without all the RLHF training to refuse "best guess" answers, situation would be even more bleak.

lkramer 3y ago

I had similar experience; a friend use chatGPT to answer a question we had (was Stalin's son involved in the crash of an airplane carrying a Hockey team?). And it combined two separate events, a crash in 1953 which was actually the one we were thinking of, and another from 2011. It didn't give any sources, though I guess it may have been able to, but it was very confident in its answer and I only realised it was wrong because I was curious and wanted to find out more, but that certainly made me look at chatGPT differently. I don't think LLMs are in any way the replacement for search engines, certainly not in their current form.

danielbln 3y ago
Yes they are, though not in isolation. Additional mechanisms are needed to inject ground truth reliable knowledge into the LLM context. Don't ask an isolated LLM about facts and expect no hallucinations. Instead, use ChatGPT with search plug-in, Bing search, Perplexity.ai or Phind.com (Im sure there are others). Once reliable truth available to the LLM via context, hallucinations go down _significantly_ to the point where (to me at least) it's no longer really much of an issue.

danielbln 3y ago

Not to take away from your observations, but ChatGPT has been around for close to a year now and LLM hallucinations have been talked about at length basically everywhere. That's far from a new or surprising thing at this point and in fact there is a plethora of mitigation strategies available (mostly centered around additional external mechanisms that find or validate the truth the LLM works with).