4 ms·

I’m surprised how many users of ChatGPT don’t realize how often it makes things up. I had a conversation with an Uber driver the other day who said he used Chat

I’m surprised how many users of ChatGPT don’t realize how often it makes things up. I had a conversation with an Uber driver the other day who said he used ChatGPT all the time. At one point I mentioned its tendency to make stuff up, and he didn’t know what I was talking about. I can think of at least two other non-technical people I’ve spoken with who had the same reaction.

kazinator 4y ago
ChatGPT is always making things up. It is correct when the things it makes up come from fragments of training data which happened to be correct, and didn't get mangled in the transformation. Just like when a diffusion model is "correct" when it creates a correct shadow or perspective, and "incorrect" when not. But both images are made up. It's the same thing with statements. A statement can correspond to something in the world, and thus be true: like a correct shadow. Or not, like a bad shadow. But in both cases, it's just made-up drivel. If it turns out that there is a teacup orbiting Jupiter, that doesn't mean that postulating its existence on a whim had been valid. Truth requires provenance, not only correspondence which can be coincidental.
- cultureswitch 4y ago
  Something can be truthfully describing the world by chance. The statement would still be true. In fact every statement we make about the world we live in is like this.
  - kazinator 4y ago
    The original assertion of such a statement has no basis, and continues not to even when the statement is independently discovered to be true. No, not every statement we make about the world is a baseless assertion that is true by chance.
nfw2 4y ago
It seems to me that even a lot of technical people are ignoring this. A lot of very smart folk seem to think that the ChatGPT either is very close to reaching AGI or already has. The inability to reason about about whether or not what it is writing is true seems like a fundamental blocker to me, and not necessarily one that can be overcome simply by adding compute resources. Can we trust AI to make critical decisions if we have no understanding for when and why it "hallucinates"?
- dragonwriter 4y ago
  > The inability to reason about about whether or not what it is writing is true seems like a fundamental blocker to me How can you reason about what is true without any source of truth? And once you give ChatGPT external resources and a framework like ReAct, it is much better at reasoning about truth. (I don’t think ChatGPT is anywhere close to AGI, but at the same time I find “when you treat it like a brain in a jar with no access to any resources outside of the conversation and talk to it, it doesn’t know what is true and what isn’t” to be a very convicing argument against it being close to AGI.)
- int_19h 4y ago
  I don't think it's very close to reaching AGI, but I also don't see what that has to do with lying (or hallucinating). Even when it hallucinates the data, it can still soundly reason from it, and to my mind it's the latter part that is key. As for trust... well, no, we can't. But the same question applies to humans. The real concern to me is that these things will get used as a replacement long before the hallucination rate and severity is on par with the humans that they replace. One other interesting thing is that GPT-4 in particular is surprisingly good at catching itself. That is, it might write some nonsense, but if you ask it to analyze and criticize its own answer, it can spot the nonsense! This actually makes sense from a human perspective - if someone asks you a serious question that requires deliberation, you'll probably think it through verbally internally (or out loud, if the format allows) before actually answering, and you'll review your own premises and reasoning in the process. I expect that we'll end up doing something similar to the LLM, such that immediate output is treated as "thinking", and there's some back and forth internally before the actual user-visible answer is produced. This doesn't really solve the hallucination problem - and I don't think anything really can? - but it might drastically improve matters, especially if we combine different models, some of which are specifically fine-tuned for nitpicking and scathing critique.
sebzim4500 4y ago
I think part of this is that in some domains it very rarely makes things up. If a kid uses for help with their history homework it will probably be 100% correct, because everything they ask it appears a thousand times in the training set.
rocketnasa 4y ago
> I’m surprised how many users of ChatGPT don’t realize how often it makes things up. "I had not realized ... that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people." - Weizenbaum, 1976
missingcolours 4y ago
Even people who have read news articles about ChatGPT lying often think "that was a weird, once in a blue moon bug that will be fixed soon" as opposed to "yes that's how this works and it happens constantly". I asked ChatGPT and Bard some factual questions about elected officials and upcoming election dates. 80% of them were flat out wrong. They told me most US states were having gubernatorial elections this November, that the governor of Texas is an independent, etc - simple basic facts that you Wikipedia could show you are wrong.

kazinator 4y ago

ChatGPT is always making things up. It is correct when the things it makes up come from fragments of training data which happened to be correct, and didn't get mangled in the transformation. Just like when a diffusion model is "correct" when it creates a correct shadow or perspective, and "incorrect" when not. But both images are made up. It's the same thing with statements. A statement can correspond to something in the world, and thus be true: like a correct shadow. Or not, like a bad shadow. But in both cases, it's just made-up drivel. If it turns out that there is a teacup orbiting Jupiter, that doesn't mean that postulating its existence on a whim had been valid. Truth requires provenance, not only correspondence which can be coincidental.

cultureswitch 4y ago
Something can be truthfully describing the world by chance. The statement would still be true. In fact every statement we make about the world we live in is like this.
- kazinator 4y ago
  The original assertion of such a statement has no basis, and continues not to even when the statement is independently discovered to be true. No, not every statement we make about the world is a baseless assertion that is true by chance.

nfw2 4y ago

It seems to me that even a lot of technical people are ignoring this. A lot of very smart folk seem to think that the ChatGPT either is very close to reaching AGI or already has. The inability to reason about about whether or not what it is writing is true seems like a fundamental blocker to me, and not necessarily one that can be overcome simply by adding compute resources. Can we trust AI to make critical decisions if we have no understanding for when and why it "hallucinates"?

dragonwriter 4y ago
> The inability to reason about about whether or not what it is writing is true seems like a fundamental blocker to me How can you reason about what is true without any source of truth? And once you give ChatGPT external resources and a framework like ReAct, it is much better at reasoning about truth. (I don’t think ChatGPT is anywhere close to AGI, but at the same time I find “when you treat it like a brain in a jar with no access to any resources outside of the conversation and talk to it, it doesn’t know what is true and what isn’t” to be a very convicing argument against it being close to AGI.)
int_19h 4y ago
I don't think it's very close to reaching AGI, but I also don't see what that has to do with lying (or hallucinating). Even when it hallucinates the data, it can still soundly reason from it, and to my mind it's the latter part that is key. As for trust... well, no, we can't. But the same question applies to humans. The real concern to me is that these things will get used as a replacement long before the hallucination rate and severity is on par with the humans that they replace. One other interesting thing is that GPT-4 in particular is surprisingly good at catching itself. That is, it might write some nonsense, but if you ask it to analyze and criticize its own answer, it can spot the nonsense! This actually makes sense from a human perspective - if someone asks you a serious question that requires deliberation, you'll probably think it through verbally internally (or out loud, if the format allows) before actually answering, and you'll review your own premises and reasoning in the process. I expect that we'll end up doing something similar to the LLM, such that immediate output is treated as "thinking", and there's some back and forth internally before the actual user-visible answer is produced. This doesn't really solve the hallucination problem - and I don't think anything really can? - but it might drastically improve matters, especially if we combine different models, some of which are specifically fine-tuned for nitpicking and scathing critique.

sebzim4500 4y ago

I think part of this is that in some domains it very rarely makes things up. If a kid uses for help with their history homework it will probably be 100% correct, because everything they ask it appears a thousand times in the training set.

rocketnasa 4y ago

> I’m surprised how many users of ChatGPT don’t realize how often it makes things up. "I had not realized ... that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people." - Weizenbaum, 1976

missingcolours 4y ago

Even people who have read news articles about ChatGPT lying often think "that was a weird, once in a blue moon bug that will be fixed soon" as opposed to "yes that's how this works and it happens constantly". I asked ChatGPT and Bard some factual questions about elected officials and upcoming election dates. 80% of them were flat out wrong. They told me most US states were having gubernatorial elections this November, that the governor of Texas is an independent, etc - simple basic facts that you Wikipedia could show you are wrong.