4 ms·

It seems to me that even a lot of technical people are ignoring this. A lot of very smart folk seem to think that the ChatGPT either is very close to reaching A

by nfw2 4y ago

It seems to me that even a lot of technical people are ignoring this. A lot of very smart folk seem to think that the ChatGPT either is very close to reaching AGI or already has. The inability to reason about about whether or not what it is writing is true seems like a fundamental blocker to me, and not necessarily one that can be overcome simply by adding compute resources. Can we trust AI to make critical decisions if we have no understanding for when and why it "hallucinates"?

dragonwriter 4y ago
> The inability to reason about about whether or not what it is writing is true seems like a fundamental blocker to me How can you reason about what is true without any source of truth? And once you give ChatGPT external resources and a framework like ReAct, it is much better at reasoning about truth. (I don’t think ChatGPT is anywhere close to AGI, but at the same time I find “when you treat it like a brain in a jar with no access to any resources outside of the conversation and talk to it, it doesn’t know what is true and what isn’t” to be a very convicing argument against it being close to AGI.)
int_19h 4y ago
I don't think it's very close to reaching AGI, but I also don't see what that has to do with lying (or hallucinating). Even when it hallucinates the data, it can still soundly reason from it, and to my mind it's the latter part that is key. As for trust... well, no, we can't. But the same question applies to humans. The real concern to me is that these things will get used as a replacement long before the hallucination rate and severity is on par with the humans that they replace. One other interesting thing is that GPT-4 in particular is surprisingly good at catching itself. That is, it might write some nonsense, but if you ask it to analyze and criticize its own answer, it can spot the nonsense! This actually makes sense from a human perspective - if someone asks you a serious question that requires deliberation, you'll probably think it through verbally internally (or out loud, if the format allows) before actually answering, and you'll review your own premises and reasoning in the process. I expect that we'll end up doing something similar to the LLM, such that immediate output is treated as "thinking", and there's some back and forth internally before the actual user-visible answer is produced. This doesn't really solve the hallucination problem - and I don't think anything really can? - but it might drastically improve matters, especially if we combine different models, some of which are specifically fine-tuned for nitpicking and scathing critique.

by nfw2 4y ago

dragonwriter 4y ago

> The inability to reason about about whether or not what it is writing is true seems like a fundamental blocker to me How can you reason about what is true without any source of truth? And once you give ChatGPT external resources and a framework like ReAct, it is much better at reasoning about truth. (I don’t think ChatGPT is anywhere close to AGI, but at the same time I find “when you treat it like a brain in a jar with no access to any resources outside of the conversation and talk to it, it doesn’t know what is true and what isn’t” to be a very convicing argument against it being close to AGI.)

int_19h 4y ago

I don't think it's very close to reaching AGI, but I also don't see what that has to do with lying (or hallucinating). Even when it hallucinates the data, it can still soundly reason from it, and to my mind it's the latter part that is key. As for trust... well, no, we can't. But the same question applies to humans. The real concern to me is that these things will get used as a replacement long before the hallucination rate and severity is on par with the humans that they replace. One other interesting thing is that GPT-4 in particular is surprisingly good at catching itself. That is, it might write some nonsense, but if you ask it to analyze and criticize its own answer, it can spot the nonsense! This actually makes sense from a human perspective - if someone asks you a serious question that requires deliberation, you'll probably think it through verbally internally (or out loud, if the format allows) before actually answering, and you'll review your own premises and reasoning in the process. I expect that we'll end up doing something similar to the LLM, such that immediate output is treated as "thinking", and there's some back and forth internally before the actual user-visible answer is produced. This doesn't really solve the hallucination problem - and I don't think anything really can? - but it might drastically improve matters, especially if we combine different models, some of which are specifically fine-tuned for nitpicking and scathing critique.