3 ms·

The biggest evidence that LLMs can’t reason is hallucinations. If it could reason it would have rejected fictional generated output that make no sense.

layer8 3y ago
Maybe it’s more accurate to say that LLMs lack (self-)awareness. Because when you point out things that make no sense, they do have some limited ability to produce reasoning about that. But I agree that this lack of awareness is a serious and maybe fundamental deficit.
- worrycue 3y ago
  And how often does it get that wrong too? It’s more likely it’s just, once again, generating the most probable answer - and if you shake the magic 8 ball enough you will get the answer you were expecting.
  - TeMPOraL 3y ago
    Yeah, but the thing is, this seems exactly what people are doing too, at the boundary of conscious and unconscious, with the "inner voice" being most directly comparable to LLMs. It too generates language that feels like best completion, regardless of whether the output is logically correct or not.
burnished 3y ago
Wouldn't that be evidence at most that its reasoning was flawed, or could contain errors?
naasking 3y ago
> The biggest evidence that LLMs can’t reason is hallucinations. If I asked you a question and you had to respond with a stream of consciousness reply, no time to reflect on the question and think about your reply, how inaccurate would your response be? The "hallucinations" aren't a problem with the LLM per se, but how we use them. Papers have shown that feeding the output back into the input, as happens when humans iterate on their own initial thoughts, helps tremendously with accuracy.
TeMPOraL 3y ago
In comparing to human minds, LLMs are better understood as the "inner voice" part, not the mind. From that perspective, it's eerie how similar the two are in success and failure modes alike. Yes, I'm saying here that peoples' inner voices are hallucinating in very similar fashion; "rejecting fictional generated output that makes no sense" is a process that's consciously observable and involves looping the inner voice on itself.

layer8 3y ago

Maybe it’s more accurate to say that LLMs lack (self-)awareness. Because when you point out things that make no sense, they do have some limited ability to produce reasoning about that. But I agree that this lack of awareness is a serious and maybe fundamental deficit.

worrycue 3y ago
And how often does it get that wrong too? It’s more likely it’s just, once again, generating the most probable answer - and if you shake the magic 8 ball enough you will get the answer you were expecting.
- TeMPOraL 3y ago
  Yeah, but the thing is, this seems exactly what people are doing too, at the boundary of conscious and unconscious, with the "inner voice" being most directly comparable to LLMs. It too generates language that feels like best completion, regardless of whether the output is logically correct or not.

burnished 3y ago

Wouldn't that be evidence at most that its reasoning was flawed, or could contain errors?

naasking 3y ago

> The biggest evidence that LLMs can’t reason is hallucinations. If I asked you a question and you had to respond with a stream of consciousness reply, no time to reflect on the question and think about your reply, how inaccurate would your response be? The "hallucinations" aren't a problem with the LLM per se, but how we use them. Papers have shown that feeding the output back into the input, as happens when humans iterate on their own initial thoughts, helps tremendously with accuracy.

TeMPOraL 3y ago

In comparing to human minds, LLMs are better understood as the "inner voice" part, not the mind. From that perspective, it's eerie how similar the two are in success and failure modes alike. Yes, I'm saying here that peoples' inner voices are hallucinating in very similar fashion; "rejecting fictional generated output that makes no sense" is a process that's consciously observable and involves looping the inner voice on itself.