7 ms·

Lot's of people in this thread seem desperate to not think clearly about this. Whether these tests, verbatim produce the same response on any given version isn

Lot's of people in this thread seem desperate to not think clearly about this. Whether these tests, verbatim produce the same response on any given version isn't the point. GPT4 doesn't engage in reasoning if it gets any answers right. Being "right" isn't a sign of reasoning. Given a dictionary mapping from questions to answers the index operation gets answers right, but it isn't reasoning. The purpose of the paper is to exhaustively list proxy indicators of reasoning. Clearly other tests will fail in every class listed, because the LLM isn't engaged in reasoning. Since LLMs are stocahstic you shouldnt expect "reporduction" in the same sense. The paper provides classes of problems. To reproduce it you only need to find a minor permutation of the problem in each class. But people subject to gross confirmation-bias seem only to seek out prompts which produce the right answers. P(next|prev), ie., P(answer word | prompt words) is just a dictionary lookup, and that's the optimisation objective for an LLM. It turns out sequences of inferences eg., {Socrates is a Man, All men are mortal, thef. Socrates is mortal} can be modelled by dictionary lookups of the above form -- but we're not interested in whether the answer can be found from the prompt. Whether this sequence of props can be stored as a sequence of index operations. We're interested in whether the system reasons. ie., whether the sequencing operation is inference by logical necessity. It's incomprehensible to me how gullible people are around AI today -- the Eliza effect coupled with a pseudoscientific impulse to "whatever appears to work".

lewhoo 3y ago
I was also taken by the capabilities of chatGPT but with time and with people reporting how it sometimes fails with simple tasks (even v4) I'm beginning to ascribe its superpowers a bit more to the super impressive magnitude of its training data. That isn't to say it can only somehow reflect the data, but even Logan Kilpatrick who is an OpenAI employee said in a youtube video somewhere that chatGPT absolutely can't reason (though there wasn't much of a followup as I recall). I think the problem is precisely as you state - there isn't a reliable way to reproduce the findings because the findings were produced with a bit of a dice roll, to put it simply. This also happened which Chomsky when he weighed in on the subject and some people pointed out his findings don't match theirs.
- mjburgess 3y ago
  The important thing for discussing this is that this kind of reproduction is irrelevant. Does homeopathy work? Well *I* went to a local hospital and saw people recovering by drinking water and resting. Well, yes, of course you did. That doesnt count as a reply. It misses the claim. The claim isnt that there arent an infinite number of Q/A prompts where A is correct. Of course there are, rather trivially just from the nature of a generative model. The claim is that the reason A is generated from Q is because of the co-occurant frequency of Q,A in a corpus, and that this is not a model of eg., logical inference, causal reasoning, abduction, etc. It's trivial to show that with a handful of cases of failure. Success is irrelevant, it always is. These arent claims about the engineering rigour of chatgpt, theyre claims about how it obtains A from Q. Which, if the GPT had remembered all useful Q,As for all of humanity so far, wouldn't be detectable by prompting. Indeed, even if this is so, the reason we care is when there are novel Qs. Eg., "what is the local post office's telephone number?" isnt answerable with all of human history up til 1900.
- kaba0 3y ago
  I think GPT4, and similar LLM models are the future of search engines, but are not intelligences in a more sci-fi sense, and we are unknowingly far from that.
visarga 3y ago
Top-down style of AI capability denial. You got an abstract idea, like "next token prediction" and that's all you need to know. Doesn't matter what the AI does, it has no chance to prove it reasons. Why not look at what the model does, maybe in all those tensors there are reasoning principles encoded.
- beepbooptheory 3y ago
  What is a "reasoning principle" and how might it be "encoded"? Also are we really calling it "denial" now? Because it's a little funny to make the move of simply psychologizing away the criticism, when your actual AI argument, presumably, is that inner consciousness/rationality is black boxes all the way down anyway. Like, how can you presume to look inside the mind of your critic so confidently, to assert they are in denial, but in the same breath say that such a thing is in principal impossible? Don't you think it maybe takes away the force of your argument? Or at least goes against its spirit?
- mjburgess 3y ago
  I've spent the last decade in the most technical parts of this industry; it is my job to expunge this credulousness. It is indeed trivial to show that P(A|B) is a poor model of "B => A" (and B causes A, and many other relata). Software engineers, philosophers and experimental scientists seem pretty good at seeing this --- people who "convert into" engineering are totally dumbfounded by it. P(A|B) becomes an increasingly 'useful' model of arbitrary relations as the implicit model P_m(A|B) grows to include all instances of A,B. That's what digitising all of human history and storing it in the weights of an LLM does. This all follows from basic stats you'd be taught in an applied statistics course; one never taken by most in the ML industry. (Note its still a broken model because there's an infinite number of novel instances of (A,B) pairs in most cases that cannot be modelled with this sort of inductive learning). Engineering at its heart, is a kind of pseudoscience (or, if you prefer: a magic trick). You find some heuristic which behaves as-if its the target under fragile but engineering-stable conditions. The problem with engineers who only have magic tricks in their toolkit is this credulousness. Homeopathy worked: you put people in beds and give them water and they recover (indeed, better than leaching them).
epups 3y ago
How do you know for sure how human reasoning occurs, and that it does not use similar mechanisms? (edited second question to make it focused)
- lordnacho 3y ago
  This is the real big question. We don't know how human reasoning works, but we are happy to identify, entirely based on external interaction, what is and isn't correct human reasoning. Then someone comes up with P(A|B) is not reasoning, which seems like an internal mechanism. How do we square that?
- freejazz 3y ago
  The old "I know you are, but what am I" defense
- DonaldPShimoda 3y ago
  We don't need to know how human reasoning occurs. If you ask an AI chatbot a question like "count the number of occurrences of 'l' in the word 'lollapalooza'" (and then follow up on its responses), it becomes obvious that it is not "reasoning"; it is simply following a sort-of script. Here is a summarized transcript of this exercise with ChatGPT: --- Me: Count the number of occurrences of the letter 'l' in the word 'lollapalooza'. AI: The letter 'l' appears four times in the word 'lollapalooza'. Me: Describe the process by which you arrived at this answer. AI: [Provided a description of counting characters and then highlighted the first 'l' in the word four separate times to justify its answer.] Me: You counted thief first occurrence four times. That is incorrect. AI: I apologize for the mistake in my previous response. You are absolutely right. Here's the correct count of occurrences of the letter 'l' in the word 'lollapalooza': AI: [Provided a list of 8 "occurrences" of 'l' in the word 'lollapalooza', including replacing the first 'o' with an 'l' ("occurrence" 3) and counting letters like 'p', 'z', and 'a'. --- So even though the correct answer was provided at first, the model was not able to faithfully produce responses that backed up how it arrived at that conclusion, and when pressed to double-check its work it simply made things up. Even a toddler would not have claimed 'z' and 'a' were instances of 'l'. AI chatbots do not reason. They produce text responses to prompts based on stochastic methods. Trying to conflate the issue by suggesting that "we don't know how humans reason, so how do we know AI bots don't reason" is, frankly, absurd. We can easily demonstrate that they are inconsistent and have no concept of what they are writing responses about, as shown above.
famouswaffles 3y ago
>Whether these tests, verbatim produce the same response on any given version isn't the point. The paper make a nonsensical claim but fails to back it with results. If anyone isn't doing any thinking here, it's you. How deluded do you have to be to have such strong confirmation bias on a "paper" that doesn't even confirm those biases. Chucking up the simple fact that the model is indeed getting right all the problems that were supposed to indicate a lack of reasoning as "not producing the same response verbatim" is ridiculous. Did you even stop to think about what you just wrote ? If you're so sure of GPT-4 failing a permutation of these results then by all means demonstrate that.
- mjburgess 3y ago
  I'm addressing the comments in this thread, the dialectic is: OP: Paper Commenters: Reply to paper Me: Reply to commenters You'll notice a different burden in each case, since the claim differs. My claim is that confirmation-bias replies to this paper are (at best) poorly founded. More broadly, the hypothesis that an LLM reasons is not confirmed by an infinite number of correct replies to prompts. It is immediately refuted by a systematic failure across types of reasoning (NOT q/a instances). What is my burden? As far as I can tell, only to provide what I have.
  - hackinthebochs 3y ago
    >More broadly, the hypothesis that an LLM reasons is not confirmed by an infinite number of correct replies to prompts. It is immediately refuted by a systematic failure across types of reasoning (NOT q/a instances). The question of reasoning in LLMs isn't a question of whether they employ sufficiently strong reasoning capabilities in all instances, but whether it has the capability of sufficiently strong reasoning. You can't confirm general reasoning abilities with many instances of correct responses, but you also can't disconfirm general reasoning abilities through systematic failure unless you have good reason to think it should have engaged those reasoning abilities in the test context. We know that LLMs selectively activate subnetworks based on the content of the prompt. There should be no expectation of any systematic reasoning abilities, but rather what are its capacities in ideal contexts. LLMs are way too sensitive to seemingly arbitrary features of context to rule out capacities from even seemingly systematic failures.
lukev 3y ago
To paraphrase what you're saying: an AI can't reason, because it is built to stochastically predicts tokens, which is not reasoning and which is different from the activity of reasoning. Which I agree with. But also, by observation, it can (at least some of the time) emit token sequences that emulate reasoning (at least on some simple tasks.) So perhaps it can reason in the same way a submarine can swim.
- hackinthebochs 3y ago
  >an AI can't reason, because it is built to stochastically predicts tokens, which is not reasoning and which is different from the activity of reasoning. A feature indispensable in the generation of a sequence is recovered in the limit of predicting the sequence. So prediction does not exclude higher level cognitive processes. Transformers being universal sequence2sequence modelers give reason to believe they can reach the limit.
  - mjburgess 3y ago
    There are an infinite number of models which generate the same exact (infinite) sequence. No model is "recovered in the limit". And, more severely, we're blind to the future. So of the infinite models of all of history, we're not even interested in the ones which are maximally retrospectively predictive. Almost all of those are maximally non-predictive of the future, and indeed, much worse than the ones which fail to predict the past well. So your 'recovering in the limit' alas is a dangerous kind of pseudoscience: fitting the past. We want models which are wrong for the right reasons. Not models which are right for the wrong ones. The latter fail catastrophically. The models we require enable us to simulate unrealised futures; the causal, abductive models of science (, say).
    - hackinthebochs 3y ago
      >There are an infinite number of models which generate the same exact (infinite) sequence. Only if there is no constraint on the model. But the fixed number of parameters, and the inductive bias of attention limits the space of models that are learnable. In the limit of infinite data but finite capacity for memorization, the only available solution will be to recover the information dynamic of the generating process. >So of the infinite models of all of history, we're not even interested in the ones which are maximally retrospectively predictive. Presumably the same processes that operated in the past will operate in the future. So accurately modelling the past is certainly very informative for the future.
zby 3y ago
But there are programs that do inferences - and programs work on Turing machines which are just lookup tables (with an infinite tape). Lookup tables can do inferences.
- AstralStorm 3y ago
  Apparently an LLM fails at doing these inferences repeatedly, which makes it inferior to a 3SAT solver. Much less a general computer like a Turing machine.
rhtgrg 3y ago
> It's incomprehensible to me how gullible people are around AI today -- the Eliza effect coupled with a pseudoscientific impulse to "whatever appears to work". Incomprehensible perhaps, but not even a smidge unpredictable. You knew exactly what you would find in this comment thread.

lewhoo 3y ago

I was also taken by the capabilities of chatGPT but with time and with people reporting how it sometimes fails with simple tasks (even v4) I'm beginning to ascribe its superpowers a bit more to the super impressive magnitude of its training data. That isn't to say it can only somehow reflect the data, but even Logan Kilpatrick who is an OpenAI employee said in a youtube video somewhere that chatGPT absolutely can't reason (though there wasn't much of a followup as I recall). I think the problem is precisely as you state - there isn't a reliable way to reproduce the findings because the findings were produced with a bit of a dice roll, to put it simply. This also happened which Chomsky when he weighed in on the subject and some people pointed out his findings don't match theirs.

mjburgess 3y ago
The important thing for discussing this is that this kind of reproduction is irrelevant. Does homeopathy work? Well *I* went to a local hospital and saw people recovering by drinking water and resting. Well, yes, of course you did. That doesnt count as a reply. It misses the claim. The claim isnt that there arent an infinite number of Q/A prompts where A is correct. Of course there are, rather trivially just from the nature of a generative model. The claim is that the reason A is generated from Q is because of the co-occurant frequency of Q,A in a corpus, and that this is not a model of eg., logical inference, causal reasoning, abduction, etc. It's trivial to show that with a handful of cases of failure. Success is irrelevant, it always is. These arent claims about the engineering rigour of chatgpt, theyre claims about how it obtains A from Q. Which, if the GPT had remembered all useful Q,As for all of humanity so far, wouldn't be detectable by prompting. Indeed, even if this is so, the reason we care is when there are novel Qs. Eg., "what is the local post office's telephone number?" isnt answerable with all of human history up til 1900.
kaba0 3y ago
I think GPT4, and similar LLM models are the future of search engines, but are not intelligences in a more sci-fi sense, and we are unknowingly far from that.

visarga 3y ago

Top-down style of AI capability denial. You got an abstract idea, like "next token prediction" and that's all you need to know. Doesn't matter what the AI does, it has no chance to prove it reasons. Why not look at what the model does, maybe in all those tensors there are reasoning principles encoded.

beepbooptheory 3y ago
What is a "reasoning principle" and how might it be "encoded"? Also are we really calling it "denial" now? Because it's a little funny to make the move of simply psychologizing away the criticism, when your actual AI argument, presumably, is that inner consciousness/rationality is black boxes all the way down anyway. Like, how can you presume to look inside the mind of your critic so confidently, to assert they are in denial, but in the same breath say that such a thing is in principal impossible? Don't you think it maybe takes away the force of your argument? Or at least goes against its spirit?
mjburgess 3y ago
I've spent the last decade in the most technical parts of this industry; it is my job to expunge this credulousness. It is indeed trivial to show that P(A|B) is a poor model of "B => A" (and B causes A, and many other relata). Software engineers, philosophers and experimental scientists seem pretty good at seeing this --- people who "convert into" engineering are totally dumbfounded by it. P(A|B) becomes an increasingly 'useful' model of arbitrary relations as the implicit model P_m(A|B) grows to include all instances of A,B. That's what digitising all of human history and storing it in the weights of an LLM does. This all follows from basic stats you'd be taught in an applied statistics course; one never taken by most in the ML industry. (Note its still a broken model because there's an infinite number of novel instances of (A,B) pairs in most cases that cannot be modelled with this sort of inductive learning). Engineering at its heart, is a kind of pseudoscience (or, if you prefer: a magic trick). You find some heuristic which behaves as-if its the target under fragile but engineering-stable conditions. The problem with engineers who only have magic tricks in their toolkit is this credulousness. Homeopathy worked: you put people in beds and give them water and they recover (indeed, better than leaching them).

epups 3y ago

How do you know for sure how human reasoning occurs, and that it does not use similar mechanisms? (edited second question to make it focused)

lordnacho 3y ago
This is the real big question. We don't know how human reasoning works, but we are happy to identify, entirely based on external interaction, what is and isn't correct human reasoning. Then someone comes up with P(A|B) is not reasoning, which seems like an internal mechanism. How do we square that?
freejazz 3y ago
The old "I know you are, but what am I" defense
DonaldPShimoda 3y ago
We don't need to know how human reasoning occurs. If you ask an AI chatbot a question like "count the number of occurrences of 'l' in the word 'lollapalooza'" (and then follow up on its responses), it becomes obvious that it is not "reasoning"; it is simply following a sort-of script. Here is a summarized transcript of this exercise with ChatGPT: --- Me: Count the number of occurrences of the letter 'l' in the word 'lollapalooza'. AI: The letter 'l' appears four times in the word 'lollapalooza'. Me: Describe the process by which you arrived at this answer. AI: [Provided a description of counting characters and then highlighted the first 'l' in the word four separate times to justify its answer.] Me: You counted thief first occurrence four times. That is incorrect. AI: I apologize for the mistake in my previous response. You are absolutely right. Here's the correct count of occurrences of the letter 'l' in the word 'lollapalooza': AI: [Provided a list of 8 "occurrences" of 'l' in the word 'lollapalooza', including replacing the first 'o' with an 'l' ("occurrence" 3) and counting letters like 'p', 'z', and 'a'. --- So even though the correct answer was provided at first, the model was not able to faithfully produce responses that backed up how it arrived at that conclusion, and when pressed to double-check its work it simply made things up. Even a toddler would not have claimed 'z' and 'a' were instances of 'l'. AI chatbots do not reason. They produce text responses to prompts based on stochastic methods. Trying to conflate the issue by suggesting that "we don't know how humans reason, so how do we know AI bots don't reason" is, frankly, absurd. We can easily demonstrate that they are inconsistent and have no concept of what they are writing responses about, as shown above.

famouswaffles 3y ago

>Whether these tests, verbatim produce the same response on any given version isn't the point. The paper make a nonsensical claim but fails to back it with results. If anyone isn't doing any thinking here, it's you. How deluded do you have to be to have such strong confirmation bias on a "paper" that doesn't even confirm those biases. Chucking up the simple fact that the model is indeed getting right all the problems that were supposed to indicate a lack of reasoning as "not producing the same response verbatim" is ridiculous. Did you even stop to think about what you just wrote ? If you're so sure of GPT-4 failing a permutation of these results then by all means demonstrate that.

mjburgess 3y ago
I'm addressing the comments in this thread, the dialectic is: OP: Paper Commenters: Reply to paper Me: Reply to commenters You'll notice a different burden in each case, since the claim differs. My claim is that confirmation-bias replies to this paper are (at best) poorly founded. More broadly, the hypothesis that an LLM reasons is not confirmed by an infinite number of correct replies to prompts. It is immediately refuted by a systematic failure across types of reasoning (NOT q/a instances). What is my burden? As far as I can tell, only to provide what I have.
- hackinthebochs 3y ago
  >More broadly, the hypothesis that an LLM reasons is not confirmed by an infinite number of correct replies to prompts. It is immediately refuted by a systematic failure across types of reasoning (NOT q/a instances). The question of reasoning in LLMs isn't a question of whether they employ sufficiently strong reasoning capabilities in all instances, but whether it has the capability of sufficiently strong reasoning. You can't confirm general reasoning abilities with many instances of correct responses, but you also can't disconfirm general reasoning abilities through systematic failure unless you have good reason to think it should have engaged those reasoning abilities in the test context. We know that LLMs selectively activate subnetworks based on the content of the prompt. There should be no expectation of any systematic reasoning abilities, but rather what are its capacities in ideal contexts. LLMs are way too sensitive to seemingly arbitrary features of context to rule out capacities from even seemingly systematic failures.

lukev 3y ago

To paraphrase what you're saying: an AI can't reason, because it is built to stochastically predicts tokens, which is not reasoning and which is different from the activity of reasoning. Which I agree with. But also, by observation, it can (at least some of the time) emit token sequences that emulate reasoning (at least on some simple tasks.) So perhaps it can reason in the same way a submarine can swim.

hackinthebochs 3y ago
>an AI can't reason, because it is built to stochastically predicts tokens, which is not reasoning and which is different from the activity of reasoning. A feature indispensable in the generation of a sequence is recovered in the limit of predicting the sequence. So prediction does not exclude higher level cognitive processes. Transformers being universal sequence2sequence modelers give reason to believe they can reach the limit.
- mjburgess 3y ago
  There are an infinite number of models which generate the same exact (infinite) sequence. No model is "recovered in the limit". And, more severely, we're blind to the future. So of the infinite models of all of history, we're not even interested in the ones which are maximally retrospectively predictive. Almost all of those are maximally non-predictive of the future, and indeed, much worse than the ones which fail to predict the past well. So your 'recovering in the limit' alas is a dangerous kind of pseudoscience: fitting the past. We want models which are wrong for the right reasons. Not models which are right for the wrong ones. The latter fail catastrophically. The models we require enable us to simulate unrealised futures; the causal, abductive models of science (, say).
  - hackinthebochs 3y ago
    >There are an infinite number of models which generate the same exact (infinite) sequence. Only if there is no constraint on the model. But the fixed number of parameters, and the inductive bias of attention limits the space of models that are learnable. In the limit of infinite data but finite capacity for memorization, the only available solution will be to recover the information dynamic of the generating process. >So of the infinite models of all of history, we're not even interested in the ones which are maximally retrospectively predictive. Presumably the same processes that operated in the past will operate in the future. So accurately modelling the past is certainly very informative for the future.

zby 3y ago

But there are programs that do inferences - and programs work on Turing machines which are just lookup tables (with an infinite tape). Lookup tables can do inferences.

AstralStorm 3y ago
Apparently an LLM fails at doing these inferences repeatedly, which makes it inferior to a 3SAT solver. Much less a general computer like a Turing machine.

rhtgrg 3y ago

> It's incomprehensible to me how gullible people are around AI today -- the Eliza effect coupled with a pseudoscientific impulse to "whatever appears to work". Incomprehensible perhaps, but not even a smidge unpredictable. You knew exactly what you would find in this comment thread.