4 ms·

We have no reason to believe that it is not reasoning. Since it looks like reasoning, the default position to be disproved is this is reasoning. I am willing t

We have no reason to believe that it is not reasoning. Since it looks like reasoning, the default position to be disproved is this is reasoning. I am willing to accept arguments that are not appeals to nature / human exceptionalism. I am even willing to accept a complete uncertainty over the whole situation since it is difficult to analyze. The silliest position, though, is a gnostic "no reasoning here" position.

mjr00 2y ago
> We have no reason to believe that it is not reasoning. We absolutely do: it's a computer, executing code, to predict tokens, based on a data set. Computers don't "reason" the same way they don't "do math". We know computers can't do math because, well, they can't sometimes[0]. > Since it looks like reasoning, the default position to be disproved is this is reasoning. Strongly disagree. Since it's a computer program, the default position to be disproved is that it's a computer program. Fundamentally these types of arguments are less about LLMs and more about whether you believe humans are mere next-token-prediction machines, which is a pointless debate because nothing is provable. [0] https://en.wikipedia.org/wiki/Pentium_FDIV_bug https://en.wikipedia.org/wiki/Pentium_FDIV_bug
Jensson 2y ago
> Since it looks like reasoning, the default position to be disproved is this is reasoning. Since we know it is a model that is trained to generate text that humans would generate, it writes down not its reasoning but what it thinks a human would write in that scenario. So it doesn't write its reasoning there, if it does reason its behind the words and not the words itself.
- ludwik 2y ago
  Sure, but we have clear evidence that generating this pseudo-reasoning text helps the model to make better decisions afterwards. Which means that it not only looks like reasoning but also effectively serves the same purpose. Additionally, the new "reasoning" models don't just train on human text - they also undergo a Reinforcement Learning training step, where they are trained to produce whatever kinds of "reasoning" text help them "reason" best (i.e., leading to correct decisions based on that reasoning). This further complicates things and makes it harder to say "this is one thing and one thing only".
semi-extrinsic 2y ago
> The silliest position, though, is a gnostic "no reasoning here" position. On the contrary - extraordinary claims require extraordinary evidence. That LLMs are performing a cognitive process similar to reasoning or intelligence is certainly an extraordinary claim, at least outside of VC hype circles. Making the model split its outputs into "answer" and "scratchpad", and then observing that these to parts are correlated, does not constitute extraordinary evidence.
- famouswaffles 2y ago
  >That LLMs are performing a cognitive process similar to reasoning or intelligence is certainly an extraordinary claim. It's not an extraordinary claim if the processes are achieving similar things under similar conditions. In fact, the extraordinary claim then becomes that it is not in fact reasoning or intelligent. Forces are required to move objects. If i saw something i thought was incapable of producing forces moving objects then the extraordinary claim starts being, "this thing cannot produce forces" not "this thing can move objects".
  - semi-extrinsic 2y ago
    Nope, that's not how it works. Correlation has never been proof of anything, despite how badly people want to believe so.
    - famouswaffles 2y ago
      It's not about correlation or proving anything. It's that something doing what you ascertained it never could changes what claims are and aren't extraordinary. You can handwave it away, i.e "the thing is moving objects by magic instead" but it's there and you can't keep acting like "this thing can produce forces" is still the extraordinary claim.
morsecodist 2y ago
The burden of proof is on the positive claim. Even if I were to make the claim that another human was reasoning I would need to provide justification for that claim. A lot of things look like something but that is not enough to shift the burden of proof. I don't even necessarily think we disagree on the conclusion. In my opinion, our notion of "reasoning" is so ill-defined this question is kind of meaningless. It is reasoning for some definitions of reasoning, it is not for others. I just don't think your shift of the burden of proof makes sense here.

mjr00 2y ago

> We have no reason to believe that it is not reasoning. We absolutely do: it's a computer, executing code, to predict tokens, based on a data set. Computers don't "reason" the same way they don't "do math". We know computers can't do math because, well, they can't sometimes[0]. > Since it looks like reasoning, the default position to be disproved is this is reasoning. Strongly disagree. Since it's a computer program, the default position to be disproved is that it's a computer program. Fundamentally these types of arguments are less about LLMs and more about whether you believe humans are mere next-token-prediction machines, which is a pointless debate because nothing is provable. [0] https://en.wikipedia.org/wiki/Pentium_FDIV_bug https://en.wikipedia.org/wiki/Pentium_FDIV_bug

Jensson 2y ago

> Since it looks like reasoning, the default position to be disproved is this is reasoning. Since we know it is a model that is trained to generate text that humans would generate, it writes down not its reasoning but what it thinks a human would write in that scenario. So it doesn't write its reasoning there, if it does reason its behind the words and not the words itself.

ludwik 2y ago
Sure, but we have clear evidence that generating this pseudo-reasoning text helps the model to make better decisions afterwards. Which means that it not only looks like reasoning but also effectively serves the same purpose. Additionally, the new "reasoning" models don't just train on human text - they also undergo a Reinforcement Learning training step, where they are trained to produce whatever kinds of "reasoning" text help them "reason" best (i.e., leading to correct decisions based on that reasoning). This further complicates things and makes it harder to say "this is one thing and one thing only".

semi-extrinsic 2y ago

> The silliest position, though, is a gnostic "no reasoning here" position. On the contrary - extraordinary claims require extraordinary evidence. That LLMs are performing a cognitive process similar to reasoning or intelligence is certainly an extraordinary claim, at least outside of VC hype circles. Making the model split its outputs into "answer" and "scratchpad", and then observing that these to parts are correlated, does not constitute extraordinary evidence.

famouswaffles 2y ago
>That LLMs are performing a cognitive process similar to reasoning or intelligence is certainly an extraordinary claim. It's not an extraordinary claim if the processes are achieving similar things under similar conditions. In fact, the extraordinary claim then becomes that it is not in fact reasoning or intelligent. Forces are required to move objects. If i saw something i thought was incapable of producing forces moving objects then the extraordinary claim starts being, "this thing cannot produce forces" not "this thing can move objects".
- semi-extrinsic 2y ago
  Nope, that's not how it works. Correlation has never been proof of anything, despite how badly people want to believe so.
  - famouswaffles 2y ago
    It's not about correlation or proving anything. It's that something doing what you ascertained it never could changes what claims are and aren't extraordinary. You can handwave it away, i.e "the thing is moving objects by magic instead" but it's there and you can't keep acting like "this thing can produce forces" is still the extraordinary claim.

morsecodist 2y ago

The burden of proof is on the positive claim. Even if I were to make the claim that another human was reasoning I would need to provide justification for that claim. A lot of things look like something but that is not enough to shift the burden of proof. I don't even necessarily think we disagree on the conclusion. In my opinion, our notion of "reasoning" is so ill-defined this question is kind of meaningless. It is reasoning for some definitions of reasoning, it is not for others. I just don't think your shift of the burden of proof makes sense here.