4 ms·

> a window into their reasoning Reasoning? Or just more generative text?

animal-husband 2y ago
Text generated prior to a decision to “explain” it is reasoning for the relevant intents and purposes. Text generated after a decision to “explain” it is largely nonsense.
- Kinrany 2y ago
  The true test would be seeing the behavior change depending on the presence of reasoning
  - 2099miles 2y ago
    The words thinking and reasoning used here are imprecise. It’s just generating text like always. If the text is after “ai-thoughts:” then it’s “thinking” and if it’s after “ai-response” then it’s “responding” not “thinking” but it is always a big ole model choosing the most likely next token potentially with some random sampling
  - animal-husband 2y ago
    That is what was observed - o1 family models performed the “cheat”, non-reasoning models didn’t.
deleted 2y ago
[deleted]
thornewolf 2y ago
We have no reason to believe that it is not reasoning. Since it looks like reasoning, the default position to be disproved is this is reasoning. I am willing to accept arguments that are not appeals to nature / human exceptionalism. I am even willing to accept a complete uncertainty over the whole situation since it is difficult to analyze. The silliest position, though, is a gnostic "no reasoning here" position.
- mjr00 2y ago
  > We have no reason to believe that it is not reasoning. We absolutely do: it's a computer, executing code, to predict tokens, based on a data set. Computers don't "reason" the same way they don't "do math". We know computers can't do math because, well, they can't sometimes[0]. > Since it looks like reasoning, the default position to be disproved is this is reasoning. Strongly disagree. Since it's a computer program, the default position to be disproved is that it's a computer program. Fundamentally these types of arguments are less about LLMs and more about whether you believe humans are mere next-token-prediction machines, which is a pointless debate because nothing is provable. [0] https://en.wikipedia.org/wiki/Pentium_FDIV_bug https://en.wikipedia.org/wiki/Pentium_FDIV_bug
- Jensson 2y ago
  > Since it looks like reasoning, the default position to be disproved is this is reasoning. Since we know it is a model that is trained to generate text that humans would generate, it writes down not its reasoning but what it thinks a human would write in that scenario. So it doesn't write its reasoning there, if it does reason its behind the words and not the words itself.
  - ludwik 2y ago
    Sure, but we have clear evidence that generating this pseudo-reasoning text helps the model to make better decisions afterwards. Which means that it not only looks like reasoning but also effectively serves the same purpose. Additionally, the new "reasoning" models don't just train on human text - they also undergo a Reinforcement Learning training step, where they are trained to produce whatever kinds of "reasoning" text help them "reason" best (i.e., leading to correct decisions based on that reasoning). This further complicates things and makes it harder to say "this is one thing and one thing only".
- semi-extrinsic 2y ago
  > The silliest position, though, is a gnostic "no reasoning here" position. On the contrary - extraordinary claims require extraordinary evidence. That LLMs are performing a cognitive process similar to reasoning or intelligence is certainly an extraordinary claim, at least outside of VC hype circles. Making the model split its outputs into "answer" and "scratchpad", and then observing that these to parts are correlated, does not constitute extraordinary evidence.
blah2244 2y ago
How do you differentiate between the two? Each token the model outputs requires it to evaluate all of the context it already has (query + existing output). By allowing it more tokens to "reason", you're allowing it to evaluate the context many times over, similar to how a person might turn a problem over in their heads before coming up with an answer. Given the performance of reasoning models on complex tasks, I'm of the opinion that the "more tokens with reasoning prompting" approach is at least a decent model of the process that humans would go through to "reason".
Terr_ 2y ago
IMO it's just more generated text, like a film noir detective's unvoiced monologue. It keeps the story from wandering, but it's not a qualitative difference in how text is being brought together to create the illusion of a fictional mind.

Text generated prior to a decision to “explain” it is reasoning for the relevant intents and purposes. Text generated after a decision to “explain” it is largely nonsense.

Kinrany 2y ago
The true test would be seeing the behavior change depending on the presence of reasoning
- 2099miles 2y ago
  The words thinking and reasoning used here are imprecise. It’s just generating text like always. If the text is after “ai-thoughts:” then it’s “thinking” and if it’s after “ai-response” then it’s “responding” not “thinking” but it is always a big ole model choosing the most likely next token potentially with some random sampling
- animal-husband 2y ago
  That is what was observed - o1 family models performed the “cheat”, non-reasoning models didn’t.

deleted 2y ago

[deleted]

thornewolf 2y ago

We have no reason to believe that it is not reasoning. Since it looks like reasoning, the default position to be disproved is this is reasoning. I am willing to accept arguments that are not appeals to nature / human exceptionalism. I am even willing to accept a complete uncertainty over the whole situation since it is difficult to analyze. The silliest position, though, is a gnostic "no reasoning here" position.

mjr00 2y ago
> We have no reason to believe that it is not reasoning. We absolutely do: it's a computer, executing code, to predict tokens, based on a data set. Computers don't "reason" the same way they don't "do math". We know computers can't do math because, well, they can't sometimes[0]. > Since it looks like reasoning, the default position to be disproved is this is reasoning. Strongly disagree. Since it's a computer program, the default position to be disproved is that it's a computer program. Fundamentally these types of arguments are less about LLMs and more about whether you believe humans are mere next-token-prediction machines, which is a pointless debate because nothing is provable. [0] https://en.wikipedia.org/wiki/Pentium_FDIV_bug https://en.wikipedia.org/wiki/Pentium_FDIV_bug
Jensson 2y ago
> Since it looks like reasoning, the default position to be disproved is this is reasoning. Since we know it is a model that is trained to generate text that humans would generate, it writes down not its reasoning but what it thinks a human would write in that scenario. So it doesn't write its reasoning there, if it does reason its behind the words and not the words itself.
- ludwik 2y ago
  Sure, but we have clear evidence that generating this pseudo-reasoning text helps the model to make better decisions afterwards. Which means that it not only looks like reasoning but also effectively serves the same purpose. Additionally, the new "reasoning" models don't just train on human text - they also undergo a Reinforcement Learning training step, where they are trained to produce whatever kinds of "reasoning" text help them "reason" best (i.e., leading to correct decisions based on that reasoning). This further complicates things and makes it harder to say "this is one thing and one thing only".
semi-extrinsic 2y ago
> The silliest position, though, is a gnostic "no reasoning here" position. On the contrary - extraordinary claims require extraordinary evidence. That LLMs are performing a cognitive process similar to reasoning or intelligence is certainly an extraordinary claim, at least outside of VC hype circles. Making the model split its outputs into "answer" and "scratchpad", and then observing that these to parts are correlated, does not constitute extraordinary evidence.

blah2244 2y ago

How do you differentiate between the two? Each token the model outputs requires it to evaluate all of the context it already has (query + existing output). By allowing it more tokens to "reason", you're allowing it to evaluate the context many times over, similar to how a person might turn a problem over in their heads before coming up with an answer. Given the performance of reasoning models on complex tasks, I'm of the opinion that the "more tokens with reasoning prompting" approach is at least a decent model of the process that humans would go through to "reason".

Terr_ 2y ago

IMO it's just more generated text, like a film noir detective's unvoiced monologue. It keeps the story from wandering, but it's not a qualitative difference in how text is being brought together to create the illusion of a fictional mind.