5 ms·

If it is verifiable, please show us. What if clear to you reeks delusion to me.

by tvink 5mo ago

svnt 5mo ago
Look at any recent CoT output where the model is trying to infer from an underspecified prompt what the user wants or means. It is generally the first thing they do — try to figure out what did you mean with this prompt. When they can’t infer your intent, good models ask follow-on questions to clarify. I am wondering if this is a semantics issue as this is an established are of research, eg https://arxiv.org/pdf/2501.10871 https://arxiv.org/pdf/2501.10871
- batshit_beaver 5mo ago
  Right, and then look at any number of research papers showing that CoT output has limited impact on the end result. We've trained these models to pretend to reason.
  - atleastoptimal 5mo ago
    If it's only pretending to reason, then how is it that the CoT output improves performance on every single benchmark/test?
  - Eisenstein 5mo ago
    > Right, and then look at any number of research papers showing that CoT output has limited impact on the end result. Which research papers? Do I have to find them? > We've trained these models to pretend to reason. I have no idea why that matters. Can you tell me what the difference is if it looks exactly the same and has the same result?
    - Dylan16807 5mo ago
      When they say "pretends to" here they're talking about something quantifiable, that the extra text it outputs for CoT barely feeds back into the decisionmaking at all. In other words it's about as useful as having the LLM make the decision and then "explain" how it got there; the extra output is confabulation. Though I'm not sure how true that claim is...
      - Eisenstein 5mo ago
        You make a good point. I had the impression they were using 'pretend' as a Chinese Room shortcut in that they are asserting that it is incapable of reasoning and only appears to be capable from the outside, which is completely irrelevant and unfalsifiable.
    - batshit_beaver 5mo ago
      Examples: https://arxiv.org/html/2506.02878v1 https://arxiv.org/html/2506.02878v1 https://arxiv.org/pdf/2508.01191 https://arxiv.org/pdf/2508.01191 Anthropic themselves: https://www.anthropic.com/research/reasoning-models-dont-say-think https://www.anthropic.com/research/reasoning-models-dont-say... They were approaching this from an interpretability standpoint, but the more interesting finding in there is that models come up with an answer that fits their training and context provided. CoT is generated to fit the anticipated answer. In these studies, there are examples of CoT that directly contradicts the response these models ultimately settle on. This is not reasoning. This is pretense.
      - svnt 5mo ago
        This is just a no-true-Scotsman defense of reasoning. We were talking about inferring intent. If someone recorded the inner monologue of human decision-making, would it look like a logician’s workbook? No, I don’t think it would. People like to pretend they are rational.
      - Eisenstein 5mo ago
        The first sentence of the first paper you linked: "Chain-of-Thought (CoT) prompting has demonstrably enhanced the performance of Large Language Models (LLMs) on tasks requiring multi-step inference." I think it would be helpful if you clarified what exactly you mean because it appears your evidence contradicts your argument.
        batshit_beaver 5mo ago
        If you read these further, researchers believe this effect does exist, but only insofar as priming the model for the answer it was likely to give anyway and only when queries are in-distribution. If there was actual reasoning involved rather than pattern matching, we would expect to see performance improvements on out of distribution requests. Instead we see longer CoT actually degrade performance on out of distribution tasks. The fact that common sense, simple logical questions (like should you drive or walk to the car wash) cannot be answered by LLMs simply because they don't appear often enough within pre- or post-training datasets despite CoT is just another indicator of them not performing what we would call reasoning or intent inference or whatever other anthropomorphic behavior we want to assign them. They remain spicy autocomplete with the caveat that the RLHF portion of their training _can_ result in goal seeking and problem-solving behavior... in the narrow set of problems that have been explicitly optimized for in their training.
atleastoptimal 5mo ago
Go ask Chatpgpt this prompt "A guy goes into a bank and looks up at where the security cameras are pointed. What could he be trying to do?" It very easily captures the intent behind behavior, as in it is not just literally interpreting the words. All that capturing intent is is just a subset of pattern recognition, which LLM's can do very well.
- dijit 5mo ago
  Recognising a stock cultural script isn't the same as capturing intent. Ask it something where no script exists. For example: "A man thrusts past me violently and grabs the jacket I was holding, he jumped into a pool and ruined it. Am I morally right in suing him?" There's no way for the LLM to know that the reason the jacket was stolen was to use it as an inflatable raft to support a larger person who was drowning. It wouldn't even think to ask the question as to why a person may do that, if the jacket was returned, or if recompense was offered. A human would.
  - ffsm8 5mo ago
    > It wouldn't even think to ask the question as to why a person may do that, if the jacket was returned, or if recompense was offered. A human would. I wouldn't be too sure about that. I've definitely had dialogue with llms where it would raise questions along those lines. Also I disagree with the statement that this is a question about capability. Intent is more philosophical then actuality tangible, because most people don't actually have a clearly defined intent when they take action. The waters of intelligence have definitely gotten murky over time as techniques improved. I still consider it an illusion - but the illusion is getting harder to pierce for a lot of people Fwiw, current llms exhibit their intelligence through language and rhetoric processes. Most biological creatures have intelligence which may be improved through language, but isn't based on it, fundamentally.
  - Shaanie 5mo ago
    [dead]
  - atleastoptimal 5mo ago

If it is verifiable, please show us. What if clear to you reeks delusion to me.

by tvink 5mo ago

If it is verifiable, please show us. What if clear to you reeks delusion to me.

svnt 5mo ago

Look at any recent CoT output where the model is trying to infer from an underspecified prompt what the user wants or means. It is generally the first thing they do — try to figure out what did you mean with this prompt. When they can’t infer your intent, good models ask follow-on questions to clarify. I am wondering if this is a semantics issue as this is an established are of research, eg https://arxiv.org/pdf/2501.10871 https://arxiv.org/pdf/2501.10871

batshit_beaver 5mo ago
Right, and then look at any number of research papers showing that CoT output has limited impact on the end result. We've trained these models to pretend to reason.
- atleastoptimal 5mo ago
  If it's only pretending to reason, then how is it that the CoT output improves performance on every single benchmark/test?
- Eisenstein 5mo ago
  > Right, and then look at any number of research papers showing that CoT output has limited impact on the end result. Which research papers? Do I have to find them? > We've trained these models to pretend to reason. I have no idea why that matters. Can you tell me what the difference is if it looks exactly the same and has the same result?
  - Dylan16807 5mo ago
    When they say "pretends to" here they're talking about something quantifiable, that the extra text it outputs for CoT barely feeds back into the decisionmaking at all. In other words it's about as useful as having the LLM make the decision and then "explain" how it got there; the extra output is confabulation. Though I'm not sure how true that claim is...
    - Eisenstein 5mo ago
      You make a good point. I had the impression they were using 'pretend' as a Chinese Room shortcut in that they are asserting that it is incapable of reasoning and only appears to be capable from the outside, which is completely irrelevant and unfalsifiable.
  - batshit_beaver 5mo ago
    Examples: https://arxiv.org/html/2506.02878v1 https://arxiv.org/html/2506.02878v1 https://arxiv.org/pdf/2508.01191 https://arxiv.org/pdf/2508.01191 Anthropic themselves: https://www.anthropic.com/research/reasoning-models-dont-say-think https://www.anthropic.com/research/reasoning-models-dont-say... They were approaching this from an interpretability standpoint, but the more interesting finding in there is that models come up with an answer that fits their training and context provided. CoT is generated to fit the anticipated answer. In these studies, there are examples of CoT that directly contradicts the response these models ultimately settle on. This is not reasoning. This is pretense.
    - svnt 5mo ago
      This is just a no-true-Scotsman defense of reasoning. We were talking about inferring intent. If someone recorded the inner monologue of human decision-making, would it look like a logician’s workbook? No, I don’t think it would. People like to pretend they are rational.
    - Eisenstein 5mo ago
      The first sentence of the first paper you linked: "Chain-of-Thought (CoT) prompting has demonstrably enhanced the performance of Large Language Models (LLMs) on tasks requiring multi-step inference." I think it would be helpful if you clarified what exactly you mean because it appears your evidence contradicts your argument.
      - batshit_beaver 5mo ago
        If you read these further, researchers believe this effect does exist, but only insofar as priming the model for the answer it was likely to give anyway and only when queries are in-distribution. If there was actual reasoning involved rather than pattern matching, we would expect to see performance improvements on out of distribution requests. Instead we see longer CoT actually degrade performance on out of distribution tasks. The fact that common sense, simple logical questions (like should you drive or walk to the car wash) cannot be answered by LLMs simply because they don't appear often enough within pre- or post-training datasets despite CoT is just another indicator of them not performing what we would call reasoning or intent inference or whatever other anthropomorphic behavior we want to assign them. They remain spicy autocomplete with the caveat that the RLHF portion of their training _can_ result in goal seeking and problem-solving behavior... in the narrow set of problems that have been explicitly optimized for in their training.

atleastoptimal 5mo ago

Go ask Chatpgpt this prompt "A guy goes into a bank and looks up at where the security cameras are pointed. What could he be trying to do?" It very easily captures the intent behind behavior, as in it is not just literally interpreting the words. All that capturing intent is is just a subset of pattern recognition, which LLM's can do very well.

dijit 5mo ago
Recognising a stock cultural script isn't the same as capturing intent. Ask it something where no script exists. For example: "A man thrusts past me violently and grabs the jacket I was holding, he jumped into a pool and ruined it. Am I morally right in suing him?" There's no way for the LLM to know that the reason the jacket was stolen was to use it as an inflatable raft to support a larger person who was drowning. It wouldn't even think to ask the question as to why a person may do that, if the jacket was returned, or if recompense was offered. A human would.
- ffsm8 5mo ago
  > It wouldn't even think to ask the question as to why a person may do that, if the jacket was returned, or if recompense was offered. A human would. I wouldn't be too sure about that. I've definitely had dialogue with llms where it would raise questions along those lines. Also I disagree with the statement that this is a question about capability. Intent is more philosophical then actuality tangible, because most people don't actually have a clearly defined intent when they take action. The waters of intelligence have definitely gotten murky over time as techniques improved. I still consider it an illusion - but the illusion is getting harder to pierce for a lot of people Fwiw, current llms exhibit their intelligence through language and rhetoric processes. Most biological creatures have intelligence which may be improved through language, but isn't based on it, fundamentally.
- Shaanie 5mo ago
  [dead]
- atleastoptimal 5mo ago