5 ms·

No one can provide a technical definition of what it means to think. That doesn’t mean LLMs are thinking. This style of argument, which often boils down to “you

No one can provide a technical definition of what it means to think. That doesn’t mean LLMs are thinking. This style of argument, which often boils down to “you can’t solidly define the thought process, therefore LLMs can think!” is a fallacy. Why would we expect LLMs to be capable of thought in the first place? The fact that these models have devoured the contents of the entire internet and still aren’t AGI is an indication that LLMs won’t develop a capacity for cognition.

nullstyle 1y ago
> Why would we expect LLMs to be capable of thought in the first place? "We" didn't have the expectation, but upon interacting with LLMs, we can see and feel and hear that thought is taking place, and have realized that maybe the special place we held for human thought isn't so special as we had hoped. > The fact that these models have devoured the contents of the entire internet and still aren’t AGI is an indication that LLMs won’t develop a capacity for cognition. Why do you assume that training on the internet would lead to AGI?
- AstroBen 1y ago
  > upon interacting with LLMs, we can see and feel and hear that thought is taking place uh.. no we can't?
  - nullstyle 1y ago
    Then you aren't paying attention, IMO. An example for the dense: A friend of mine used his conversations with ChatGPT to get through a tough spot in his marriage. The distinction between what happened in ChatGPT's internal processing and what would have happened in the internal processing of the brain of a family therapist is a meaningless distinction. My friend was able to benefit from the thoughtwork of ChatGPT.
    - AstroBen 1y ago
      Calling people dense isn't helping them engage with you
      - nullstyle 1y ago
        I was responding reciprocally to your attitude. Care to differentiate thought and thoughtwork for us?
  - simonw 1y ago
    Screenshot for you: https://gist.github.com/simonw/5b051aa10e97ac86eeb539f51a828861?permalink_comment_id=5593669#gistcomment-5593669 https://gist.github.com/simonw/5b051aa10e97ac86eeb539f51a828... That looks like thinking. If you want to argue it's not thinking you are absolutely welcome to, but you need to say more than just "no we can't?".
    - 613style 1y ago
      The sharpest cries of, "that's not thinking!" always seem to have an air of desperation about them. It's as if I'm special, and I think, and maybe thinking is what makes me special, so if LLMs think then I'm less special. At some point the cry will change to, "It only looks like thinking from the outside!" And some time after that: "It only looks conscious from the outside..."
    - emp17344 1y ago
      This paper makes the claim that the “reasoning” output these models produce isn’t a necessary component of these models, and is likely unrelated to the actual final output: https://arxiv.org/abs/2505.13775 https://arxiv.org/abs/2505.13775 Just because these models are good at outputting text that appears to be a series of thoughts, doesn’t mean that LLMs are thinking.
    - AstroBen 1y ago
      I'm finding quite often from reading these streams in my own chats there's a lack of strong logical consistency from one chunk to the next where it should clearly be there. That combined with the output not being consistent with the 'thoughts'. I'll screenshot it next time I see a good example 'thinking' here needs a clearer definition. Even for us humans the language is just an after the fact attempt at describing what went on, or is going on inside our heads Consider: https://www.unsw.edu.au/newsroom/news/2019/03/our-brains-reveal-our-choices-before-were-even-aware-of-them--st https://www.unsw.edu.au/newsroom/news/2019/03/our-brains-rev...
    - ustad 1y ago
      I’ve got one of those old 1980s chess computers with Kasparov’s face on it. Before it makes a move, a little LED labeled ‘thinking’ blinks for a while. Looks like thinking, too.
- emp17344 1y ago
  People would’ve said the same thing about ELIZA. Turns out, people are inclined to anthropomorphize non-conscious objects or systems. This is especially true for chatbots, which are able to mimic language. Again, why should we believe that LLMs are capable of thought, other than the fact that they vaguely “seem” to communicate with conscious intent?
  - nullstyle 1y ago
    > People would’ve said the same thing about ELIZA. I believe this to be the case because the appearance of thoughtfulness is a smooth function, not a line in the sand. Why are you bringing consciousness into this discussion? Consciousness isn't thoughtfulness. > Again, why should we believe that LLMs are capable of thought, other than the fact that they vaguely “seem” to communicate with conscious intent? I define "to think" as the act of applying context to a model. What do you define it as? Given my definition, llm inference seems plainly to be an act of thought.
  - nullstyle 1y ago
    > Why do you assume that training on the internet would lead to AGI? Answer my question, please. edit: let me rephrase to be more clear with what i'm asking. Why are you assuming that consuming the entire internet would lead to either "AGI is here" or "AGI will never be here"? New techniques for developing better intelligence are emerging all the time.
    - whilenot-dev 1y ago
      ...because that's the story being told?[0] [0]: https://openai.com/index/planning-for-agi-and-beyond/ https://openai.com/index/planning-for-agi-and-beyond/
      - nullstyle 1y ago
        I think if you read the article linked and thought: “I can assume that AGI will be birthed by training on the entire Internet” then you didn’t read very carefully or judge the author well.
inquirerGeneral 1y ago
[dead]
WhitneyLand 1y ago
That’s not a “style” of argument, it’s basic logic. To the extent you can’t define something the strength of your claim that it does not exist is correspondingly weakened.
- emp17344 1y ago
  Similarly, if you are unable to define something, your claim that a computer program is doing that thing is greatly weakened. If anything, the burden of proof is on you.
- keybored 1y ago
  Define God, the ineffable and all-powerful. Awaiting your effable answer.
- whilenot-dev 1y ago
  That’s certainly a style of argument and contains (with "basic logic") an informal fallacy[0]. To quote N. Chomsky when he made an analogy: "When you have a theory, there are two questions you'd need to ask: (1) Why are things this way? (2) Why are things not that way?"[1] I get it, this post-truth century is difficult to navigate. Engineering achievements get picked up through a mix of technical excitement and venture boredom, its fallacies are stylized by growth hackers as this unsolvable paradox that seems to fit just the right terminology for any marketing purposes, only to be piped through one hype cycle that just paints another picture of the doomsday of capitalism with black and white colors. /rant [0]: https://en.wikipedia.org/wiki/Argument_from_ignorance https://en.wikipedia.org/wiki/Argument_from_ignorance [1]: https://www.youtube.com/watch?v=axuGfh4UR9Q&t=9438s https://www.youtube.com/watch?v=axuGfh4UR9Q&t=9438s
simonw 1y ago
My argument is more "if you can't solidly define the thought process, you can't definitively state that LLMs can NOT think". That doesn't mean I think they can think myself. I just get frustrated at the quality of discussion every time this topic comes up.
- keybored 1y ago
  If you can’t solidly define your love for your wife/husband, you can’t definitely state that you couldn’t hypothetically love this bonobo that I have in my lab on the same level. If you zoom out enough they are virtually indistinguishable. > That doesn't mean I think they can think myself. I just get frustrated at the quality of discussion every time this topic comes up. You’re arguing for the potential of a ghost in a machine based on people not being able to answer hard philosophical questions behind human behavior and properties.[1] That’s a kind of Vulgar Theism quality of argumentation. [1] Not exclusively human. I’m sure killer whales can think.
- whilenot-dev 1y ago
  Isn't the thought process currently understood as a blend of deterministic & probabilistic decision-making and free will? So if LLMs lack the "free will" part, isn't that enough deductive reasoning to satisfy the argument "LLMs don't think"?

nullstyle 1y ago

> Why would we expect LLMs to be capable of thought in the first place? "We" didn't have the expectation, but upon interacting with LLMs, we can see and feel and hear that thought is taking place, and have realized that maybe the special place we held for human thought isn't so special as we had hoped. > The fact that these models have devoured the contents of the entire internet and still aren’t AGI is an indication that LLMs won’t develop a capacity for cognition. Why do you assume that training on the internet would lead to AGI?

AstroBen 1y ago
> upon interacting with LLMs, we can see and feel and hear that thought is taking place uh.. no we can't?
- nullstyle 1y ago
  Then you aren't paying attention, IMO. An example for the dense: A friend of mine used his conversations with ChatGPT to get through a tough spot in his marriage. The distinction between what happened in ChatGPT's internal processing and what would have happened in the internal processing of the brain of a family therapist is a meaningless distinction. My friend was able to benefit from the thoughtwork of ChatGPT.
  - AstroBen 1y ago
    Calling people dense isn't helping them engage with you
    - nullstyle 1y ago
      I was responding reciprocally to your attitude. Care to differentiate thought and thoughtwork for us?
- simonw 1y ago
  Screenshot for you: https://gist.github.com/simonw/5b051aa10e97ac86eeb539f51a828861?permalink_comment_id=5593669#gistcomment-5593669 https://gist.github.com/simonw/5b051aa10e97ac86eeb539f51a828... That looks like thinking. If you want to argue it's not thinking you are absolutely welcome to, but you need to say more than just "no we can't?".
  - 613style 1y ago
    The sharpest cries of, "that's not thinking!" always seem to have an air of desperation about them. It's as if I'm special, and I think, and maybe thinking is what makes me special, so if LLMs think then I'm less special. At some point the cry will change to, "It only looks like thinking from the outside!" And some time after that: "It only looks conscious from the outside..."
  - emp17344 1y ago
    This paper makes the claim that the “reasoning” output these models produce isn’t a necessary component of these models, and is likely unrelated to the actual final output: https://arxiv.org/abs/2505.13775 https://arxiv.org/abs/2505.13775 Just because these models are good at outputting text that appears to be a series of thoughts, doesn’t mean that LLMs are thinking.
  - AstroBen 1y ago
    I'm finding quite often from reading these streams in my own chats there's a lack of strong logical consistency from one chunk to the next where it should clearly be there. That combined with the output not being consistent with the 'thoughts'. I'll screenshot it next time I see a good example 'thinking' here needs a clearer definition. Even for us humans the language is just an after the fact attempt at describing what went on, or is going on inside our heads Consider: https://www.unsw.edu.au/newsroom/news/2019/03/our-brains-reveal-our-choices-before-were-even-aware-of-them--st https://www.unsw.edu.au/newsroom/news/2019/03/our-brains-rev...
  - ustad 1y ago
    I’ve got one of those old 1980s chess computers with Kasparov’s face on it. Before it makes a move, a little LED labeled ‘thinking’ blinks for a while. Looks like thinking, too.
emp17344 1y ago
People would’ve said the same thing about ELIZA. Turns out, people are inclined to anthropomorphize non-conscious objects or systems. This is especially true for chatbots, which are able to mimic language. Again, why should we believe that LLMs are capable of thought, other than the fact that they vaguely “seem” to communicate with conscious intent?
- nullstyle 1y ago
  > People would’ve said the same thing about ELIZA. I believe this to be the case because the appearance of thoughtfulness is a smooth function, not a line in the sand. Why are you bringing consciousness into this discussion? Consciousness isn't thoughtfulness. > Again, why should we believe that LLMs are capable of thought, other than the fact that they vaguely “seem” to communicate with conscious intent? I define "to think" as the act of applying context to a model. What do you define it as? Given my definition, llm inference seems plainly to be an act of thought.
- nullstyle 1y ago
  > Why do you assume that training on the internet would lead to AGI? Answer my question, please. edit: let me rephrase to be more clear with what i'm asking. Why are you assuming that consuming the entire internet would lead to either "AGI is here" or "AGI will never be here"? New techniques for developing better intelligence are emerging all the time.
  - whilenot-dev 1y ago
    ...because that's the story being told?[0] [0]: https://openai.com/index/planning-for-agi-and-beyond/ https://openai.com/index/planning-for-agi-and-beyond/
    - nullstyle 1y ago
      I think if you read the article linked and thought: “I can assume that AGI will be birthed by training on the entire Internet” then you didn’t read very carefully or judge the author well.

inquirerGeneral 1y ago

[dead]

WhitneyLand 1y ago

That’s not a “style” of argument, it’s basic logic. To the extent you can’t define something the strength of your claim that it does not exist is correspondingly weakened.

emp17344 1y ago
Similarly, if you are unable to define something, your claim that a computer program is doing that thing is greatly weakened. If anything, the burden of proof is on you.
keybored 1y ago
Define God, the ineffable and all-powerful. Awaiting your effable answer.
whilenot-dev 1y ago
That’s certainly a style of argument and contains (with "basic logic") an informal fallacy[0]. To quote N. Chomsky when he made an analogy: "When you have a theory, there are two questions you'd need to ask: (1) Why are things this way? (2) Why are things not that way?"[1] I get it, this post-truth century is difficult to navigate. Engineering achievements get picked up through a mix of technical excitement and venture boredom, its fallacies are stylized by growth hackers as this unsolvable paradox that seems to fit just the right terminology for any marketing purposes, only to be piped through one hype cycle that just paints another picture of the doomsday of capitalism with black and white colors. /rant [0]: https://en.wikipedia.org/wiki/Argument_from_ignorance https://en.wikipedia.org/wiki/Argument_from_ignorance [1]: https://www.youtube.com/watch?v=axuGfh4UR9Q&t=9438s https://www.youtube.com/watch?v=axuGfh4UR9Q&t=9438s

simonw 1y ago

My argument is more "if you can't solidly define the thought process, you can't definitively state that LLMs can NOT think". That doesn't mean I think they can think myself. I just get frustrated at the quality of discussion every time this topic comes up.

keybored 1y ago
If you can’t solidly define your love for your wife/husband, you can’t definitely state that you couldn’t hypothetically love this bonobo that I have in my lab on the same level. If you zoom out enough they are virtually indistinguishable. > That doesn't mean I think they can think myself. I just get frustrated at the quality of discussion every time this topic comes up. You’re arguing for the potential of a ghost in a machine based on people not being able to answer hard philosophical questions behind human behavior and properties.[1] That’s a kind of Vulgar Theism quality of argumentation. [1] Not exclusively human. I’m sure killer whales can think.
whilenot-dev 1y ago
Isn't the thought process currently understood as a blend of deterministic & probabilistic decision-making and free will? So if LLMs lack the "free will" part, isn't that enough deductive reasoning to satisfy the argument "LLMs don't think"?