4 ms·

> Why would we expect LLMs to be capable of thought in the first place? "We" didn't have the expectation, but upon interacting with LLMs, we can see and feel a

> Why would we expect LLMs to be capable of thought in the first place? "We" didn't have the expectation, but upon interacting with LLMs, we can see and feel and hear that thought is taking place, and have realized that maybe the special place we held for human thought isn't so special as we had hoped. > The fact that these models have devoured the contents of the entire internet and still aren’t AGI is an indication that LLMs won’t develop a capacity for cognition. Why do you assume that training on the internet would lead to AGI?

AstroBen 1y ago
> upon interacting with LLMs, we can see and feel and hear that thought is taking place uh.. no we can't?
- nullstyle 1y ago
  Then you aren't paying attention, IMO. An example for the dense: A friend of mine used his conversations with ChatGPT to get through a tough spot in his marriage. The distinction between what happened in ChatGPT's internal processing and what would have happened in the internal processing of the brain of a family therapist is a meaningless distinction. My friend was able to benefit from the thoughtwork of ChatGPT.
  - AstroBen 1y ago
    Calling people dense isn't helping them engage with you
    - nullstyle 1y ago
      I was responding reciprocally to your attitude. Care to differentiate thought and thoughtwork for us?
- simonw 1y ago
  Screenshot for you: https://gist.github.com/simonw/5b051aa10e97ac86eeb539f51a828861?permalink_comment_id=5593669#gistcomment-5593669 https://gist.github.com/simonw/5b051aa10e97ac86eeb539f51a828... That looks like thinking. If you want to argue it's not thinking you are absolutely welcome to, but you need to say more than just "no we can't?".
  - 613style 1y ago
    The sharpest cries of, "that's not thinking!" always seem to have an air of desperation about them. It's as if I'm special, and I think, and maybe thinking is what makes me special, so if LLMs think then I'm less special. At some point the cry will change to, "It only looks like thinking from the outside!" And some time after that: "It only looks conscious from the outside..."
  - emp17344 1y ago
    This paper makes the claim that the “reasoning” output these models produce isn’t a necessary component of these models, and is likely unrelated to the actual final output: https://arxiv.org/abs/2505.13775 https://arxiv.org/abs/2505.13775 Just because these models are good at outputting text that appears to be a series of thoughts, doesn’t mean that LLMs are thinking.
  - AstroBen 1y ago
    I'm finding quite often from reading these streams in my own chats there's a lack of strong logical consistency from one chunk to the next where it should clearly be there. That combined with the output not being consistent with the 'thoughts'. I'll screenshot it next time I see a good example 'thinking' here needs a clearer definition. Even for us humans the language is just an after the fact attempt at describing what went on, or is going on inside our heads Consider: https://www.unsw.edu.au/newsroom/news/2019/03/our-brains-reveal-our-choices-before-were-even-aware-of-them--st https://www.unsw.edu.au/newsroom/news/2019/03/our-brains-rev...
  - ustad 1y ago
    I’ve got one of those old 1980s chess computers with Kasparov’s face on it. Before it makes a move, a little LED labeled ‘thinking’ blinks for a while. Looks like thinking, too.
emp17344 1y ago
People would’ve said the same thing about ELIZA. Turns out, people are inclined to anthropomorphize non-conscious objects or systems. This is especially true for chatbots, which are able to mimic language. Again, why should we believe that LLMs are capable of thought, other than the fact that they vaguely “seem” to communicate with conscious intent?
- nullstyle 1y ago
  > People would’ve said the same thing about ELIZA. I believe this to be the case because the appearance of thoughtfulness is a smooth function, not a line in the sand. Why are you bringing consciousness into this discussion? Consciousness isn't thoughtfulness. > Again, why should we believe that LLMs are capable of thought, other than the fact that they vaguely “seem” to communicate with conscious intent? I define "to think" as the act of applying context to a model. What do you define it as? Given my definition, llm inference seems plainly to be an act of thought.
- nullstyle 1y ago
  > Why do you assume that training on the internet would lead to AGI? Answer my question, please. edit: let me rephrase to be more clear with what i'm asking. Why are you assuming that consuming the entire internet would lead to either "AGI is here" or "AGI will never be here"? New techniques for developing better intelligence are emerging all the time.
  - whilenot-dev 1y ago
    ...because that's the story being told?[0] [0]: https://openai.com/index/planning-for-agi-and-beyond/ https://openai.com/index/planning-for-agi-and-beyond/
    - nullstyle 1y ago
      I think if you read the article linked and thought: “I can assume that AGI will be birthed by training on the entire Internet” then you didn’t read very carefully or judge the author well.

AstroBen 1y ago

> upon interacting with LLMs, we can see and feel and hear that thought is taking place uh.. no we can't?

nullstyle 1y ago
Then you aren't paying attention, IMO. An example for the dense: A friend of mine used his conversations with ChatGPT to get through a tough spot in his marriage. The distinction between what happened in ChatGPT's internal processing and what would have happened in the internal processing of the brain of a family therapist is a meaningless distinction. My friend was able to benefit from the thoughtwork of ChatGPT.
- AstroBen 1y ago
  Calling people dense isn't helping them engage with you
  - nullstyle 1y ago
    I was responding reciprocally to your attitude. Care to differentiate thought and thoughtwork for us?
simonw 1y ago
Screenshot for you: https://gist.github.com/simonw/5b051aa10e97ac86eeb539f51a828861?permalink_comment_id=5593669#gistcomment-5593669 https://gist.github.com/simonw/5b051aa10e97ac86eeb539f51a828... That looks like thinking. If you want to argue it's not thinking you are absolutely welcome to, but you need to say more than just "no we can't?".
- 613style 1y ago
  The sharpest cries of, "that's not thinking!" always seem to have an air of desperation about them. It's as if I'm special, and I think, and maybe thinking is what makes me special, so if LLMs think then I'm less special. At some point the cry will change to, "It only looks like thinking from the outside!" And some time after that: "It only looks conscious from the outside..."
- emp17344 1y ago
  This paper makes the claim that the “reasoning” output these models produce isn’t a necessary component of these models, and is likely unrelated to the actual final output: https://arxiv.org/abs/2505.13775 https://arxiv.org/abs/2505.13775 Just because these models are good at outputting text that appears to be a series of thoughts, doesn’t mean that LLMs are thinking.
- AstroBen 1y ago
  I'm finding quite often from reading these streams in my own chats there's a lack of strong logical consistency from one chunk to the next where it should clearly be there. That combined with the output not being consistent with the 'thoughts'. I'll screenshot it next time I see a good example 'thinking' here needs a clearer definition. Even for us humans the language is just an after the fact attempt at describing what went on, or is going on inside our heads Consider: https://www.unsw.edu.au/newsroom/news/2019/03/our-brains-reveal-our-choices-before-were-even-aware-of-them--st https://www.unsw.edu.au/newsroom/news/2019/03/our-brains-rev...
- ustad 1y ago
  I’ve got one of those old 1980s chess computers with Kasparov’s face on it. Before it makes a move, a little LED labeled ‘thinking’ blinks for a while. Looks like thinking, too.

emp17344 1y ago

People would’ve said the same thing about ELIZA. Turns out, people are inclined to anthropomorphize non-conscious objects or systems. This is especially true for chatbots, which are able to mimic language. Again, why should we believe that LLMs are capable of thought, other than the fact that they vaguely “seem” to communicate with conscious intent?

nullstyle 1y ago
> People would’ve said the same thing about ELIZA. I believe this to be the case because the appearance of thoughtfulness is a smooth function, not a line in the sand. Why are you bringing consciousness into this discussion? Consciousness isn't thoughtfulness. > Again, why should we believe that LLMs are capable of thought, other than the fact that they vaguely “seem” to communicate with conscious intent? I define "to think" as the act of applying context to a model. What do you define it as? Given my definition, llm inference seems plainly to be an act of thought.
nullstyle 1y ago
> Why do you assume that training on the internet would lead to AGI? Answer my question, please. edit: let me rephrase to be more clear with what i'm asking. Why are you assuming that consuming the entire internet would lead to either "AGI is here" or "AGI will never be here"? New techniques for developing better intelligence are emerging all the time.
- whilenot-dev 1y ago
  ...because that's the story being told?[0] [0]: https://openai.com/index/planning-for-agi-and-beyond/ https://openai.com/index/planning-for-agi-and-beyond/
  - nullstyle 1y ago
    I think if you read the article linked and thought: “I can assume that AGI will be birthed by training on the entire Internet” then you didn’t read very carefully or judge the author well.