6 ms·

Models can't think. It uses the input context to predict an output. So if you have a problem that needs to be solved iteratively, those intermediate steps need

Models can't think. It uses the input context to predict an output. So if you have a problem that needs to be solved iteratively, those intermediate steps need to be persisted to the context, because there is nowhere for them to go otherwise.

stavros 3y ago
> Models can't think. It uses the input context to predict an output. The first claim doesn't follow from the second. What is it about using the input to predict an output that makes you believe they can't think? What if that's all thinking is? We don't know.
- deleted 3y ago
  [deleted]
- phailhaus 3y ago
  It's not the the second statement follows from the first; I'm asserting that models can't think, and that what they're really doing is prediction based on context. The fact that chain-of-thought reasoning yields significantly better results is your hint: that means that the model doesn't think like a human does when it comes up with responses. If it's not in the context, it doesn't exist. You can't ask a model "why did you answer that way" without it generating from whole cloth a plausible retroactive reason. But there is no memory, so it can't really tell you. > What if that's all thinking is? I think that this is roughly true. But we actually have memory outside of what we say, so when we think, all those intermediate steps are persisted in our brains. For a model, the context is the memory. If you delete your question from the context and ask it "why did you answer that way", it will have no idea what you're talking about.
  - stavros 3y ago
    > You can't ask a model "why did you answer that way" without it generating from whole cloth a plausible retroactive reason. I've caught myself generating from whole cloth a plausible retroactive reason for some of my actions, which I later realized wasn't true at all. Does that mean I can't think either? Is there a way for an external observer to tell if someone is thinking or not, or is it something that, axiomatically, only humans can do, and nothing else?
    - phailhaus 3y ago
      Again, you have memory but models don't. If it's not in the context, it doesn't exist. This is really easy to prove: delete your question from the context and ask the model why it answered the way it did. It will not know, because the model prediction service is stateless. It only takes the context as input. It's like if you and I weren't able to remember anything that wasn't written down. That's why you can get models to approximate thought by telling it to write down its intermediate steps.
      - stavros 3y ago
        Right, exactly, the context is the model's memory. All you're saying is "except for their memory, they have no memory", which goes for humans as well.
        phailhaus 3y ago
        If you say that the models have memory and can think, you are heavily implying that there is a brain in the cloud that is capable of "thinking" or "reasoning" through your request. That's what it means for humans: we're self-contained. Models aren't like people. The context is not "part" of the model, the context is given to the model when you ask it for a prediction. It's like you cut a small part of a person's brain out. Alone, it can't think. It's a pedantic distinction, but an important one. The model itself isn't capable of thinking, but if you package it up with a context that it can manipulate at-will, that combination of parts can be said to "think". That's probably going to be the next step in LLM tech.
        stavros 3y ago
        By that definition, ChatGPT can already think.
        water-your-self 3y ago
        If this is your conclusion, I would recommend reading more of the research.
- andoando 3y ago
  I think a fundamental difference is we are able to learn new things by reasoning through our existing knowledge. Moreover our beliefs are mostly consistent with each other, and we can be argued with and have our beliefs changed. As far as I understand, GPT isn't going to alter its whole worldview if you show it that its thinking is flawed. But perhaps this is possible if upon discovering a flaw, it looped through its corpus and altered its connections?

stavros 3y ago

> Models can't think. It uses the input context to predict an output. The first claim doesn't follow from the second. What is it about using the input to predict an output that makes you believe they can't think? What if that's all thinking is? We don't know.

deleted 3y ago
[deleted]
phailhaus 3y ago
It's not the the second statement follows from the first; I'm asserting that models can't think, and that what they're really doing is prediction based on context. The fact that chain-of-thought reasoning yields significantly better results is your hint: that means that the model doesn't think like a human does when it comes up with responses. If it's not in the context, it doesn't exist. You can't ask a model "why did you answer that way" without it generating from whole cloth a plausible retroactive reason. But there is no memory, so it can't really tell you. > What if that's all thinking is? I think that this is roughly true. But we actually have memory outside of what we say, so when we think, all those intermediate steps are persisted in our brains. For a model, the context is the memory. If you delete your question from the context and ask it "why did you answer that way", it will have no idea what you're talking about.
- stavros 3y ago
  > You can't ask a model "why did you answer that way" without it generating from whole cloth a plausible retroactive reason. I've caught myself generating from whole cloth a plausible retroactive reason for some of my actions, which I later realized wasn't true at all. Does that mean I can't think either? Is there a way for an external observer to tell if someone is thinking or not, or is it something that, axiomatically, only humans can do, and nothing else?
  - phailhaus 3y ago
    Again, you have memory but models don't. If it's not in the context, it doesn't exist. This is really easy to prove: delete your question from the context and ask the model why it answered the way it did. It will not know, because the model prediction service is stateless. It only takes the context as input. It's like if you and I weren't able to remember anything that wasn't written down. That's why you can get models to approximate thought by telling it to write down its intermediate steps.
    - stavros 3y ago
      Right, exactly, the context is the model's memory. All you're saying is "except for their memory, they have no memory", which goes for humans as well.
      - phailhaus 3y ago
        If you say that the models have memory and can think, you are heavily implying that there is a brain in the cloud that is capable of "thinking" or "reasoning" through your request. That's what it means for humans: we're self-contained. Models aren't like people. The context is not "part" of the model, the context is given to the model when you ask it for a prediction. It's like you cut a small part of a person's brain out. Alone, it can't think. It's a pedantic distinction, but an important one. The model itself isn't capable of thinking, but if you package it up with a context that it can manipulate at-will, that combination of parts can be said to "think". That's probably going to be the next step in LLM tech.
        stavros 3y ago
        By that definition, ChatGPT can already think.
        water-your-self 3y ago
        If this is your conclusion, I would recommend reading more of the research.
andoando 3y ago
I think a fundamental difference is we are able to learn new things by reasoning through our existing knowledge. Moreover our beliefs are mostly consistent with each other, and we can be argued with and have our beliefs changed. As far as I understand, GPT isn't going to alter its whole worldview if you show it that its thinking is flawed. But perhaps this is possible if upon discovering a flaw, it looped through its corpus and altered its connections?