4 ms·

It is incapable of reasoning, actually - at least in this case. It has no internal understanding of chess which is why it makes illegal moves.

RyanCavanaugh 4y ago
What experiment would you run to determine if a given text input / text output interface had an "internal understanding of chess"?
- gwright 4y ago
  What if you prompted with something like: Let's play a game chess. Use the standard rules except that .... Basically perturb the context to something a human would easily adapt to if they first knew the rules of chess but that would be difficult (or at least not obvious) to extrapolate from training data by ChatGPT (or more generally an LLM)
- jltsiren 4y ago
  I think internal understanding requires internal processing. According to this functional definition, the way we are currently using language models basically excludes understanding. We are asking them to dream up or brainstorm things – to tell us the first things they associate with the prompt. Maybe it's possible to set up the system with some kind of self-feedback loop, where it continues evaluating and improving its answers without further prompts. If that works, it would be one step closer to a true AGI that can be said to understand things. There is a lot of confusion around the Chinese Room Argument. I think it makes a valid point by demonstrating that input/output behavior alone is insufficient for evaluating whether a system is intelligent and understands things. In order to do that, we need to see (or assume) the internal mechanism.
  - chpatrick 4y ago
    > Maybe it's possible to set up the system with some kind of self-feedback loop, where it continues evaluating and improving its answers without further prompts. It can do that while it generates output. Humans do the same thing when they figure out what they really mean while they're trying to express it.
    - jltsiren 4y ago
      I was thinking more about the equivalent of a human noticing that the initial answer they were going to give is wrong and then thinking about the topic for a while before coming up with a better answer.
sebzim4500 4y ago
Mostly it didn't make illegal moves though, since illegal moves mean resignation and it won more than it lost. Making 60 legal moves in a row in one game would be the coincidence of the century unless it had some knowledge of the rules of chess.
- henryfjordan 4y ago
  It's a probabilistic text model. If it has a 99% probability of generating an acceptable "next" thing to say, that means it would have a 50/50 chance of generating 60 legal moves in a row, which doesn't seem all that coincidental.
  - baq 4y ago
    Markov chains are probabilistic text models and rather far from 1400 elo
  - chpatrick 4y ago
    And the 99% probability isn't an evidence of understanding chess?
    - henryfjordan 4y ago
      I don't know. Part of me wants to say no, that the model "thinks" in terms of text it has seen and so knows from chess forums it has seen that certain text representing moves come naturally after previous moves' text. It doesn't understand anything other than certain text comes after other text. But yeah at the same time I can see how it is thinking inside the world we built for it. We have senses like touch, smell, sight. The only "sense" these models have are an input text box. Would we even necessarily recognize intelligence when it is so different from our own? So does it understand chess like I do? No, it cannot. Does it understand chess at all? I'm not sure. I'm not sure I'd understand chess in it's world either though.
chpatrick 4y ago
How did it win 11 out of 19 games then, blind luck?
- root_axis 4y ago
  raw statistical power.
  - bsaul 4y ago
    a game of chess becomes « new » after a few moves. starting middlegame, you’re in unknown territories and have no statistics to refer to..
    - root_axis 4y ago
      I'm referring to the statistical power of the model. For example, if you replace GPT4 with GPT2 it will lose every game, because the statistical power is lower. Increasing the statistical power doesn't make the model understand any better, it just makes it more likely to generate a response that aligns with human expectations.
      - chpatrick 4y ago
        "Statistical power" isn't some magic property of GPT4. It can produce statistically more likely moves because somewhere deep down it can model chess.
        root_axis 4y ago
        It isn't a model of chess, it's a model of internet text, if it was a model of chess it wouldn't make illegal moves.
        bsaul 4y ago
        if it didn't have at least some kind of model of chess, it wouldn't be able to play past midgame. Simply because on a new position, moves from other positions aren't applicable at all.
baq 4y ago
How do you know that? It has billions of parameters, some of them may well be for internal understanding of chess?

RyanCavanaugh 4y ago

What experiment would you run to determine if a given text input / text output interface had an "internal understanding of chess"?

gwright 4y ago
What if you prompted with something like: Let's play a game chess. Use the standard rules except that .... Basically perturb the context to something a human would easily adapt to if they first knew the rules of chess but that would be difficult (or at least not obvious) to extrapolate from training data by ChatGPT (or more generally an LLM)
jltsiren 4y ago
I think internal understanding requires internal processing. According to this functional definition, the way we are currently using language models basically excludes understanding. We are asking them to dream up or brainstorm things – to tell us the first things they associate with the prompt. Maybe it's possible to set up the system with some kind of self-feedback loop, where it continues evaluating and improving its answers without further prompts. If that works, it would be one step closer to a true AGI that can be said to understand things. There is a lot of confusion around the Chinese Room Argument. I think it makes a valid point by demonstrating that input/output behavior alone is insufficient for evaluating whether a system is intelligent and understands things. In order to do that, we need to see (or assume) the internal mechanism.
- chpatrick 4y ago
  > Maybe it's possible to set up the system with some kind of self-feedback loop, where it continues evaluating and improving its answers without further prompts. It can do that while it generates output. Humans do the same thing when they figure out what they really mean while they're trying to express it.
  - jltsiren 4y ago
    I was thinking more about the equivalent of a human noticing that the initial answer they were going to give is wrong and then thinking about the topic for a while before coming up with a better answer.

sebzim4500 4y ago

Mostly it didn't make illegal moves though, since illegal moves mean resignation and it won more than it lost. Making 60 legal moves in a row in one game would be the coincidence of the century unless it had some knowledge of the rules of chess.

henryfjordan 4y ago
It's a probabilistic text model. If it has a 99% probability of generating an acceptable "next" thing to say, that means it would have a 50/50 chance of generating 60 legal moves in a row, which doesn't seem all that coincidental.
- baq 4y ago
  Markov chains are probabilistic text models and rather far from 1400 elo
- chpatrick 4y ago
  And the 99% probability isn't an evidence of understanding chess?
  - henryfjordan 4y ago
    I don't know. Part of me wants to say no, that the model "thinks" in terms of text it has seen and so knows from chess forums it has seen that certain text representing moves come naturally after previous moves' text. It doesn't understand anything other than certain text comes after other text. But yeah at the same time I can see how it is thinking inside the world we built for it. We have senses like touch, smell, sight. The only "sense" these models have are an input text box. Would we even necessarily recognize intelligence when it is so different from our own? So does it understand chess like I do? No, it cannot. Does it understand chess at all? I'm not sure. I'm not sure I'd understand chess in it's world either though.

chpatrick 4y ago

How did it win 11 out of 19 games then, blind luck?

root_axis 4y ago
raw statistical power.
- bsaul 4y ago
  a game of chess becomes « new » after a few moves. starting middlegame, you’re in unknown territories and have no statistics to refer to..
  - root_axis 4y ago
    I'm referring to the statistical power of the model. For example, if you replace GPT4 with GPT2 it will lose every game, because the statistical power is lower. Increasing the statistical power doesn't make the model understand any better, it just makes it more likely to generate a response that aligns with human expectations.
    - chpatrick 4y ago
      "Statistical power" isn't some magic property of GPT4. It can produce statistically more likely moves because somewhere deep down it can model chess.
      - root_axis 4y ago
        It isn't a model of chess, it's a model of internet text, if it was a model of chess it wouldn't make illegal moves.
        bsaul 4y ago
        if it didn't have at least some kind of model of chess, it wouldn't be able to play past midgame. Simply because on a new position, moves from other positions aren't applicable at all.

baq 4y ago

How do you know that? It has billions of parameters, some of them may well be for internal understanding of chess?