3 ms·

What experiment would you run to determine if a given text input / text output interface had an "internal understanding of chess"?

gwright 4y ago
What if you prompted with something like: Let's play a game chess. Use the standard rules except that .... Basically perturb the context to something a human would easily adapt to if they first knew the rules of chess but that would be difficult (or at least not obvious) to extrapolate from training data by ChatGPT (or more generally an LLM)
jltsiren 4y ago
I think internal understanding requires internal processing. According to this functional definition, the way we are currently using language models basically excludes understanding. We are asking them to dream up or brainstorm things – to tell us the first things they associate with the prompt. Maybe it's possible to set up the system with some kind of self-feedback loop, where it continues evaluating and improving its answers without further prompts. If that works, it would be one step closer to a true AGI that can be said to understand things. There is a lot of confusion around the Chinese Room Argument. I think it makes a valid point by demonstrating that input/output behavior alone is insufficient for evaluating whether a system is intelligent and understands things. In order to do that, we need to see (or assume) the internal mechanism.
- chpatrick 4y ago
  > Maybe it's possible to set up the system with some kind of self-feedback loop, where it continues evaluating and improving its answers without further prompts. It can do that while it generates output. Humans do the same thing when they figure out what they really mean while they're trying to express it.
  - jltsiren 4y ago
    I was thinking more about the equivalent of a human noticing that the initial answer they were going to give is wrong and then thinking about the topic for a while before coming up with a better answer.

gwright 4y ago

What if you prompted with something like: Let's play a game chess. Use the standard rules except that .... Basically perturb the context to something a human would easily adapt to if they first knew the rules of chess but that would be difficult (or at least not obvious) to extrapolate from training data by ChatGPT (or more generally an LLM)

jltsiren 4y ago

I think internal understanding requires internal processing. According to this functional definition, the way we are currently using language models basically excludes understanding. We are asking them to dream up or brainstorm things – to tell us the first things they associate with the prompt. Maybe it's possible to set up the system with some kind of self-feedback loop, where it continues evaluating and improving its answers without further prompts. If that works, it would be one step closer to a true AGI that can be said to understand things. There is a lot of confusion around the Chinese Room Argument. I think it makes a valid point by demonstrating that input/output behavior alone is insufficient for evaluating whether a system is intelligent and understands things. In order to do that, we need to see (or assume) the internal mechanism.

chpatrick 4y ago
> Maybe it's possible to set up the system with some kind of self-feedback loop, where it continues evaluating and improving its answers without further prompts. It can do that while it generates output. Humans do the same thing when they figure out what they really mean while they're trying to express it.
- jltsiren 4y ago
  I was thinking more about the equivalent of a human noticing that the initial answer they were going to give is wrong and then thinking about the topic for a while before coming up with a better answer.