5 ms·

> We found that the trained Othello-GPT usually makes legal moves. The error rate is 0.01%; and for comparison, the untrained Othello-GPT has an error rate of 9

> We found that the trained Othello-GPT usually makes legal moves. The error rate is 0.01%; and for comparison, the untrained Othello-GPT has an error rate of 93.29%. This is much like the observation in our parable that the crow was announcing the next moves. I read the whole article, but I think this statement largely disproves the hypothesis? We only need a single counter example to show that Othello-GPT does not have a systemic understanding of the rules, only a statistical inference of them. A model that "knows" the rules will make no errors, a model that makes any errors does not "know" the rules. Simple as. And I feel this way about much of the article, they state they change intermediate activations and therefore the observed valid results prove that the layers make up a rules engine instead of a statistical engine. And I don't make that leap? Why would that necessarily be the case? Obviously a statistical engine trained on legal moves will mostly produce legal output. A rules engine will always produce legal output.

cjg 4y ago
I know the rules of chess, but my rate of attempting illegal moves is above 0%.
- nickelpro 4y ago
  A human mind is not a rules model for Chess, and comparisons to a human mind are worthless. We're not asking if the LLM is human, only if it is modeling the game rules or modeling statistics My point is this: * If the results of the model were perfect, not a single counter-example, that would be a strong argument that the LLM has built an internal model of Othello * If counter-examples exist, the result of the experiment is indistinguishable from a sufficiently clever statistical inference and proves nothing
  - jackmott 4y ago
    [dead]
  - hcks 4y ago
    So you assert that parent doesn’t have an internal model of chess? Or that only proved expert systems built specifically to play chess have an internal model of it?
    - nickelpro 4y ago
      > So you assert that parent doesn’t have an internal model of chess? I think it's an interesting question, I find it more likely we have an imperfect ability to perceive the whole chess board, whatever the hell that might mean. In any case it's irrelevant to an LLM which has perfect information from the input set. So perhaps we rephrase the question, I think the parent, if given a list of individual chess moves (in whatever notation pleases them), should be able to tell if a given move is valid or not for the given piece. If the parent failed to do that, I would say they do not have a valid model of chess, ya.
      - shawnz 4y ago
        > If the parent failed to do that, I would say they do not have a valid model of chess Surely even the world's foremost chess experts will occasionally make errors in such a situation, so what's the point of holding language models to an intelligence standard that even humans can't meet? I see two ways of interpreting this: either language models do approximately learn world models, or else world models aren't necessary for human intelligence EDIT: I see from your other posts in this thread you're probably more inclined to believe the latter statement, which wasn't obvious to me from your first post and I think that explains why there is so much contention in this thread.
  - noiv 4y ago
    Well, if the model has a 100% grasp of the rules, why call it a model?
    - nickelpro 4y ago
      Because it's not a physical othello board? It's just an internal model of an othello board encoded in the weights of a neural net? Insert whatever semantics float your particular goat
      - noiv 4y ago
        I thought the whole point was about modelling rules not atoms.
- bjornsing 4y ago
  Especially if you’re playing blind. I’d say an error rate of less than %1 is pretty good in that case.
karmakaze 4y ago
Just because a move has never been played, doesn't make it illegal, only untried. Only when the ruleset itself is laid out as overarching the data (as in AlphaGo/Zero) should they always be respected. Things that are observed to be true are called (in math/science) "laws" aka rules of thumb. Actual rules have other names. Most people have a world-view/mental-model of how money works. Few actually know the rules and what are possible things under the extreme conditions. Edit: in the case of AlphaGo/Zero the interesting thing wasn't the rules. They were hard-coded. The interesting thing is that it likely did form mental/world-models in higher layers above the rules in the form of tactics and strategies, these are the valuable parts.
- nickelpro 4y ago
  If the only understanding of the rules is what has been tried/untried, that is definitionally a statistical model. A valid rules model does not require to see all possible rules be played to understand the legality of a move. So the LLM under study here could be at most, an invalid rules model. But an invalid rules model is indistinguishable from a statistical model, and the authors give no evidence for why we should believe it to be the former instead of the latter.
  - pyinstallwoes 4y ago
    So, quantum mechanics?
  - karmakaze 4y ago
    An interesting comparison is between AlphaZero and MuZero[0], the latter having not been given the rules and has improved upon AlphaZero in Go. > "MuZero really is discovering for itself how to build a model and understand it just from first principles." — David Silver, DeepMind So yes to both: statistical and a world model. [0] https://en.wikipedia.org/wiki/MuZero https://en.wikipedia.org/wiki/MuZero
- crosen99 4y ago
  > Just because a move has never been played, doesn't make it illegal, only untried. Well, I think this is nickelpro's point. Namely that the model "does not have a systemic understanding of the rules, only a statistical inference of them." However, I would still counter nickelpro's point with the idea that what we call systemic understanding is fallible - and most likely statistical in nature. Another response below gives the example of a human understanding the rules of chess but still making incorrect moves. I might even say that any system truly capable of understanding is inherently fallible. A digital calculator or conventional database is impressive in its accuracy and consistency precisely because the rigor within its programming operates beyond the realm of understanding.
williamcotton 4y ago
> A model that "knows" the rules will make no errors, a model that makes any errors does not "know" the rules. Simple as. Wittgenstein would ask, does it matter if the model “knows all the precise rules” or not? Does it matter if I know all the precise rules? Can two of us have a fun game of chess if we basically know all the rules and mainly make legal moves or at least not notice when we break a rule or can agree on an action when we’re confused? What’s important is that we or the model play the same game with others. We don’t need to worry too much about the definitions. We just need to observe the results to deduce if there was any sort of meaningful interactions. What exactly is offsides in soccer? Or pass interference in football? Or a balk in baseball?
- nickelpro 4y ago
  If your claim is that you've produced a computer program that accurately models the rules, it matters a great deal. It is, perhaps, the only thing that matters. A calculator that says 2+2=5, even 0.01% of the time, is not a very good calculator.
  - williamcotton 4y ago
    Yeah, but we’re trying to play football here. We’ve already got calculators!
GaggiX 4y ago
Even chess grandmasters may rarely make illegal moves, but I doubt you can argue that they don't know the rules unless you also believe that humans can only learn surface statistics and not world models.
- nickelpro 4y ago
  If the chess grandmaster makes an illegal move, than in that moment, they are not playing by the rules. It's definitional. The GM was, in that moment, operating by some other model to evaluate the board state, likely a pattern recognition/statistical model that did not fully evaluate the state of the board. ChatGPT can, today, list all the rules of chess accurately. ChatGPT cannot play chess with any great degree of accuracy. "Knowing" the rules, in so much as the ability to list them out accurately, is not the same as following the rules. There's no reason to believe that humans operate as rules engines in all things, and a great deal of evidence (as with your example of chess GMs making mistakes) that they absolutely are not. That human intelligence is itself partially, and perhaps even largely, statistical in nature.
  - GaggiX 4y ago
    >he is not playing by the rules. It's definitional. Sure but the question is: does the grandmaster know the rules? Because if what you said is true: >A model that "knows" the rules will make no errors, a model that makes any errors does not "know" the rules. Simple as. Then the grandmaster doesn't know the rules. "Simple as".
    - nickelpro 4y ago
      The GM is not a game model of chess. The GM is a human. The GM does not evaluate their world according to the rules of chess at all times or in all things. The claim is that OthelloGPT is a game model of Othello. The only input it has is a move list, the only output it produces is a resulting move. Actual game models, such as Stockfish for chess, are incapable of producing invalid moves, it is not in their set of possible outputs. OthelloGPT is not a game model of Othello. It might be a game model of something that is not Othello, but this is hard to tell. The reliance on what humans can do is not a useful comparison here. Humans are also not game models of Othello. Actual game models, programs that model the rules of Othello, are the useful point of comparison. Again, claiming to know the rules, "knowledge" of them, and following those rules, acting according to them, are totally different things. The former is outside the scope, "what is uppercase-k Knowledge" is a question of philosophy. Demonstrably following the rules is the world of game models. If the semantics of putting the word "know" in quotes is what you're hung up on, fine, replace it with whatever pleases you: A model that "follows/encodes/exhibits" the rules will make no errors, a model that makes any errors does not "follow/encode/exhibit" the rules
naasking 4y ago
> A model that "knows" the rules will make no errors, a model that makes any errors does not "know" the rules. Simple as. This is the problem of induction. How many white swans do you have to see before you can claim to "know" that all swans are white? There is no amount where you can make this claim with certainty, right? The reason you can claim knowledge of fixed rules here is because you already knew there were fixed rules going in. You didn't have to infer the existence of these rules merely from looking at the inputs and outputs. GPT is analyzing this as a black box and it's doing exactly the rational thing: it's inferring a strong confidence that a certain class of moves are legal, but leaving a small room for doubt in case there's some corner case it hadn't yet considered. I agree there is more room for better generalization [1,2] that will take statistical correlations and attempt to infer explicit models to explain those correlations, but there must always remain some quantifiable doubt that those models are true, and the "desire" to make the wrong move sometimes to test your "knowledge". [1] https://pubmed.ncbi.nlm.nih.gov/31639888/ https://pubmed.ncbi.nlm.nih.gov/31639888/ [2] https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.128.180201 https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.12...
- nickelpro 4y ago
  That's why I said if it made no mistakes it would offer strong evidence. We couldn't know it would never make a mistake, but the evidence would be wiggling its eyebrows suggestively. Similarly, the presence of mistakes doesn't completely damn the assertion that there is a world model at play. The world model could simply be inaccurate. All I'm saying is that since there are mistakes, we know for a fact there isn't an accurate world model at play. Absent that possibility, there's not compelling evidence to say anything about the nature OthelloGPT. It's the exact same kind of inscrutable as every other LLM.
  - naasking 4y ago
    > All I'm saying is that since there are mistakes, we know for a fact there isn't an accurate world model at play. But I'm saying something different: even if a system has a world model that it believes is accurate, it still should be making intentional "mistakes" to test the world model in unusual circumstances, just like we continue to stress test thermodynamics, electromagnetism, gravity, etc. So the evidence that GPT still makes mistakes is not necessarily evidence that it doesn't have an accurate world model. It could be evidence that it has a very rational approach to handling evidence (you should never have 100% confidence!) and is continuously testing world models.
colordrops 4y ago
> A model that "knows" the rules will make no errors, a model that makes any errors does not "know" the rules. Simple as. Does this mean that human minds are merely statistical inference engines and don't truly get the models either? Because they make tons of mistakes. I'm starting to think that at scale, quantity starts becoming quality, and there isn't a fundamental difference between a model and statistical inference. It's like vector graphics vs pixels, after a certain level of detail they are the same.
- nickelpro 4y ago
  Yes, there's not reason to think that every movement, every decision, that humans make is made according to a purely rules-modeled decision engine. If a GM makes an invalid move at chess, they demonstrably were not following the rules in that moment. That decision wasn't made according to a rules engine. They scanned the board, made a series of statistical inferences and other processes still well beyond our ability to describe, and made a move without validating it against all the rules and a complete observation of the board. This doesn't tell us anything about OthelloGPT, and honestly I think it was self-evident about humans already. "We are not calculators" shouldn't be a surprise to anyone.
bsk7 4y ago
> We only need a single counter example to show that Othello-GPT does not have a systemic understanding of the rules, only a statistical inference of them. I don't agree with this binary interpretation. This only indicates that any systemic understanding the model has internally built is not complete. We are not trying to assess completeness of that understanding. E.g. if you had a traditional rules engine for Othello, and you removed one rule that could result in illegal moves, does that make that rule engine a statistical model all of a sudden?
- nickelpro 4y ago
  No, it makes it an engine for a game that is not Othello. An inaccurate engine and a statistical one are indistinguishable. And the authors provide no evidence to prefer the former. Nothing is ruled out, of course, but there's not strong or compelling evidence either.
  - bsk7 4y ago
    > An inaccurate engine and a statistical one are indistinguishable. How so? The paper is trying to assess if some semantic understanding is being created by the underlying model, not if a completely accurate one is. If such a world model maps to Othello-prime (gleaning concepts like tiles, colors etc), that is still a very interesting result.
CuriouslyC 4y ago
Children also occasionally make incorrect moves in simple games, and I've seen grown adults who I know understand how knights move still accidentally place them in the wrong spot.
- saberience 4y ago
  But these are temporary accidents that given a few seconds to look at the board and the move that was made, any person would realize they had made a mistake. I.e. because we have the rules and understanding of the game modeled internally. A computer that only had a statistically inferred understanding wouldn't "realize" they had made a mistake and take back their move.
deleted 4y ago
[deleted]
codeulike 4y ago
No-one ever told it the rules, it was just shown a load of game-moves. If you explained Othello to a human using _only_ example moves, they would think they probably knew the rules, but they wouldn't be 100% sure either.

cjg 4y ago

I know the rules of chess, but my rate of attempting illegal moves is above 0%.

nickelpro 4y ago
A human mind is not a rules model for Chess, and comparisons to a human mind are worthless. We're not asking if the LLM is human, only if it is modeling the game rules or modeling statistics My point is this: * If the results of the model were perfect, not a single counter-example, that would be a strong argument that the LLM has built an internal model of Othello * If counter-examples exist, the result of the experiment is indistinguishable from a sufficiently clever statistical inference and proves nothing
- jackmott 4y ago
  [dead]
- hcks 4y ago
  So you assert that parent doesn’t have an internal model of chess? Or that only proved expert systems built specifically to play chess have an internal model of it?
  - nickelpro 4y ago
    > So you assert that parent doesn’t have an internal model of chess? I think it's an interesting question, I find it more likely we have an imperfect ability to perceive the whole chess board, whatever the hell that might mean. In any case it's irrelevant to an LLM which has perfect information from the input set. So perhaps we rephrase the question, I think the parent, if given a list of individual chess moves (in whatever notation pleases them), should be able to tell if a given move is valid or not for the given piece. If the parent failed to do that, I would say they do not have a valid model of chess, ya.
    - shawnz 4y ago
      > If the parent failed to do that, I would say they do not have a valid model of chess Surely even the world's foremost chess experts will occasionally make errors in such a situation, so what's the point of holding language models to an intelligence standard that even humans can't meet? I see two ways of interpreting this: either language models do approximately learn world models, or else world models aren't necessary for human intelligence EDIT: I see from your other posts in this thread you're probably more inclined to believe the latter statement, which wasn't obvious to me from your first post and I think that explains why there is so much contention in this thread.
- noiv 4y ago
  Well, if the model has a 100% grasp of the rules, why call it a model?
  - nickelpro 4y ago
    Because it's not a physical othello board? It's just an internal model of an othello board encoded in the weights of a neural net? Insert whatever semantics float your particular goat
    - noiv 4y ago
      I thought the whole point was about modelling rules not atoms.
bjornsing 4y ago
Especially if you’re playing blind. I’d say an error rate of less than %1 is pretty good in that case.

karmakaze 4y ago

Just because a move has never been played, doesn't make it illegal, only untried. Only when the ruleset itself is laid out as overarching the data (as in AlphaGo/Zero) should they always be respected. Things that are observed to be true are called (in math/science) "laws" aka rules of thumb. Actual rules have other names. Most people have a world-view/mental-model of how money works. Few actually know the rules and what are possible things under the extreme conditions. Edit: in the case of AlphaGo/Zero the interesting thing wasn't the rules. They were hard-coded. The interesting thing is that it likely did form mental/world-models in higher layers above the rules in the form of tactics and strategies, these are the valuable parts.

nickelpro 4y ago
If the only understanding of the rules is what has been tried/untried, that is definitionally a statistical model. A valid rules model does not require to see all possible rules be played to understand the legality of a move. So the LLM under study here could be at most, an invalid rules model. But an invalid rules model is indistinguishable from a statistical model, and the authors give no evidence for why we should believe it to be the former instead of the latter.
- pyinstallwoes 4y ago
  So, quantum mechanics?
- karmakaze 4y ago
  An interesting comparison is between AlphaZero and MuZero[0], the latter having not been given the rules and has improved upon AlphaZero in Go. > "MuZero really is discovering for itself how to build a model and understand it just from first principles." — David Silver, DeepMind So yes to both: statistical and a world model. [0] https://en.wikipedia.org/wiki/MuZero https://en.wikipedia.org/wiki/MuZero
crosen99 4y ago
> Just because a move has never been played, doesn't make it illegal, only untried. Well, I think this is nickelpro's point. Namely that the model "does not have a systemic understanding of the rules, only a statistical inference of them." However, I would still counter nickelpro's point with the idea that what we call systemic understanding is fallible - and most likely statistical in nature. Another response below gives the example of a human understanding the rules of chess but still making incorrect moves. I might even say that any system truly capable of understanding is inherently fallible. A digital calculator or conventional database is impressive in its accuracy and consistency precisely because the rigor within its programming operates beyond the realm of understanding.

williamcotton 4y ago

> A model that "knows" the rules will make no errors, a model that makes any errors does not "know" the rules. Simple as. Wittgenstein would ask, does it matter if the model “knows all the precise rules” or not? Does it matter if I know all the precise rules? Can two of us have a fun game of chess if we basically know all the rules and mainly make legal moves or at least not notice when we break a rule or can agree on an action when we’re confused? What’s important is that we or the model play the same game with others. We don’t need to worry too much about the definitions. We just need to observe the results to deduce if there was any sort of meaningful interactions. What exactly is offsides in soccer? Or pass interference in football? Or a balk in baseball?

nickelpro 4y ago
If your claim is that you've produced a computer program that accurately models the rules, it matters a great deal. It is, perhaps, the only thing that matters. A calculator that says 2+2=5, even 0.01% of the time, is not a very good calculator.
- williamcotton 4y ago
  Yeah, but we’re trying to play football here. We’ve already got calculators!

GaggiX 4y ago

Even chess grandmasters may rarely make illegal moves, but I doubt you can argue that they don't know the rules unless you also believe that humans can only learn surface statistics and not world models.

nickelpro 4y ago
If the chess grandmaster makes an illegal move, than in that moment, they are not playing by the rules. It's definitional. The GM was, in that moment, operating by some other model to evaluate the board state, likely a pattern recognition/statistical model that did not fully evaluate the state of the board. ChatGPT can, today, list all the rules of chess accurately. ChatGPT cannot play chess with any great degree of accuracy. "Knowing" the rules, in so much as the ability to list them out accurately, is not the same as following the rules. There's no reason to believe that humans operate as rules engines in all things, and a great deal of evidence (as with your example of chess GMs making mistakes) that they absolutely are not. That human intelligence is itself partially, and perhaps even largely, statistical in nature.
- GaggiX 4y ago
  >he is not playing by the rules. It's definitional. Sure but the question is: does the grandmaster know the rules? Because if what you said is true: >A model that "knows" the rules will make no errors, a model that makes any errors does not "know" the rules. Simple as. Then the grandmaster doesn't know the rules. "Simple as".
  - nickelpro 4y ago
    The GM is not a game model of chess. The GM is a human. The GM does not evaluate their world according to the rules of chess at all times or in all things. The claim is that OthelloGPT is a game model of Othello. The only input it has is a move list, the only output it produces is a resulting move. Actual game models, such as Stockfish for chess, are incapable of producing invalid moves, it is not in their set of possible outputs. OthelloGPT is not a game model of Othello. It might be a game model of something that is not Othello, but this is hard to tell. The reliance on what humans can do is not a useful comparison here. Humans are also not game models of Othello. Actual game models, programs that model the rules of Othello, are the useful point of comparison. Again, claiming to know the rules, "knowledge" of them, and following those rules, acting according to them, are totally different things. The former is outside the scope, "what is uppercase-k Knowledge" is a question of philosophy. Demonstrably following the rules is the world of game models. If the semantics of putting the word "know" in quotes is what you're hung up on, fine, replace it with whatever pleases you: A model that "follows/encodes/exhibits" the rules will make no errors, a model that makes any errors does not "follow/encode/exhibit" the rules

naasking 4y ago

> A model that "knows" the rules will make no errors, a model that makes any errors does not "know" the rules. Simple as. This is the problem of induction. How many white swans do you have to see before you can claim to "know" that all swans are white? There is no amount where you can make this claim with certainty, right? The reason you can claim knowledge of fixed rules here is because you already knew there were fixed rules going in. You didn't have to infer the existence of these rules merely from looking at the inputs and outputs. GPT is analyzing this as a black box and it's doing exactly the rational thing: it's inferring a strong confidence that a certain class of moves are legal, but leaving a small room for doubt in case there's some corner case it hadn't yet considered. I agree there is more room for better generalization [1,2] that will take statistical correlations and attempt to infer explicit models to explain those correlations, but there must always remain some quantifiable doubt that those models are true, and the "desire" to make the wrong move sometimes to test your "knowledge". [1] https://pubmed.ncbi.nlm.nih.gov/31639888/ https://pubmed.ncbi.nlm.nih.gov/31639888/ [2] https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.128.180201 https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.12...

nickelpro 4y ago
That's why I said if it made no mistakes it would offer strong evidence. We couldn't know it would never make a mistake, but the evidence would be wiggling its eyebrows suggestively. Similarly, the presence of mistakes doesn't completely damn the assertion that there is a world model at play. The world model could simply be inaccurate. All I'm saying is that since there are mistakes, we know for a fact there isn't an accurate world model at play. Absent that possibility, there's not compelling evidence to say anything about the nature OthelloGPT. It's the exact same kind of inscrutable as every other LLM.
- naasking 4y ago
  > All I'm saying is that since there are mistakes, we know for a fact there isn't an accurate world model at play. But I'm saying something different: even if a system has a world model that it believes is accurate, it still should be making intentional "mistakes" to test the world model in unusual circumstances, just like we continue to stress test thermodynamics, electromagnetism, gravity, etc. So the evidence that GPT still makes mistakes is not necessarily evidence that it doesn't have an accurate world model. It could be evidence that it has a very rational approach to handling evidence (you should never have 100% confidence!) and is continuously testing world models.

colordrops 4y ago

> A model that "knows" the rules will make no errors, a model that makes any errors does not "know" the rules. Simple as. Does this mean that human minds are merely statistical inference engines and don't truly get the models either? Because they make tons of mistakes. I'm starting to think that at scale, quantity starts becoming quality, and there isn't a fundamental difference between a model and statistical inference. It's like vector graphics vs pixels, after a certain level of detail they are the same.

nickelpro 4y ago
Yes, there's not reason to think that every movement, every decision, that humans make is made according to a purely rules-modeled decision engine. If a GM makes an invalid move at chess, they demonstrably were not following the rules in that moment. That decision wasn't made according to a rules engine. They scanned the board, made a series of statistical inferences and other processes still well beyond our ability to describe, and made a move without validating it against all the rules and a complete observation of the board. This doesn't tell us anything about OthelloGPT, and honestly I think it was self-evident about humans already. "We are not calculators" shouldn't be a surprise to anyone.

bsk7 4y ago

> We only need a single counter example to show that Othello-GPT does not have a systemic understanding of the rules, only a statistical inference of them. I don't agree with this binary interpretation. This only indicates that any systemic understanding the model has internally built is not complete. We are not trying to assess completeness of that understanding. E.g. if you had a traditional rules engine for Othello, and you removed one rule that could result in illegal moves, does that make that rule engine a statistical model all of a sudden?

nickelpro 4y ago
No, it makes it an engine for a game that is not Othello. An inaccurate engine and a statistical one are indistinguishable. And the authors provide no evidence to prefer the former. Nothing is ruled out, of course, but there's not strong or compelling evidence either.
- bsk7 4y ago
  > An inaccurate engine and a statistical one are indistinguishable. How so? The paper is trying to assess if some semantic understanding is being created by the underlying model, not if a completely accurate one is. If such a world model maps to Othello-prime (gleaning concepts like tiles, colors etc), that is still a very interesting result.

CuriouslyC 4y ago

Children also occasionally make incorrect moves in simple games, and I've seen grown adults who I know understand how knights move still accidentally place them in the wrong spot.

saberience 4y ago
But these are temporary accidents that given a few seconds to look at the board and the move that was made, any person would realize they had made a mistake. I.e. because we have the rules and understanding of the game modeled internally. A computer that only had a statistically inferred understanding wouldn't "realize" they had made a mistake and take back their move.

deleted 4y ago

[deleted]

codeulike 4y ago

No-one ever told it the rules, it was just shown a load of game-moves. If you explained Othello to a human using _only_ example moves, they would think they probably knew the rules, but they wouldn't be 100% sure either.