4 ms·

That's why I said if it made no mistakes it would offer strong evidence. We couldn't know it would never make a mistake, but the evidence would be wiggling its

That's why I said if it made no mistakes it would offer strong evidence. We couldn't know it would never make a mistake, but the evidence would be wiggling its eyebrows suggestively. Similarly, the presence of mistakes doesn't completely damn the assertion that there is a world model at play. The world model could simply be inaccurate. All I'm saying is that since there are mistakes, we know for a fact there isn't an accurate world model at play. Absent that possibility, there's not compelling evidence to say anything about the nature OthelloGPT. It's the exact same kind of inscrutable as every other LLM.

naasking 4y ago
> All I'm saying is that since there are mistakes, we know for a fact there isn't an accurate world model at play. But I'm saying something different: even if a system has a world model that it believes is accurate, it still should be making intentional "mistakes" to test the world model in unusual circumstances, just like we continue to stress test thermodynamics, electromagnetism, gravity, etc. So the evidence that GPT still makes mistakes is not necessarily evidence that it doesn't have an accurate world model. It could be evidence that it has a very rational approach to handling evidence (you should never have 100% confidence!) and is continuously testing world models.
- mountainriver 4y ago
  I don’t think we’ve seen any evidence of experimentation in these models, it is in fact the key component they are missing
  - naasking 4y ago
    Sure, but my point is simply that evidence of mistakes does not necessarily entail a lack of an accurate world model. When underlying models can only be inferred, there can never be 100% certainty that any inferred model is truly accurate, even when it is. Humans are only certain of Othello's rules because we have direct access to those rules. If you put a human being in GPT's place in this exact same scenario, they too would make comparable mistakes. Humans are clearly capable of world models, therefore those mistakes are not an indication that world models are not in use.
- nickelpro 4y ago
  > even if a system has a world model that it believes is accurate, it still should be making intentional "mistakes" to test the world model in unusual circumstances I have no idea how to address this. StockFish is an accurate world model of chess. It doesn't make mistakes, it cannot make mistakes. That's what an accurate world model is. A system that can intentionally make mistakes is likely not a world model of anything at all. If it is a world model of something, it is a world model of something other than Othello, which does not have rules for "stress test"ing. The OP is claiming that they have created an accurate world model for Othello, so I think discussion of intentional mistakes and thermodynamics is outside the scope.
  - riwsky 4y ago
    Under your definition, humans don’t have world models either. If you just gave someone the move sequences, and no other knowledge of Othello, they too would have no way to be 100% sure that other moves won’t show up. It’s not a very useful definition.
  - naasking 4y ago
    > The OP is claiming that they have created an accurate world model for Othello, so I think discussion of intentional mistakes and thermodynamics is outside the scope. It's not out of scope because these all fall under the problem of induction, which is what I mentioned in my first post. There is no such thing as achieving "certainty" in such scenarios where you don't have direct access to the underlying model, there is only quantified uncertainty. This was all formalized under Solomonoff Induction. So I'm making three points: 1. Your requirement for "knowledge" to be "100% certainty" in such scenarios is just the wrong way to look at it, because such certainty isn't possible in these scenarios, even in principle, even for humans that are capable of world models, eg. swap a human in for GPT and they'll also achieve a non-zero error rate, even if they try to build a model of the rules. There is no well-defined, quantifiable threshold at which "quantified uncertainty" can become "certainty" and thus what you define as "knowledge". Therefore "knowledge" cannot be equated with "certainty" in these domains. The kind of "knowledge" you want is only possible in cases when you have direct access to the model to begin with, like being told the rules of Othello. 2. Even if you do happen upon an accurate model, you'd never know it and so have to continuously retest it by trying to violate it, so you cannot infer the lack of a internal model from the existence of an error rate. The point I was making with this is that your argument that "a non-zero error rate entails a lack of an accurate world model" is invalid, not that GPT necessarily has an accurate world model in this case. 3. I also dispute the distinction you're trying to make between statistical "pattern matching" and "understanding". "Understanding" must have some mechanistic basis which will have something like this kind of pattern matching. I assume you agree that a formalization of Othello's rules in classical logic would qualify as a world model. Bayesian probability theory where all the probabilities are pinned to 0 or 1 reduces to classical logic. Therefore, an inferred statistical model that asymptotically approaches this classical logic model, which is all we can do in these black box scenarios, is arguably operating based on an inferred world model with some inevitable degree of uncertainty as to specifically which world it's inhabiting.
fenomas 4y ago
> if it made no mistakes it would offer strong evidence I don't see how this has anything to do with the article. TFA's authors weren't trying to make the GPT "know the rules" of Othello, or anything similar. The GPT doesn't even know there are rules, or that a game is being played, so the whole question is neither here nor there. Rather, TFA is asking: given that the GPT learned to predict token sequences that correspond to Othello moves, did it do that with or without inferring the presence of a gameboard and simulating the on/off states of that board? The answer to that question has nothing to do with whether the GPT predicts valid token sequences with 100% or 99% or 95% accuracy, the issue is or isn't simulating the state of the Othello pieces.
- nickelpro 4y ago
  This is not what they claim, the central claim is that OthelloGPT has developed a "causal model", "an understandable model of the process producing the sequences" (For example: "are they picking up the rules of English grammar?"). Ie, does OthelloGPT know the rules of Othello? They make a clear distinction between world representation ("board state"/"internal activation"), and world model ("game engine"/"LLM Layers"), see their diagram [1]. That state exists it not interesting or new, a LLM necessarily transforms input sequences into a form of state in order to predict the next word in a sentence, and OthelloGPT necessarily transforms the move sequence into an internal state in order to predict the next move. The only interesting question is what mechanism picks the next move, simply following the patterns (the crow, calling out the next move), or an understanding of the rules of the game (the crow, analyzing a fresh board). Except that's a false dichotomy, the analogy doesn't work, the crow "analyzing a fresh board" is still capable of just doing pattern recognition. [1]: https://lh6.googleusercontent.com/zvQObJtkqyFey9TD2Ibzx5K9s5MaMI_YswkHgBWCNf6wBfKhboAElcRUAsPzXSbToBsXsK8C-fsfOV4Uku7sq5u2rNTdhqtpCEMJ2rj_OUXIsfL9QrCmmxfTYPoJ67vDV0aLHJ7cKEhBFIxQ41IEolu0KbruZnemSx_Orf5TabEdKsso1NXw_FB-rzLH https://lh6.googleusercontent.com/zvQObJtkqyFey9TD2Ibzx5K9s5...
  - fenomas 4y ago
    The crow never analyzes any board - that's the point of the analogy. It only hears a sequence of tokens, and it learns to predict future tokens, and TFA sets out to answer whether it does that by "surface statistics" (i.e. which tokens are more likely at which times), or with a "world model" (by laying out a bunch of seeds and flipping some of them over after each token in order to simulate the state of the game). Edit: from other replies, I think I see the issue here. In TFA, "world model" is referring to the insight that "D4, E5, B3" are more than just tokens in a grammar - that they represent graph nodes with state, and that when each token occurs it toggles the states of other graph nodes. TFA is asking whether the LLM has learned that insight, as opposed to merely detecting patterns in the token sequences without any model of what they represent. You seem to be taking "world model" to mean something quite different - like "a rule-based understanding of game rules that isn't statistical" or such. TFA doesn't discuss or investigate anything along those lines, just whether the LLM is simulating the on/off state of Othello pieces.
Isinlor 4y ago
Humans make mistakes with rules too.

naasking 4y ago

> All I'm saying is that since there are mistakes, we know for a fact there isn't an accurate world model at play. But I'm saying something different: even if a system has a world model that it believes is accurate, it still should be making intentional "mistakes" to test the world model in unusual circumstances, just like we continue to stress test thermodynamics, electromagnetism, gravity, etc. So the evidence that GPT still makes mistakes is not necessarily evidence that it doesn't have an accurate world model. It could be evidence that it has a very rational approach to handling evidence (you should never have 100% confidence!) and is continuously testing world models.

mountainriver 4y ago
I don’t think we’ve seen any evidence of experimentation in these models, it is in fact the key component they are missing
- naasking 4y ago
  Sure, but my point is simply that evidence of mistakes does not necessarily entail a lack of an accurate world model. When underlying models can only be inferred, there can never be 100% certainty that any inferred model is truly accurate, even when it is. Humans are only certain of Othello's rules because we have direct access to those rules. If you put a human being in GPT's place in this exact same scenario, they too would make comparable mistakes. Humans are clearly capable of world models, therefore those mistakes are not an indication that world models are not in use.
nickelpro 4y ago
> even if a system has a world model that it believes is accurate, it still should be making intentional "mistakes" to test the world model in unusual circumstances I have no idea how to address this. StockFish is an accurate world model of chess. It doesn't make mistakes, it cannot make mistakes. That's what an accurate world model is. A system that can intentionally make mistakes is likely not a world model of anything at all. If it is a world model of something, it is a world model of something other than Othello, which does not have rules for "stress test"ing. The OP is claiming that they have created an accurate world model for Othello, so I think discussion of intentional mistakes and thermodynamics is outside the scope.
- riwsky 4y ago
  Under your definition, humans don’t have world models either. If you just gave someone the move sequences, and no other knowledge of Othello, they too would have no way to be 100% sure that other moves won’t show up. It’s not a very useful definition.
- naasking 4y ago
  > The OP is claiming that they have created an accurate world model for Othello, so I think discussion of intentional mistakes and thermodynamics is outside the scope. It's not out of scope because these all fall under the problem of induction, which is what I mentioned in my first post. There is no such thing as achieving "certainty" in such scenarios where you don't have direct access to the underlying model, there is only quantified uncertainty. This was all formalized under Solomonoff Induction. So I'm making three points: 1. Your requirement for "knowledge" to be "100% certainty" in such scenarios is just the wrong way to look at it, because such certainty isn't possible in these scenarios, even in principle, even for humans that are capable of world models, eg. swap a human in for GPT and they'll also achieve a non-zero error rate, even if they try to build a model of the rules. There is no well-defined, quantifiable threshold at which "quantified uncertainty" can become "certainty" and thus what you define as "knowledge". Therefore "knowledge" cannot be equated with "certainty" in these domains. The kind of "knowledge" you want is only possible in cases when you have direct access to the model to begin with, like being told the rules of Othello. 2. Even if you do happen upon an accurate model, you'd never know it and so have to continuously retest it by trying to violate it, so you cannot infer the lack of a internal model from the existence of an error rate. The point I was making with this is that your argument that "a non-zero error rate entails a lack of an accurate world model" is invalid, not that GPT necessarily has an accurate world model in this case. 3. I also dispute the distinction you're trying to make between statistical "pattern matching" and "understanding". "Understanding" must have some mechanistic basis which will have something like this kind of pattern matching. I assume you agree that a formalization of Othello's rules in classical logic would qualify as a world model. Bayesian probability theory where all the probabilities are pinned to 0 or 1 reduces to classical logic. Therefore, an inferred statistical model that asymptotically approaches this classical logic model, which is all we can do in these black box scenarios, is arguably operating based on an inferred world model with some inevitable degree of uncertainty as to specifically which world it's inhabiting.

fenomas 4y ago

> if it made no mistakes it would offer strong evidence I don't see how this has anything to do with the article. TFA's authors weren't trying to make the GPT "know the rules" of Othello, or anything similar. The GPT doesn't even know there are rules, or that a game is being played, so the whole question is neither here nor there. Rather, TFA is asking: given that the GPT learned to predict token sequences that correspond to Othello moves, did it do that with or without inferring the presence of a gameboard and simulating the on/off states of that board? The answer to that question has nothing to do with whether the GPT predicts valid token sequences with 100% or 99% or 95% accuracy, the issue is or isn't simulating the state of the Othello pieces.

nickelpro 4y ago
This is not what they claim, the central claim is that OthelloGPT has developed a "causal model", "an understandable model of the process producing the sequences" (For example: "are they picking up the rules of English grammar?"). Ie, does OthelloGPT know the rules of Othello? They make a clear distinction between world representation ("board state"/"internal activation"), and world model ("game engine"/"LLM Layers"), see their diagram [1]. That state exists it not interesting or new, a LLM necessarily transforms input sequences into a form of state in order to predict the next word in a sentence, and OthelloGPT necessarily transforms the move sequence into an internal state in order to predict the next move. The only interesting question is what mechanism picks the next move, simply following the patterns (the crow, calling out the next move), or an understanding of the rules of the game (the crow, analyzing a fresh board). Except that's a false dichotomy, the analogy doesn't work, the crow "analyzing a fresh board" is still capable of just doing pattern recognition. [1]: https://lh6.googleusercontent.com/zvQObJtkqyFey9TD2Ibzx5K9s5MaMI_YswkHgBWCNf6wBfKhboAElcRUAsPzXSbToBsXsK8C-fsfOV4Uku7sq5u2rNTdhqtpCEMJ2rj_OUXIsfL9QrCmmxfTYPoJ67vDV0aLHJ7cKEhBFIxQ41IEolu0KbruZnemSx_Orf5TabEdKsso1NXw_FB-rzLH https://lh6.googleusercontent.com/zvQObJtkqyFey9TD2Ibzx5K9s5...
- fenomas 4y ago
  The crow never analyzes any board - that's the point of the analogy. It only hears a sequence of tokens, and it learns to predict future tokens, and TFA sets out to answer whether it does that by "surface statistics" (i.e. which tokens are more likely at which times), or with a "world model" (by laying out a bunch of seeds and flipping some of them over after each token in order to simulate the state of the game). Edit: from other replies, I think I see the issue here. In TFA, "world model" is referring to the insight that "D4, E5, B3" are more than just tokens in a grammar - that they represent graph nodes with state, and that when each token occurs it toggles the states of other graph nodes. TFA is asking whether the LLM has learned that insight, as opposed to merely detecting patterns in the token sequences without any model of what they represent. You seem to be taking "world model" to mean something quite different - like "a rule-based understanding of game rules that isn't statistical" or such. TFA doesn't discuss or investigate anything along those lines, just whether the LLM is simulating the on/off state of Othello pieces.

Isinlor 4y ago

Humans make mistakes with rules too.