7 ms·

Chess-GPT's Internal World Model

120 points by homarp 3y ago

homarp 3y ago
code here: https://github.com/adamkarvonen/chess_llm_interpretability https://github.com/adamkarvonen/chess_llm_interpretability
sinuhe69 3y ago
World model might be a too big word here. When we talk of a world model (in the context of AI motels), we refer to its understanding of the world, at least in the context we trained it. But what I see is just a visualization of the output in a fashion similar to a chess board. A stronger evidence would be a for example a map of the next move, which will show whether it truly understood the game’s rules. If it show probability larger than zero on illegal board fields, it will show us why it sometimes makes illegal moves. And obviously, it didn’t fully understand the rules of the game.
- canjobear 3y ago
  No, it is not a visualization of the output, it is a visualization of the information about pawn position contained in the model’s internal state.
- mitthrowaway2 3y ago
  > probability larger than zero Strictly speaking, it should be a mistake to assign a probability equal to zero to any moves, even for illegal board moves, but especially for an AI that learns by example and self-play. It never gets taught the rules, it only gets shown the games -- there's no reason that it should conclude that the probability of a rook moving diagonally is exactly zero just because it's never seen it happen in the data, and gets penalized in training every time it tries it. But even for a human, assigning probability of exactly zero is too strong. It would forbid any possibility that you misunderstand any rules, or forgot any special cases. It's a good idea to always maintain at least a small amount of epistemic humility that you might be mistaken about the rules, so that sufficiently overwhelmingly strong evidence could convince you that a move you thought was illegal turns out to be legal.
  - redcobra762 3y ago
    There's got to be a probability cut-off, though. LLMs don't infinitely connect every token with every other token, some aren't connected at all, even if some association is taught, right?
    - the8472 3y ago
      The weights have finite precision which means they represent value-ranges / have error bars. So even if the weight is exactly 0 it does not represent complete confidence in it never occurring.
      - redcobra762 3y ago
        A weight necessitates a relationship, but I’m arguing LLMs don’t create all relationships. So a connection wouldn’t even exist.
        yorwba 3y ago
        When relationships are represented implicitly by the magnitude of the dot product between two vectors, there's no particular advantage to not "creating" all relationships (i.e. enforcing orthogonality for "uncreated" relationships). On the contrary, by allowing vectors for unrelated concepts to be only almost orthogonal, it's possible to represent a much larger number of unrelated concepts. https://terrytao.wordpress.com/2013/07/18/a-cheap-version-of-the-kabatjanskii-levenstein-bound-for-almost-orthogonal-vectors/ https://terrytao.wordpress.com/2013/07/18/a-cheap-version-of... In machine learning, this phenomenon is known as polysemanticity or superposition https://transformer-circuits.pub/2022/toy_model/index.html https://transformer-circuits.pub/2022/toy_model/index.html
        redcobra762 3y ago
        That’s not right; there are many vectors that go unbuilt between unrelated tokens. Creating a ton of empty relationships would obviously generate an immense amount of useless data. Your links are not about actually orthogonal vectors, so they’re not relevant. Also that’s not what superposition is defined as in your own links: > In this paper, we use toy models — small ReLU networks trained on synthetic data with sparse input features — to investigate how and when models represent more features than they have dimensions. We call this phenomenon superposition
ericbarrett 3y ago
Thanks for linking the actual post—it was a great read. I'm not an ML expert, but the author really made it easy to follow their experiment's method and results.
triyambakam 3y ago
Is a linear probe part of observability/interpretability?
- canjobear 3y ago
  Yes, a pretty fundamental technique and one of the earliest. It lets you determine which layers contain what information among other things.
  - Legend2440 3y ago
    The downside is that it's a supervised technique, so you need to already know what you're looking for. It would be nice to have an unsupervised tool that could list out all the things the network has learned.
    - JoshuaDavid 3y ago
      Anthropic has published some cool stuff in that direction: https://transformer-circuits.pub/2023/monosemantic-features https://transformer-circuits.pub/2023/monosemantic-features
      - firejake308 3y ago
        Whoa, this is super cool! I can imagine if we had something like this for ChatGPT, we could use it to do some serious prompt engineering. Imagine seeing what specific neurons you were activating with your prompt, and being able to identify which word in your prompt was triggering an undesired behavior. Super cool stuff, excited to see if it scales
sanxiyn 3y ago
I mean, this seems obvious to me. How would the model predict the next move WITHOUT calculating the board state first? Yes, by memorization, but memorization hypothesis is easily rejected by comparison to training dataset in this case. It is possible the model calculates an approximate board state, which is different from the board state but equivalent for most games, but not all games. It would be interesting to train adversarial policy to check this. From KataGo attack we know this does happen for Go AIs: Go rules have a concept of liberty, but so called pseudoliberty is easier to calculate and equivalent for most cases (but not all cases). In fact, human programmers also used pseudoliberty to optimize their engines. Adversarial attack found Go AIs also use pseudoliberty.
- taneq 3y ago
  It’s one thing to think it’s obvious, but quite another to prove it. I think this is the true value of this kind of work, is that it’s helping to decipher what these models are actually doing. Far too often we hear “NNs / LLMs are black boxes” as if that’s the end of the conversation.
- Sharlin 3y ago
  Surprisingly many people seem to believe LLMs cannot form any deeper world models beyond superficial relationships between words, even if figuring out a "hidden" model allows for a big leap in prediction performance – in this case, a hypothesis corresponding to chess rules happens to be give the best bang for the buck for predicting strings that have chess notation structure. But the model could in principle just have learned a long list of rote heuristics that happen to predict notation strings well, without having made the inferential leap to a much simpler set of rules, and a learner weaker than a LLM could well have got stuck at that stage. I wonder how well a human (or a group of humans) would fare at the same task and if they could also successfully reconstruct chess even if they had no prior knowledge of chess rules or notation. (OTOH a GPT3+ level LLM certainly does know that chess notation is related to something called "chess", which is a "game" and has certain "rules", but to what extent is it able to actually utilize that information?)
- HarHarVeryFunny 3y ago
  > It is possible the model calculates an approximate board state Yes - this is exactly what the probes show. One interesting aspect is that it still learns to play when trained on blocks of move sequences starting from the MIDDLE of the game, so it seems it must be incrementally inferring the board state by what's being played rather than just by tracking the moves.
wavemode 3y ago
If you take a neural network that already knows the basic rules of chess and train it on chess games, you produce a chess engine. From the Wikipedia page on one of the strongest ever[1]: "Like Leela Zero and AlphaGo Zero, Leela Chess Zero starts with no intrinsic chess-specific knowledge other than the basic rules of the game. Leela Chess Zero then learns how to play chess by reinforcement learning from repeated self-play" [1]: https://en.wikipedia.org/wiki/Leela_Chess_Zero https://en.wikipedia.org/wiki/Leela_Chess_Zero
- btown 3y ago
  As described in the OP's blog post https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c... - one of the incredible things here is that the standard GPT architecture, trained from scratch from PGN strings alone, can intuit the rules of the game from those examples, without any notion of the rules of chess or even that it is playing a game. Leela, by contrast, requires a specialized structure of iterative tree searching to generate move recommendations: https://lczero.org/dev/wiki/technical-explanation-of-leela-chess-zero/ https://lczero.org/dev/wiki/technical-explanation-of-leela-c... Which is not to diminish the work of the Leela team at all! But I find it fascinating that an unmodified GPT architecture can build up internal neural representations that correspond closely to board states, despite not having been designed for that task. As they say, attention may indeed be all you need.
  - goatlover 3y ago
    What's the strength of play for the GPT architecture? It's impressive that it figures out the rules, but does it play strong chess? >> As they say, attention may indeed be all you need. I don't think drawing general conclusions about intelligence from a board game is warranted. We didn't evolve to play chess or Go.
    - jksk61 3y ago
      > What's the strength of play for the GPT architecture? It's impressive that it figures out the rules, but does it play strong chess? sometimes it is not a matter of "is it better? is it larger? is it more efficient?", but just a question. mountains are mountains, men are men.
MenhirMike 3y ago
I hope it performs better than ChatGPT: https://old.reddit.com/r/AnarchyChess/comments/10ydnbb/i_placed_stockfish_white_against_chatgpt_black/ https://old.reddit.com/r/AnarchyChess/comments/10ydnbb/i_pla... Though I will give it to ChatGPT, castling across the bishop was a genius move.
- silasdavis 3y ago
  I've definitely played against some children who would respawn pieces. Human level AI clearly.
Imnimo 3y ago
I'd be curious to see if, in the 1-2% of cases where the linear probe fails to predict board occupancy, the LLM also predicts (or at least assigns non-trivial probability to) a corresponding illegal move. For example, if the linear probe incorrectly thinks there's a bishop on b4, does the LLM give more probability to illegal bishop moves along the corresponding diagonals than to other illegal bishop moves?
tromp 3y ago
Would love to see a similar experiment for 9x9 Go, where the model also needs to learn the concepts of connected group and its liberties.
tromp 3y ago
I'm impressed and surprised that a relatively small model can learn so much from just the textual move records (not even full algebraic notation 1.e2-e4 e7-e5 2.Ng1-f3 but just 1.e4 e5 2.Nf3). Would love to see a similar experiment for 9x9 Go, where the model also needs to learn the concepts of connected group and its liberties.
tromp 3y ago
I'm impressed and surprised that a relatively small model can learn so much from just the textual move records. Not even full algebraic notation 1.e2-e4 e7-e5 2.Ng1-f3 but just 1.e4 e5 2.Nf3 ... It has to figure out what o-o and o-o-o mean just from what King and Rook moves appear later. And the only way it could learn that the King cannot move through a checked square while castling is that such situations never appear in its training set. Would love to see a similar experiment for 9x9 Go, where the model also needs to learn the concepts of connected group and its liberties.
kybernetikos 3y ago
> I fine-tuned GPT-2 on a 50 / 50 mix of OpenWebText and chess games, and it learned to play chess and continued to output plausible looking text. Maybe there’s something interesting to look at there? To me that suggests investigating whether there are aspects of human culture that can improve chess playing performance - i.e. whether just training on games produces less good results than training on games and literature. This seems plausible to me, even beyond literature that is explictly about the game - learning go proverbs, which are often phrased as life advice is a part of learning go, and games are embedded all through our culture, with some stories really illustrating that you have to 'know when to hold em, know when to fold em, know when to walk away, know when to run'.
emporas 3y ago
Nice experiment, even though we know that LLMs distill an internal world model representation of whatever they are trained on. The experiment could be a little better by using a more descriptive form of notation than PGN. PGN notation's strength is the shorthand properties of it, because it is used by humans while playing the game. That is far from being a strength as LLM training data. ML algorithms, and LLMs are trained better by feeding them more descriptive and accurate data, and verbosity is not a problem at all. There is the FEN notation in which in every move the entire board is encoded. One could easily imagine many different ways to describe a game, like encoding vertical and horizontal lines, listing what exact squares each piece is covering, what color squares, which of the pieces are able to move, and in each move generate one whole page of the board situation. I call this spatial navigation, in which the LLM learns the ins and outs of it's training data and it is able to make more informed guesses. Chess is fun and all, but code generation has the potential to be a lot better than just writing functions. By feeding the LLM the AST representation of the code, the tree of workspace files, public items, module hierarchy alongside with the code, it could be a significant improvement.
- anotherpaulg 3y ago
  By feeding the LLM the AST representation of the code, the tree of workspace files, public items, module hierarchy alongside with the code, it could be a significant improvement. Aider does this, using tree-sitter to build a “repository map”. This helps the LLM understand the overall code base and how it relates to the specific coding task at hand. https://aider.chat/docs/repomap.html https://aider.chat/docs/repomap.html More broadly, I agree with your sentiment that there is a lot of value in considering the best ways to structure the data we share with LLMs. Especially in the context of coding.
  - emporas 3y ago
    >Aider does this, using tree-sitter to build a “repository map”. This helps the LLM understand the overall code base and how it relates to the specific coding task at hand. Great stuff. >More broadly, I agree with your sentiment that there is a lot of value in considering the best ways to structure the data we share with LLMs. Especially in the context of coding. As the experiments on PHI-1 and PHI-2 from microsoft show, training data make a difference. The "textbooks is all you need" moto means better structured data, more clear data make a difference. https://arxiv.org/abs/2306.11644 https://arxiv.org/abs/2306.11644
aaron695 3y ago
[dead]
sjducb 3y ago
I’m curious how human like this LLM feels when you play it. One of the challenges to making fun chess bots is to make it play like a low or mid ranked human. The problem is that a stockfish based bot knows some very strong moves, but deliberately plays bad moves so it’s about the right skill level. The problem is that these bad moves are often very obvious. For example I’ll threaten a queen capture. Any human would see it and move their queen. The bot “blunders” and loses the queen to an obvious attack. It feels like the bot is letting you win which kills the enjoyment of playing with the bot. I think that this approach would create very human like games.
- Solvency 3y ago
  Isn't that more of a design issue than a bot AI issue?
  - HarHarVeryFunny 3y ago
    I'd call it an approach issue: LLM vs brute-force lookahead. An LLM is predicting what comes next per it's training set. If it's trained on human games then it should play like a human; if it's trained on Stockfish games, then it should play more like Stockfish. Stockfish, or any chess engine using brute force lookahead, is just trying to find the optimal move - not copying any style of play - and it's moves are therefore sometimes going to look very un-human. Imagine if the human player is looking 10-15 moves ahead, but Stockfish 40-50 moves ahead... what looks good 40-50 moves out might be quite different than what looks good to the human.
  - sjducb 3y ago
    Humans and machines find good moves in different ways. Most humans have fast pattern matching that is quite good at finding some reasonable moves. There are also classes of moves that all humans will spot. (You just moved your bishop, now it’s pointing at my queen) The problem is that stockfish scores all moves with a number based on how good the move is. You have no idea if a human would agree. For example mis-calculating a series of trades 4 moves deep is a very human mistake, but it’s scored the same as moving the bishop to a square where it can easily be taken by a pawn. They both result in you being a bishop down. A nerfed stockfish bot is equally likely to play either of those moves. You might think that you could have a list of dumb move types that the bot might play, but there are thousands of possible obviously dumb moves. This is a problem for machine learning.
mjburgess 3y ago
The 'world model' question seems "not even understood" by those in the field who provide these answers to it -- and use terms like "concepts" (see the linked paper on sentiment where the NN has apparently discovered a sentiment "concept"). Consider the world to contain causal properties which bring about regularities in text, eg., Alice likes chocolate so Alice says, "I like chocolate". Alice's liking, ie., her capacity for preference, desire, taste, asethetic juddgement etc is the cause of "like". Now these causal properties brings about significant regularities in text, so "like" occurring early in the paragraph comes to be extremely predictive of other text tokens occurring (eg., b-e-s-t, etc.) No one in this debate doubts, whatsoever, that NNs contain "subnetworks" which divide the problem up into detecting these token correlations. This is trivially observable in CNNs where it is trivial to demonstrate subnetworks "activating" on, say, an eye-shape. The issue is that when a competent language user judges someone's sentiment, or the implied sentiment the speaker of some text would have -- they are not using a model of how some subset of terms (like, etc.) comes to be predictive of others. They're using the fact that the know the relevant causal properties (liking, preference, desire, etc.) and how these cause certain linguistic phrases. It is for this reason a competent language user can trivially detect irony ("of course I like going to the dentist!" -- here since we know how unlikely it is to desire this, we know this phrase is unlikely to express such a preference, etc.). To say NNs, or any ML system, is sensitive to these mere correlations is not to say that these correlations are not formed by tracking the symptoms of real causes (eg., desire). Rather it is to say they do not track desire. This seems obvious, since the mechanism to train them is just sensitive to patterns in tokens. These patterns are not their causes, and are not models of their causes. They're only predictive of them under highly constrained circumstances. Astrological signs are predictive of birth dates, but they arent models of being born -- nor of time, or anything else. No one here doubts whether NNs are sensitive to patterns in text caused by causal properties -- the issue is that they arent models of these properties; they are models of (some of) their effects as encoded in text.
- gillesjacobs 3y ago
  To be fair, the term "world model" does not presume scientific understanding, factfullness or causality. In an ideal AI model this would be the aim though.
HL33tibCe7 3y ago
I’ve skimmed this, but if it is really true that it can play at 1800 ELO based purely on the moves, without seeing the board at each turn, that is insane. 1800 ELO is a strong human rating even with seeing the board. 1800 ELO essentially blindfolded is incredible
- ch3cksout 3y ago
  [dead]

120 points by homarp 3y ago

code here: https://github.com/adamkarvonen/chess_llm_interpretability https://github.com/adamkarvonen/chess_llm_interpretability

sinuhe69 3y ago

World model might be a too big word here. When we talk of a world model (in the context of AI motels), we refer to its understanding of the world, at least in the context we trained it. But what I see is just a visualization of the output in a fashion similar to a chess board. A stronger evidence would be a for example a map of the next move, which will show whether it truly understood the game’s rules. If it show probability larger than zero on illegal board fields, it will show us why it sometimes makes illegal moves. And obviously, it didn’t fully understand the rules of the game.

canjobear 3y ago
No, it is not a visualization of the output, it is a visualization of the information about pawn position contained in the model’s internal state.
mitthrowaway2 3y ago
> probability larger than zero Strictly speaking, it should be a mistake to assign a probability equal to zero to any moves, even for illegal board moves, but especially for an AI that learns by example and self-play. It never gets taught the rules, it only gets shown the games -- there's no reason that it should conclude that the probability of a rook moving diagonally is exactly zero just because it's never seen it happen in the data, and gets penalized in training every time it tries it. But even for a human, assigning probability of exactly zero is too strong. It would forbid any possibility that you misunderstand any rules, or forgot any special cases. It's a good idea to always maintain at least a small amount of epistemic humility that you might be mistaken about the rules, so that sufficiently overwhelmingly strong evidence could convince you that a move you thought was illegal turns out to be legal.
- redcobra762 3y ago
  There's got to be a probability cut-off, though. LLMs don't infinitely connect every token with every other token, some aren't connected at all, even if some association is taught, right?
  - the8472 3y ago
    The weights have finite precision which means they represent value-ranges / have error bars. So even if the weight is exactly 0 it does not represent complete confidence in it never occurring.
    - redcobra762 3y ago
      A weight necessitates a relationship, but I’m arguing LLMs don’t create all relationships. So a connection wouldn’t even exist.
      - yorwba 3y ago
        When relationships are represented implicitly by the magnitude of the dot product between two vectors, there's no particular advantage to not "creating" all relationships (i.e. enforcing orthogonality for "uncreated" relationships). On the contrary, by allowing vectors for unrelated concepts to be only almost orthogonal, it's possible to represent a much larger number of unrelated concepts. https://terrytao.wordpress.com/2013/07/18/a-cheap-version-of-the-kabatjanskii-levenstein-bound-for-almost-orthogonal-vectors/ https://terrytao.wordpress.com/2013/07/18/a-cheap-version-of... In machine learning, this phenomenon is known as polysemanticity or superposition https://transformer-circuits.pub/2022/toy_model/index.html https://transformer-circuits.pub/2022/toy_model/index.html
        redcobra762 3y ago
        That’s not right; there are many vectors that go unbuilt between unrelated tokens. Creating a ton of empty relationships would obviously generate an immense amount of useless data. Your links are not about actually orthogonal vectors, so they’re not relevant. Also that’s not what superposition is defined as in your own links: > In this paper, we use toy models — small ReLU networks trained on synthetic data with sparse input features — to investigate how and when models represent more features than they have dimensions. We call this phenomenon superposition

ericbarrett 3y ago

Thanks for linking the actual post—it was a great read. I'm not an ML expert, but the author really made it easy to follow their experiment's method and results.

triyambakam 3y ago

Is a linear probe part of observability/interpretability?

canjobear 3y ago
Yes, a pretty fundamental technique and one of the earliest. It lets you determine which layers contain what information among other things.
- Legend2440 3y ago
  The downside is that it's a supervised technique, so you need to already know what you're looking for. It would be nice to have an unsupervised tool that could list out all the things the network has learned.
  - JoshuaDavid 3y ago
    Anthropic has published some cool stuff in that direction: https://transformer-circuits.pub/2023/monosemantic-features https://transformer-circuits.pub/2023/monosemantic-features
    - firejake308 3y ago
      Whoa, this is super cool! I can imagine if we had something like this for ChatGPT, we could use it to do some serious prompt engineering. Imagine seeing what specific neurons you were activating with your prompt, and being able to identify which word in your prompt was triggering an undesired behavior. Super cool stuff, excited to see if it scales

sanxiyn 3y ago

I mean, this seems obvious to me. How would the model predict the next move WITHOUT calculating the board state first? Yes, by memorization, but memorization hypothesis is easily rejected by comparison to training dataset in this case. It is possible the model calculates an approximate board state, which is different from the board state but equivalent for most games, but not all games. It would be interesting to train adversarial policy to check this. From KataGo attack we know this does happen for Go AIs: Go rules have a concept of liberty, but so called pseudoliberty is easier to calculate and equivalent for most cases (but not all cases). In fact, human programmers also used pseudoliberty to optimize their engines. Adversarial attack found Go AIs also use pseudoliberty.

taneq 3y ago
It’s one thing to think it’s obvious, but quite another to prove it. I think this is the true value of this kind of work, is that it’s helping to decipher what these models are actually doing. Far too often we hear “NNs / LLMs are black boxes” as if that’s the end of the conversation.
Sharlin 3y ago
Surprisingly many people seem to believe LLMs cannot form any deeper world models beyond superficial relationships between words, even if figuring out a "hidden" model allows for a big leap in prediction performance – in this case, a hypothesis corresponding to chess rules happens to be give the best bang for the buck for predicting strings that have chess notation structure. But the model could in principle just have learned a long list of rote heuristics that happen to predict notation strings well, without having made the inferential leap to a much simpler set of rules, and a learner weaker than a LLM could well have got stuck at that stage. I wonder how well a human (or a group of humans) would fare at the same task and if they could also successfully reconstruct chess even if they had no prior knowledge of chess rules or notation. (OTOH a GPT3+ level LLM certainly does know that chess notation is related to something called "chess", which is a "game" and has certain "rules", but to what extent is it able to actually utilize that information?)
HarHarVeryFunny 3y ago
> It is possible the model calculates an approximate board state Yes - this is exactly what the probes show. One interesting aspect is that it still learns to play when trained on blocks of move sequences starting from the MIDDLE of the game, so it seems it must be incrementally inferring the board state by what's being played rather than just by tracking the moves.

wavemode 3y ago

If you take a neural network that already knows the basic rules of chess and train it on chess games, you produce a chess engine. From the Wikipedia page on one of the strongest ever[1]: "Like Leela Zero and AlphaGo Zero, Leela Chess Zero starts with no intrinsic chess-specific knowledge other than the basic rules of the game. Leela Chess Zero then learns how to play chess by reinforcement learning from repeated self-play" [1]: https://en.wikipedia.org/wiki/Leela_Chess_Zero https://en.wikipedia.org/wiki/Leela_Chess_Zero

btown 3y ago
As described in the OP's blog post https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c... - one of the incredible things here is that the standard GPT architecture, trained from scratch from PGN strings alone, can intuit the rules of the game from those examples, without any notion of the rules of chess or even that it is playing a game. Leela, by contrast, requires a specialized structure of iterative tree searching to generate move recommendations: https://lczero.org/dev/wiki/technical-explanation-of-leela-chess-zero/ https://lczero.org/dev/wiki/technical-explanation-of-leela-c... Which is not to diminish the work of the Leela team at all! But I find it fascinating that an unmodified GPT architecture can build up internal neural representations that correspond closely to board states, despite not having been designed for that task. As they say, attention may indeed be all you need.
- goatlover 3y ago
  What's the strength of play for the GPT architecture? It's impressive that it figures out the rules, but does it play strong chess? >> As they say, attention may indeed be all you need. I don't think drawing general conclusions about intelligence from a board game is warranted. We didn't evolve to play chess or Go.
  - jksk61 3y ago
    > What's the strength of play for the GPT architecture? It's impressive that it figures out the rules, but does it play strong chess? sometimes it is not a matter of "is it better? is it larger? is it more efficient?", but just a question. mountains are mountains, men are men.

MenhirMike 3y ago

I hope it performs better than ChatGPT: https://old.reddit.com/r/AnarchyChess/comments/10ydnbb/i_placed_stockfish_white_against_chatgpt_black/ https://old.reddit.com/r/AnarchyChess/comments/10ydnbb/i_pla... Though I will give it to ChatGPT, castling across the bishop was a genius move.

silasdavis 3y ago
I've definitely played against some children who would respawn pieces. Human level AI clearly.

Imnimo 3y ago

I'd be curious to see if, in the 1-2% of cases where the linear probe fails to predict board occupancy, the LLM also predicts (or at least assigns non-trivial probability to) a corresponding illegal move. For example, if the linear probe incorrectly thinks there's a bishop on b4, does the LLM give more probability to illegal bishop moves along the corresponding diagonals than to other illegal bishop moves?

tromp 3y ago

Would love to see a similar experiment for 9x9 Go, where the model also needs to learn the concepts of connected group and its liberties.

tromp 3y ago

I'm impressed and surprised that a relatively small model can learn so much from just the textual move records (not even full algebraic notation 1.e2-e4 e7-e5 2.Ng1-f3 but just 1.e4 e5 2.Nf3). Would love to see a similar experiment for 9x9 Go, where the model also needs to learn the concepts of connected group and its liberties.

tromp 3y ago

I'm impressed and surprised that a relatively small model can learn so much from just the textual move records. Not even full algebraic notation 1.e2-e4 e7-e5 2.Ng1-f3 but just 1.e4 e5 2.Nf3 ... It has to figure out what o-o and o-o-o mean just from what King and Rook moves appear later. And the only way it could learn that the King cannot move through a checked square while castling is that such situations never appear in its training set. Would love to see a similar experiment for 9x9 Go, where the model also needs to learn the concepts of connected group and its liberties.

kybernetikos 3y ago

> I fine-tuned GPT-2 on a 50 / 50 mix of OpenWebText and chess games, and it learned to play chess and continued to output plausible looking text. Maybe there’s something interesting to look at there? To me that suggests investigating whether there are aspects of human culture that can improve chess playing performance - i.e. whether just training on games produces less good results than training on games and literature. This seems plausible to me, even beyond literature that is explictly about the game - learning go proverbs, which are often phrased as life advice is a part of learning go, and games are embedded all through our culture, with some stories really illustrating that you have to 'know when to hold em, know when to fold em, know when to walk away, know when to run'.

emporas 3y ago

Nice experiment, even though we know that LLMs distill an internal world model representation of whatever they are trained on. The experiment could be a little better by using a more descriptive form of notation than PGN. PGN notation's strength is the shorthand properties of it, because it is used by humans while playing the game. That is far from being a strength as LLM training data. ML algorithms, and LLMs are trained better by feeding them more descriptive and accurate data, and verbosity is not a problem at all. There is the FEN notation in which in every move the entire board is encoded. One could easily imagine many different ways to describe a game, like encoding vertical and horizontal lines, listing what exact squares each piece is covering, what color squares, which of the pieces are able to move, and in each move generate one whole page of the board situation. I call this spatial navigation, in which the LLM learns the ins and outs of it's training data and it is able to make more informed guesses. Chess is fun and all, but code generation has the potential to be a lot better than just writing functions. By feeding the LLM the AST representation of the code, the tree of workspace files, public items, module hierarchy alongside with the code, it could be a significant improvement.

anotherpaulg 3y ago
By feeding the LLM the AST representation of the code, the tree of workspace files, public items, module hierarchy alongside with the code, it could be a significant improvement. Aider does this, using tree-sitter to build a “repository map”. This helps the LLM understand the overall code base and how it relates to the specific coding task at hand. https://aider.chat/docs/repomap.html https://aider.chat/docs/repomap.html More broadly, I agree with your sentiment that there is a lot of value in considering the best ways to structure the data we share with LLMs. Especially in the context of coding.
- emporas 3y ago
  >Aider does this, using tree-sitter to build a “repository map”. This helps the LLM understand the overall code base and how it relates to the specific coding task at hand. Great stuff. >More broadly, I agree with your sentiment that there is a lot of value in considering the best ways to structure the data we share with LLMs. Especially in the context of coding. As the experiments on PHI-1 and PHI-2 from microsoft show, training data make a difference. The "textbooks is all you need" moto means better structured data, more clear data make a difference. https://arxiv.org/abs/2306.11644 https://arxiv.org/abs/2306.11644

aaron695 3y ago

[dead]

sjducb 3y ago

I’m curious how human like this LLM feels when you play it. One of the challenges to making fun chess bots is to make it play like a low or mid ranked human. The problem is that a stockfish based bot knows some very strong moves, but deliberately plays bad moves so it’s about the right skill level. The problem is that these bad moves are often very obvious. For example I’ll threaten a queen capture. Any human would see it and move their queen. The bot “blunders” and loses the queen to an obvious attack. It feels like the bot is letting you win which kills the enjoyment of playing with the bot. I think that this approach would create very human like games.

Solvency 3y ago
Isn't that more of a design issue than a bot AI issue?
- HarHarVeryFunny 3y ago
  I'd call it an approach issue: LLM vs brute-force lookahead. An LLM is predicting what comes next per it's training set. If it's trained on human games then it should play like a human; if it's trained on Stockfish games, then it should play more like Stockfish. Stockfish, or any chess engine using brute force lookahead, is just trying to find the optimal move - not copying any style of play - and it's moves are therefore sometimes going to look very un-human. Imagine if the human player is looking 10-15 moves ahead, but Stockfish 40-50 moves ahead... what looks good 40-50 moves out might be quite different than what looks good to the human.
- sjducb 3y ago
  Humans and machines find good moves in different ways. Most humans have fast pattern matching that is quite good at finding some reasonable moves. There are also classes of moves that all humans will spot. (You just moved your bishop, now it’s pointing at my queen) The problem is that stockfish scores all moves with a number based on how good the move is. You have no idea if a human would agree. For example mis-calculating a series of trades 4 moves deep is a very human mistake, but it’s scored the same as moving the bishop to a square where it can easily be taken by a pawn. They both result in you being a bishop down. A nerfed stockfish bot is equally likely to play either of those moves. You might think that you could have a list of dumb move types that the bot might play, but there are thousands of possible obviously dumb moves. This is a problem for machine learning.

mjburgess 3y ago

The 'world model' question seems "not even understood" by those in the field who provide these answers to it -- and use terms like "concepts" (see the linked paper on sentiment where the NN has apparently discovered a sentiment "concept"). Consider the world to contain causal properties which bring about regularities in text, eg., Alice likes chocolate so Alice says, "I like chocolate". Alice's liking, ie., her capacity for preference, desire, taste, asethetic juddgement etc is the cause of "like". Now these causal properties brings about significant regularities in text, so "like" occurring early in the paragraph comes to be extremely predictive of other text tokens occurring (eg., b-e-s-t, etc.) No one in this debate doubts, whatsoever, that NNs contain "subnetworks" which divide the problem up into detecting these token correlations. This is trivially observable in CNNs where it is trivial to demonstrate subnetworks "activating" on, say, an eye-shape. The issue is that when a competent language user judges someone's sentiment, or the implied sentiment the speaker of some text would have -- they are not using a model of how some subset of terms (like, etc.) comes to be predictive of others. They're using the fact that the know the relevant causal properties (liking, preference, desire, etc.) and how these cause certain linguistic phrases. It is for this reason a competent language user can trivially detect irony ("of course I like going to the dentist!" -- here since we know how unlikely it is to desire this, we know this phrase is unlikely to express such a preference, etc.). To say NNs, or any ML system, is sensitive to these mere correlations is not to say that these correlations are not formed by tracking the symptoms of real causes (eg., desire). Rather it is to say they do not track desire. This seems obvious, since the mechanism to train them is just sensitive to patterns in tokens. These patterns are not their causes, and are not models of their causes. They're only predictive of them under highly constrained circumstances. Astrological signs are predictive of birth dates, but they arent models of being born -- nor of time, or anything else. No one here doubts whether NNs are sensitive to patterns in text caused by causal properties -- the issue is that they arent models of these properties; they are models of (some of) their effects as encoded in text.

gillesjacobs 3y ago
To be fair, the term "world model" does not presume scientific understanding, factfullness or causality. In an ideal AI model this would be the aim though.

HL33tibCe7 3y ago

I’ve skimmed this, but if it is really true that it can play at 1800 ELO based purely on the moves, without seeing the board at each turn, that is insane. 1800 ELO is a strong human rating even with seeing the board. 1800 ELO essentially blindfolded is incredible

ch3cksout 3y ago
[dead]