4 ms·

Nobody who is 1400 plays outright illegal moves.

by WoodenChair 4y ago

SanderNL 4y ago
Humans don’t, but that is an unnecessary bar IMO.
- WoodenChair 4y ago
  > Humans don’t, but that is an unnecessary bar IMO. People are always telling me that I'm moving the goalposts when I challenge the hyperbole about LLMs. But now you're moving the goalposts about chess. Not playing illegal moves is a pre-requisite for any strong understanding of how to play chess. That is definitely the goal post.
  - scj 4y ago
    If a two year old had an ELO of 1400 but sometimes took a nap halfway through a game, it'd be a prodigy. It's not like an AI making silly mistakes when driving a car.
mrbungie 4y ago
Yet it achieves 1400. Add hard rules to stop it spewing out said moves and you have a 1400 ELO Player (most UIs won't even let you make illegal moves). It is difficult to say that is not impressive due to it being an emergent ability.
- WoodenChair 4y ago
  > It is difficult to say that is not impressive due to it being an emergent ability. I don't know why you think it's an emergent ability. It's seeing a sequence of moves, and playing the most likely next move (i.e. the most likely next token) given the previous complete move sequences it was trained on. That's the baseline of what an LLM does—not something emergent. Games in online chess databases tend to be of relatively good players. Nobody wants to look up games played by two 800 ELO players. As an aside, there have been chess programs for years that show you for a given position all of the previous games in its database with the same position and the win outcome % of each move. That's all that's going on here.
  - oezi 4y ago
    > here have been chess programs for years that show you for a given position all of the previous games in its database with the same position and the win outcome % of each move. That's all that's going on here. It could be, but would you think that of the 100-300 bn parameters in the model a lot are dedicated to chess move sequences? It seems likely that it has seen such data, but I would be surprised if it is using a considerable chunk to store chess database information.
    - Jensson 4y ago
      The web has millions of grandmaster chess games and probably billions of chess games overall. So I wouldn't be surprised if it has like 0.01% to chess games since there are so many, if so that would mean it has 10-30 million parameters to play chess with, for comparison stockfish has 10 million parameters to its chess engine.
  - mrbungie 4y ago
    Because I don't think that the model learned the literal memorization of chess moves. It must've at least compressed said information in some way way. And since the model is not biased to play chess on its structure nor sampling policy, I think it's fair to consider it an emergent ability. Chess moves are a tiny/diminute part of all text learned by the model. This memorization argument is very similar to the "Stable Diffusion just takes bits of the images in the original dataset and parches them together".
vanviegen 4y ago
Does that still hold when the player doesn't have a board in front of them, but just a list of previous moves?
mtlmtlmtlmtl 4y ago
I have seen 1400 FIDE elo(much much stronger than chess.com 1400) players make illegal moves in real life long time control tournament games after thinking for over 10 minutes. People regularly create an erroneous board state in their head while thinking. This is often how blunders happen; merging together a possible future board state with the current one. Illegal moves are more rare because you have to then also miss that the move is illegal while physically making it. But they do happen. 1400s on chess.com never play illegal moves. 300s on chess.com never play illegal moves. Because it's impossible to do. In the real world, even grandmasters can make illegal moves, though they almost always have to be under time pressure. This idea that the illegal moves completely invalidate this result is just ill-conceived. On the other hand I do agree this is mostly returning common sequences of moves. And if you actually analyse the games, especially the ones with illegal moves, you'll find plenty of qualitative evidence of that. But I'm fed up of doing people's thinking for them for today, so this is peace out for me today. See my others comments on this post to see a more detailed analysis of what this is doing.

Nobody who is 1400 plays outright illegal moves.

SanderNL 4y ago

Humans don’t, but that is an unnecessary bar IMO.

WoodenChair 4y ago
> Humans don’t, but that is an unnecessary bar IMO. People are always telling me that I'm moving the goalposts when I challenge the hyperbole about LLMs. But now you're moving the goalposts about chess. Not playing illegal moves is a pre-requisite for any strong understanding of how to play chess. That is definitely the goal post.
- scj 4y ago
  If a two year old had an ELO of 1400 but sometimes took a nap halfway through a game, it'd be a prodigy. It's not like an AI making silly mistakes when driving a car.

mrbungie 4y ago

Yet it achieves 1400. Add hard rules to stop it spewing out said moves and you have a 1400 ELO Player (most UIs won't even let you make illegal moves). It is difficult to say that is not impressive due to it being an emergent ability.

WoodenChair 4y ago
> It is difficult to say that is not impressive due to it being an emergent ability. I don't know why you think it's an emergent ability. It's seeing a sequence of moves, and playing the most likely next move (i.e. the most likely next token) given the previous complete move sequences it was trained on. That's the baseline of what an LLM does—not something emergent. Games in online chess databases tend to be of relatively good players. Nobody wants to look up games played by two 800 ELO players. As an aside, there have been chess programs for years that show you for a given position all of the previous games in its database with the same position and the win outcome % of each move. That's all that's going on here.
- oezi 4y ago
  > here have been chess programs for years that show you for a given position all of the previous games in its database with the same position and the win outcome % of each move. That's all that's going on here. It could be, but would you think that of the 100-300 bn parameters in the model a lot are dedicated to chess move sequences? It seems likely that it has seen such data, but I would be surprised if it is using a considerable chunk to store chess database information.
  - Jensson 4y ago
    The web has millions of grandmaster chess games and probably billions of chess games overall. So I wouldn't be surprised if it has like 0.01% to chess games since there are so many, if so that would mean it has 10-30 million parameters to play chess with, for comparison stockfish has 10 million parameters to its chess engine.
- mrbungie 4y ago
  Because I don't think that the model learned the literal memorization of chess moves. It must've at least compressed said information in some way way. And since the model is not biased to play chess on its structure nor sampling policy, I think it's fair to consider it an emergent ability. Chess moves are a tiny/diminute part of all text learned by the model. This memorization argument is very similar to the "Stable Diffusion just takes bits of the images in the original dataset and parches them together".

vanviegen 4y ago

Does that still hold when the player doesn't have a board in front of them, but just a list of previous moves?

mtlmtlmtlmtl 4y ago

I have seen 1400 FIDE elo(much much stronger than chess.com 1400) players make illegal moves in real life long time control tournament games after thinking for over 10 minutes. People regularly create an erroneous board state in their head while thinking. This is often how blunders happen; merging together a possible future board state with the current one. Illegal moves are more rare because you have to then also miss that the move is illegal while physically making it. But they do happen. 1400s on chess.com never play illegal moves. 300s on chess.com never play illegal moves. Because it's impossible to do. In the real world, even grandmasters can make illegal moves, though they almost always have to be under time pressure. This idea that the illegal moves completely invalidate this result is just ill-conceived. On the other hand I do agree this is mostly returning common sequences of moves. And if you actually analyse the games, especially the ones with illegal moves, you'll find plenty of qualitative evidence of that. But I'm fed up of doing people's thinking for them for today, so this is peace out for me today. See my others comments on this post to see a more detailed analysis of what this is doing.