What it surprised me the most is that according to the author the model do not make illegal moves meaning it has developed a pretty good model about the game (i

What it surprised me the most is that according to the author the model do not make illegal moves meaning it has developed a pretty good model about the game (it's not given any representation about the board during the game). Even though GPT models should not really be that good to model the game of chess in general as any new tokens starts from zero and it needs to retrieve all information it needs, and the state of the board cannot be "saved" at the same layer because after the token has retrived it it would need to add the new semi-move on top and it's already one layer more deep, RNN seems they would be better at this as the cells take the h vector as input, add new information on it and pass it to the next tokens at the same layer, so it would be perfect for saving the state of the game; of course this would not allow parallel training like transformer models, this is the tradeoff.