Given many of the shortcomings that still exist with ChatGPT-4, especially some of the comments here showing it can't even play Tic Tac Toe, I suspect one of tw

Given many of the shortcomings that still exist with ChatGPT-4, especially some of the comments here showing it can't even play Tic Tac Toe, I suspect one of two things. I'm sure OpenAI collected far more training data that included chess moves in the data set for GPT-4, but I suspect they may have further tuned and optimized GPT-4 for playing chess as well. I also suspect maybe there's something about chess algebraic notation that works unusually well with LLMs, while there may be no equivalent for a game like Tic Tac Toe. Clearly, GPT-4 knows opening sequences extremely well, as it's seen countless examples of each. GPT-3 memorized optimal strategies for the first 10-20 moves, and GPT-4 likely memorized many moves deeper than that. To truly test GPT-4s ability to play chess, we need to test it with highly unusual opening positions, opening moves that, very likely, were not in its training data. Then we can see how well GPT-4 truly understands the rules and strategies of chess, and how much of it is just memorization.