I just opened a random recent chess game on lichess ( https://lichess.org/YpxTUUbO/white#88 https://lichess.org/YpxTUUbO/white#88 ) . I'm pretty sure ChatGPT ca

I just opened a random recent chess game on lichess ( https://lichess.org/YpxTUUbO/white#88 https://lichess.org/YpxTUUbO/white#88 ) . I'm pretty sure ChatGPT can't be trained on games that were just played, so this ensures the game is not in its training data. I gave the position before checkmate to ChatGPT to see if it would produce the checkmating move. It played an illegal move, replying with "Be5#" even there's no bishop of either color in the position. Unfortunately I'm rate limited at the moment so I can't try other games, but this looks like a solid method to evaluate how often ChatGPT plays legal / good moves.