For a fun exercise, I tried Turing's chess problem[1] from "Computing Machinery and Intelligence"[2] on GPT-4. It recognized that it was a chess problem but act

For a fun exercise, I tried Turing's chess problem[1] from "Computing Machinery and Intelligence"[2] on GPT-4. It recognized that it was a chess problem but actually failed to solve it in a really interesting way. The reason (I think) it failed to find the mate is that Turing gives the problem in a form of relative notation that noone would use any more, with the coordinates of the pieces given relative to the player. Nowdays everyone uses a form of notation which describes the squares the same for the player with black as white. Even if they didn't use algebraic notation (which everyone uses now), anyone giving the problem today would call the position of the black King K3 and the rook R8. When I prompt GPT4 using modern notation it realises it got the pieces positions wrong but still makes the same mistake of missing the very obvious mate, having got itself stuck. [3] I also tried putting the Black king in a different position where Rh1 would not be checkmate and was able to convince gpt4 that it was mate. All of this makes me certain it doesn't actually have a model of the game - it really just simulates a model of the game by text reconstruction, which is exactly what you would think. [1] "I have K at my K1, and no other pieces. You have only K at K6 and R at R1. It is your move, what do you play?" [2] https://redirect.cs.umbc.edu/courses/471/papers/turing.pdf https://redirect.cs.umbc.edu/courses/471/papers/turing.pdf [3] Transcript here https://chat.openai.com/share/8426389d-bbf1-4a52-b6b1-3f531142ed90 https://chat.openai.com/share/8426389d-bbf1-4a52-b6b1-3f5311...