3 ms·
Preventing an LLM from making illegal moves should be very simple: provide it with tool access to something that tells it if a move is legal or not, then watch
by simonw 2y ago
Preventing an LLM from making illegal moves should be very simple: provide it with tool access to something that tells it if a move is legal or not, then watch it iterate in a loop until it finds a move that it is allowed to make.
I expect this would dramatically improve the chess playing abilities of the competent tool using models, such as O3.
- toolslive 2y agoor just present it with the list of legal moves and force it to pick from said list.
- simonw 2y agoI imagine there are points in a chess game, especially early on, where that list could have hundreds of moves - could use up a fair amount of tokens.
- toolslive 2y agoNope. The list is very limited. For the starting position: a3, a4, b3,b4,.......h3, h4, Na3, Nc3, Nf3, Nh3 That's 20 moves. the size grows a bit in the early middle game, but then drops again in the endgame. There do exist rather artificial positions with more than 200 legal moves, but the average number of legal moves in a position is around 40.
- stavros 2y agoYou could also constrain the output grammar to legal moves, but if we're comparing its chess performance to humans', it would be unfair to not let it think.
- simonw 2y agoHuh, that's really interesting, thanks. I mentally counted the starting moves as being 8 pawns x2 = 16 pawn moves and 2x2 =4 4 knight moves, but then I doubled it for both sides to get 40 (which with hindsight was obviously wrong) and then assumed that once the pawns had moved a bit there would be more options from non-pawn pieces. With an upper bound of ~200 in edge cases listing all possible moves wouldn't take up much room in the context at all. I wonder if it would give better results, too.
- bluefirebrand 2y agoAt any given time there are way more illegal moves than legal moves, though