3 ms·
What if you encoded the whole game state into a one-shot completion that fits into the context window every turn? It would likely not make those illegal moves.
by vczf 3y ago
What if you encoded the whole game state into a one-shot completion that fits into the context window every turn? It would likely not make those illegal moves. I suspect it's an artifact of the context window management that is designed to summarize lengthy chat conversations, rather than an actual limitation of GPT4's internal model of chess.
- actionfromafar 3y agoI am sorry, but I thought it was a bold assumption it has an internal model of chess?
- vidarh 3y agoHaving an internal model of chess and maintaining an internal model of the game state of a specific given game when it's unable to see the board are two very different things. EDIT: On re-reading I think I misunderstood you. No, I don't think it's a bold assumption to think it has an internal model of it at all. It may not be a sophisticated model, but it's fairly clear that LLM training builds world models.
- PoignardAzur 3y agoNot that bold, given the results from OthelloGPT. We know with reasonable certainty that an LLM fed on enough chess games will eventually develop an internal chess model. The only question is whether GPT4 got that far.
- tedajax 3y agoDoesn't really seem like an internal chess model if it's still probabalistic in nature. Seems like it could still produce illegal moves.
- vidarh 3y agoSo can humans. And nothing stops probabilities in a probabilistic model from approaching or reaching 0 or 1 unless your architecture explicitly prevents that.
- baq 3y agoWhy? Or, given https://thegradient.pub/othello/ https://thegradient.pub/othello/, why wouldn't it have an internal model of chess? It probably saw more than enough example games and quite a few chess books during training.