3 ms·
Fine, just wrap the LLM in a simple function that detects illegal moves and replaces them with "I resign" or "I cannot conceive of a winning move from here". Th
by JellyBeanThief 4y ago
Fine, just wrap the LLM in a simple function that detects illegal moves and replaces them with "I resign" or "I cannot conceive of a winning move from here". Then you aren't "reinterpreting" anymore.
My point is, it sounds like Elo doesn't measure what we want it to measure. If we care about the way an agent wins a game and not just whether it wins a game, then we need an instrument that measures strategy, not outcome.
- illiarian 4y ago> Fine, just wrap the LLM in a simple function that detects illegal moves and replaces them with "I resign" or "I cannot conceive of a winning move from here". Then you aren't "reinterpreting" anymore. Then it's still isn't anywhere near ELO 1400.
- deleted 4y ago[deleted]
- vidarh 4y agoUnder FIDE rules it's first a forfeit after the second illegal move, so if anything it would seem that the interpretation used by the article author underestimates its ELO ranking.
- illiarian 4y agoNope, still not even close to what the author claims. If I understand it correctly, it made illegal moves in 3 out of 19 games. That's probably a few orders of magnitude more illegal moves than even a 1400 ELO player would make of their entire lifetime.
- ogogmad 4y agoThere's the "it" which has no post-processing, and there's the "it" where the output is post-processed to announce a resignation when it attempts an illegal move. Some things about the two "it"s: - They differ trivially. - They enable new capabilities, such as the ability to explain why a move got made. Current chess AIs are not good at this. So I think you're making too much of a big deal from a comparative triviality. [edit] We might be talking past each other. And some people above have come to doubt the article's results even with the right prompt engineering.
- vidarh 4y agoThe ranking takes into account wins and losses, not illegal moves, and so the fact that it plays in a way where a higher proportion of its losses is down to illegal moves than a human player is not relevant to its ranking. It may suggest that the ranking ought to take that into account, but that's a separate issue.
- pedrosorio 4y agoRepeating what others have said in this thread: The author claims: chatGPT has a 1400 chess ELO based on games played. You appear to think author claims: chatGPT plays chess like a human rated 1400. Your observations do not contradict the authors’ claim that based on games won and lost against opponents of a specific strength, the estimated ELO is 1400. A non-human player can make illegal moves at a much higher rate and make up for that by being stronger when it does not make illegal moves to achieve the same rating as a human player who plays the game in a completely different way.