5 ms·
> These people used bad prompts and came to the conclusion that ChatGPT can’t play a legal chess game. (…) > With this prompt ChatGPT almost always plays fully
by latexr 4y ago
> These people used bad prompts and came to the conclusion that ChatGPT can’t play a legal chess game. (…)
> With this prompt ChatGPT almost always plays fully legal games.
> Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table (…)
> (…) with GPT4 (…) in the two games I attempted, it made numerous illegal moves.
So you’ve ostensibly¹ found a way to reduce the error rate and then deliberately ignored the cases where it failed. In short: it may play valid chess under certain conditions but can’t be trusted to do so. That doesn’t contradict previous findings.
¹ 19 games is a small sample and the supposedly more advanced system failed in your tries.
- ballenf 4y agoThe illegal moves were counted as losses/resignations, not ignored.
- psychphysic 4y agoThat's how one uses any tool.
- qwytw 4y agoThe behavior of pretty much every other tool is much easier to interpret though.
- kdmccormick 4y agoIf the title of the article was: > A trivial wrapper around ChatGPT has a Chess Elo of 1400 would you have any issue? Afaict, the thesis is the article is not "ChatGPT is the ideal tool for playing AI chess", but "it is interesting how well ChatGPT can play chess with some very simple tweaks."
- Out_of_Characte 4y agoYes, but it also completely invalidates the measurement of a 1400 elo rating. By comparison, any player making an illegal move is forfeiting the game, almost all people from ~300 elo can play without making illegal moves, chatgpt cant.
- renewiltord 4y agoThe article: > So whenever it wanted to make an illegal move, it resigned. You: > By comparison, any player making an illegal move is forfeiting the game... By comparison indeed.
- sebzim4500 4y agoChatGPT did forfeit whenever it made an illegal move, read the article.
- swatcoder 4y agoNo, the writer arbitrarily decided to interpret illegal moves as resignations in order to support the conclusion they wanted. That's very different and grossly unscientific.
- epups 4y agoThis is not a scientific paper, and I at least find this decision justified, as he could have been more lenient and grab headlines with a bigger ELO.
- mynameisvlad 4y agoI mean, that's more lenient than the official "interpretation" (rule) which is that your second illegal move results in a forfeit.
- hgsgm 4y agoAll this above, and people are claiming that ChatGPT lacks human level comprehension of the text it consumes. In Chess.com, you absolutely can attempt an illegal move, and many players do, and you will not get punished for it, so chatgpt is better then a 1400 human player.
- nsxwolf 4y agoWhy do illegal moves forfeit? In online play, they're validated. You can't make illegal moves. What's the ELO score if ChatGPT is corrected, and chooses a new move?
- dudeinjapan 4y agoObviously the article should be taken with a giant grain of salt. That being said, not many things what aren't designed to play chess can play chess, with or without coaxing. My dog cannot, for instance, nor can my coffee table.
- hectorlorenzo 4y ago> My dog cannot, for instance, nor can my coffee table. You must be giving them the wrong prompts.
- ogogmad 4y ago[redacted]
- AndrewPGameDev 4y agoIt's a joke
- __s 4y ago> So whenever it wanted to make an illegal move, it resigned. Doesn't sound like ignoring the cases where it failed
- vidarh 4y agoFuller context from the article: > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table and saying “this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” So whenever it wanted to make an illegal move, it resigned. (my emphasis) So the illegal moves are at least part of the reasons for the 6 losses, and factored into the rating. Quickly scanning the game, it seems 3 of the losses ended in checkmate, so that leaves 3 illegal moves in 19 games. Could be better, but for a system not intentionally built to play chess, it's pretty decent.
- jmull 4y agoThe article also says in one game chatgpt when crazy so they continued the game with a fresh chat. That probably should have been counted a resignation loss too.
- swatcoder 4y agoNo ELO 1400 player will have that rate of illegal moves, so saying it that it plays with an ELO 1400 rating is disingenuous. Reinterpreting illegal moves as resignation is absurd when an LLM is formally capable of expressing statements "I resign" or "I cannot conceive of a winning move from here" just as well as any human player. It just doesn't do so because it's not actually playing chess the way we think of an ELO 1400 player playing chess.
- vidarh 4y agoThat no human ELO 1400 player will have that rate of illegal moves may be true, but if anything treating the very first illegal move as forfeit appears to be stricter than most rules
- JellyBeanThief 4y agoFine, just wrap the LLM in a simple function that detects illegal moves and replaces them with "I resign" or "I cannot conceive of a winning move from here". Then you aren't "reinterpreting" anymore. My point is, it sounds like Elo doesn't measure what we want it to measure. If we care about the way an agent wins a game and not just whether it wins a game, then we need an instrument that measures strategy, not outcome.
- hgsgm 4y agoIs this the top comment (and not even grey) because more people failed to read the article than read it?
- sebzim4500 4y agoIt does seem that way.
- whimsicalism 4y agoA baffling thread. They quoted the article, so clearly they read it... but not very well?
- Waterluvian 4y agoI’m going to float something ridiculous: An illegal move is a valid play. You might not get caught. I think there are some Magnus games where illegal moves went overlooked and impacted the game. You could interpret this as “ChatGPT wants to cheat sometimes.” But I personally interpret it as “ChatGPT doesn’t understand what it’s doing. It’s just a really really good simulacrum.”
- whimsicalism 4y agoI'm confused. If you read the article, you know that you are wrong - but you are quoting the article?