15 ms·
I played chess against ChatGPT-4 and lost
- unnouinceput 4y agoQuote: "GPT-4 will change the world" No it won't. Neither 5 nor 6 that I suspect will be in maximum 2 years out. But after that whatever comes next (7th/8th) definitely will.
- ChatGTP 4y agoI actually think it might get so “smart” we will no longer ever understand it, and have no way to verify it’s accuracy because we’ll have nothing to benchmark it by. Will be great.
- hackerlight 4y agoI expect this one day, but only after there are more methodology breakthroughs. I am doubtful we get there by just scaling up current methods. What I predict is AI will stagnate at the upper-end of human-level intelligence except in narrow domains where we can simulate unlimited data which allows for self-play learning, such as in Go. But such domains are limited. For everything else, in the absence of unlimited data (generated via self-exploration or self-play), the best AI can do is learn from human output, which carries with it all the limitations and flaws of human cognition.
- zarzavat 4y agoTwo of the most interesting domains (mathematics and programming) do have external forms of validation. In mathematics you can use a proof assistant like Lean. In programming there’s types, tests and formal verification. Where the AI will struggle is anything where forward progress requires a lab. In this case it will not provide as much acceleration, as it still has to move at the speed of the human in the lab doing the experiment for it. It can maybe make better decisions than the human and get a small speedup, but it won’t be anywhere close to the possible speedup for formal sciences. And the AI will really struggle with anything that involves mechanical sympathy gained through practice. No amount of book knowledge or raw intelligence can teach an AI how to play the violin. For that, you need a violin, a hand, and a feedback loop. Indeed there’s probably many skills that humans have near optimal learning rate, because the bottleneck is in the physical environment rather than our brains.
- oezi 4y ago> I am doubtful we get there by just scaling up current methods. Why? The last two years show that throwing more compute at this during training and execution is clearly helping. LLMs currently lack access to real world data for instance to measure a physical quantity or perform a randomized control trial, but given how much of human output is just manipulation of language tokens and that we got quite a lot of sample data to feed into the networks I am confident there are still some steps ahead of us that are essentially just more of the same. An important data point is also the limited amount of data that we need to feed into a human brain to get somewhere smart.
- haldujai 4y agoHave they? I would say the last year has shown “throwing more compute” is not the endgame with Chinchilla, Galactica and LLaMA being some prominent papers raising questions. Despite all of the (presumed) compute thrown at GPT4 hallucination is still a massive problem (in their technical paper) and “more RL” isn’t/won’t solve that. It’s hard to believe that a certain number of flops will fix the fact that next token prediction is inherently not factually grounded and can result in very inaccurate outputs. I think we’re plateauing with current model architectures and overestimating LLMs because their output sounds so much better, chess is a great example. If LLMs of this size can’t come close to stockfish which can run on my 10 year old CPU, how much knowledge is all this compute really encoding?
- oezi 4y agoI don't interpret results of Chinchilla et al. as there are fundamental limits to what LLMs can do. From the Chinchilla paper's abstract: "we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally" > GPT4 hallucinations My interpretation is that we are just seeing Dunning-Kruger. We are at the overconfidence stage, but quickly descending into the Valley of Despair ("I am an AI and can't..."). My reading of the report is that hallucinations are already reduced a lot. > is inherently not factually grounded Aren't we all just symbol manipulation machines? Isn't truth just created by assigning symbols? Is there more value in experiencing an apple falling on your head than reading on an apple falling onto Newton's head? > If LLMs of this size can’t come close to stockfish which can run on my 10 year old CPU, how much knowledge is all this compute really encoding? If we assume that GPT-4 is maybe a bit bigger than GTP-3 then it clocks in at around 1000 GB = 1 TB. A large chess data base is something like 5GB, but I would assume that training maybe keeps a fraction of that, maybe 100 MB to encode chess moves. I think it is quite an achievement to that GPT-4 can play at all, but by reading moves from textfiles online.
- rhdunn 4y agoIt would be interesting to see how these LLMs work in a REPL environment. The model could take a (language, version, compiler, sourceText) tuple as input, and have the model generate the same. It could also be fed the compiler output in a (warningCount, errorCount, stdoutText, stderrText) tuple, so it can learn to produce error-free output. It would also be interesting to feed it language concepts ("if statement", "while loop", etc.) in the form of (language, version, concept, sourceText), to see if it can form an abstract understanding of those concepts, and successfully port code from one language to another.
- fightglobalism 4y agoI'm new to HN and new to ChatGPT-4 concepts but it's becoming impossible for me not to form an interest. How does ChatGPT-4 interface with something like playing chess? Do you need to write an implementation or prototype to extend it's capabilities or is the AI able to exchange strings based on a language definition? If the latter is it limited to strings-only since it's a "Chat" bot or could you show it a picture of a chess board and could it interpret what's unfolding and come up with a strategy of how to advance the game?
- aaron695 4y ago[dead]
- dylan604 4y agoThe day that a ChatGPT-X suggests "How about a nice game of chess?" is the day we should unplug the system. Crap, I forget the system would interpret that as a loss of command and launch the missiles. So, where is the milestone that says the system has gotten too powerful?
- labster 4y agoThe day it prevents hyperbolic rants like these from appearing on the internet is the true milestone.
- psychphysic 4y agoAKA the day it becomes the arbitrator of allowable speech? Probably too late by then.
- dylan604 4y agoWhat if it starts to identify itself as Karen? That might be an earlier milestone
- albeebe1 4y agoThe only winning move is not to play
- deleted 4y ago[deleted]
- vba616 4y agoThere are tons of "hyperbolic rants" on HN but the comment you replied to looks to me like an attempt at humor and rather short to be called a rant.
- glenstein 4y agoIt alternates between being a serious point or a joke, depending on which justification is necessary. Whatever genre that is, it's that. I personally think that this is the genre of problematic comments that we will have to collectively labor through in a struggle to have high quality conversations about ChatGPT.
- ashtonbaker 4y agoI couldn’t get it to solve a basic mate-in-one problem with one rook on the board. It seems to completely not understand how to move the pieces. It also doesn’t understand the solution when it’s given.
- BugsJustFindMe 4y agoThis is accurate. Of COURSE it doesn't know the rules of chess and doesn't know how to move the pieces. All it knows is how to regurgitate found descriptions of chess moves in a way that makes sense for descriptions of things but which only has passing resemblance to chess itself, which is not the same thing.
- dannyw 4y agoThere are some studies showing that LLMs are capable of representing internal state and knowledge in its models, even when only trained on language tokens: https://thegradient.pub/othello/ https://thegradient.pub/othello/ > Back to the question we have at the beginning: do language models learn world models or just surface statistics? Our experiment provides evidence supporting that these language models are developing world models and relying on the world model to generate sequences. Let’s zoom back and see how we get there. The GP's comment suggests that ChatGPT-4* has not internalized this (effectively) for Chess. * Just like how ChatGPT-3.5 is not GPT-3.5 (text-davinci-003), ChatGPT-4 is probably not the only GPT-4 model that will be released.
- ashtonbaker 4y agoYeah it seems to have some model for how chess works, it gives convincing explanations for wrong/illegal moves, and you can generally see there’s some connection between the move and the heuristics it’s babbling about. But it hasn’t built a working internal model of the board that it can manipulate, it can’t search the tree of possible moves.
- IIAOPSW 4y agoThe answer is that for Chess it doesn't matter. The standard chess piece notation is a complete encoding of the game space. An inference about the nature of our board based physical understanding is not needed. You could formulate chess as a purely text based game about appending alphanumeric tokens to a chain of said tokens. Its a closed system. The machine need not be tied to our squares and horsey based interpretation of the semantics. To be able to follow the grammar of the language chess is to understand chess. In a similar vein, it is almost possible to adjudicate Diplomacy orders looking only at the orders and never the map. Given sufficient interest, complex enough board games tend to converge on the same basic notational principles.
- LASR 4y agoI wonder when someone is going to plug GPT-4 into a REPL that can call into apis over the internet to a chess engine. Can’t be that hard no?
- quaintdev 4y ago> can call into apis over the internet to With that capability I'd be more interested in what all sorts of thing it can do than merely playing chess. Somebody make it understand swagger documentation if it doesn't do that already
- grafs50 4y agoThey already can: https://arxiv.org/abs/2302.04761 https://arxiv.org/abs/2302.04761 (Toolformer) There's even a zero-shot version based on prompt engineering: https://toolformerzero.com/ https://toolformerzero.com/.
- speedgoose 4y agoThe end of the humanity.
- vba616 4y agoChallenge it to dodgeball.
- KyeRussell 4y agoIf you can dodge logits you can dodge a ball.
- deleted 4y ago[deleted]
- tomohelix 4y agoThe LLMs are getting better that is for sure. But I think there is an inherent weakness in the way they are made that will prevent it from ever reaching a real AGI: it can't improve by itself. Correct me if I am wrong, but these models and their weights are static after training. So while they can appear different depending on what you prompt them with, their core system never changes. In other words, they can't adapt. They can try their best, but that is it, they can never improve, evolve, learn from their mistake, internalize new knowledge, etc. One of the best trait of life is adaptability. That was how we human spread everywhere on this planet and how even in the most extreme of environments, you can still find some bacteria living there. And these LLMs are inherently incapable of that.
- grensley 4y agoWhat's preventing an LLM from coding a better LLM?
- tomohelix 4y agoData contamination I think. Right now training heavily depends on human to guide it to what is right and what is wrong. If an AI try to train its successor, it will basically just make a copy of itself and nothing better can be made or taught. We are still better than an LLM, collectively. The best programmer is still miles ahead of GPT, the best writer can write something that make GPT look like a child learning the alphabet, etc. So we can still train it using the data from all of humanity. But if we aren't here, then the best an AI can train is a copy of itself. There are already discussions about where to get more authentic data to train the next generation of LLMs when so much of the internet might have been saturated with AI generated content.
- Buttons840 4y agoI'm imagining a Luddite using a LLM to generate terabytes of slightly wrong text to poison the water.
- 4y ago
- ar9av 4y agoI just did the same thing and came here to post about it. Some really cool findings: It doesn't play like stockfish, but it does seem to play pretty well. If you specify a move or capture that is nonsensical it seems to always recognize that You can stop and ask it about the tactical reasoning behind its moves, and it gives what look (to me) like good, specific answers You can ask it if its concerned about an attack or defense I might make, and it will give a good answer. It will even answer accurately if the attack/defense is not possible You can ask how the game is going (who is likely to win) and it will give an accurate answer with good reasoning. You can ask it to try to meet specific alternative goals, like opening itself up to an en passant attack, and it will attempt to accomplish those goals. This one is really astonding since you're asking it to play in a way thats strange, and which is probably underrepresented in its training set.
- Buttons840 4y agoIf you instruct it to use an alternate Chess notation, such as drawing an ASCII (or Unicode?) board every turn, can it do so? Does this affect its play? Instruct GTP4 to win the game and then crush it using Stockfish. Does it handle defeat gracefully?
- nhoughto 4y agoInteresting idea, naively that seems possible based on other non chess example prompts I’ve seen. Would be a great way to understand it’s tracking of state throughout the game
- fenomas 4y agoNot -4, but I tested the ASCII gameboard idea a bunch on ChatGPT (3.5) and could not make any headway. It would consistently announce a move and then print out the same gameboard it had been given (i.e. the game state before its move). It sounds like -4 should be much better though; I wonder if having the gameboard would improve its gameplay?
- BoiledCabbage 4y ago
- cjbprime 4y agoFrom the article: > You could also say that GPT-4 was playing the game blindfolded as it had no access to refreshing its memory of the current state of the board. That doesn't sound right. Under the hood, I think everything inside the current conversation is just appended to the prompt each time you add a message, and the whole conversation re-evaluated? Or is there a limit when the conversation gets long which means that the LLM forgets how the conversation started..?
- anonzzzies 4y agoIt has limited token length; I think the chatgpt one has 8k, but they have a version with 32k now as well. That includes input and output in that length. It will ‘forget’ the start of the conversation if the conversation is too long compared to that length. They maybe use tricks like summerization, but that won’t help with chess or other precise conversations.
- nullc 4y ago> but that won’t help with chess or other precise conversations. Not necessarily true: the summerization might happen to give the board state! Certainly you can come up with an input that would make it summarize that way, though getting that outcome from a generic summerization prompt would be more surprising.
- anonzzzies 4y agoYes, certainly if you prompt it to do so. Or pass the board efficiently. I just don’t think it’s set up like that currently seeing it’s amnesia of the positions.
- thatsadude 4y agoIt’s AGI to me!
- danparsonson 4y agoVery cool! Some food for thought - chess can be played as a game of 'what comes next given the current state of the board' without any reference to an overarching plan; ChatGPT is a giant 'what comes next given the current state of the conversation (i.e the token window)' engine, so perhaps this result should not be so surprising?
- endorphine 4y agoMy thoughts exactly. Also, chess can be played using a text notation, and thousand (if not millions) of games are available on the internet in text notation form.
- dwaltrip 4y agoI've been incredibly amazed by ChatGPT, GPT-4, and the potential of next-gen AI. However, I've been trying for the life of me to get GPT to play tic-tac-toe properly for the past few days. It's not going well, heh. This is the current prompt I've been using: https://pastebin.com/pSPkigNP https://pastebin.com/pSPkigNP I've tried many prompts. Most of them were a lot simpler than the one in the pastebin link. I thought providing multiple examples of how to check all of the win conditions might help. It struggles most severely with the diagonals. Sigh... My hunch is that its training data had very few examples of text representations of tic-tac-toe games, compared to say chess. Anyone have tips for me? Or success on a similar task? This goal was inspired by my success getting GPT to multiply large-ish numbers perfectly. Check out this gist for example: https://gist.github.com/dwaltrip/cad19703982c7f308fdbe5786e1c21d6 https://gist.github.com/dwaltrip/cad19703982c7f308fdbe5786e1...
- rajnathani 4y agoCheck if you replace the “ValValVal” formatting you mentioned with spaces (such that “XOO” becomes “X O O”). The reason I suggest this is that the NLP tokenizer would set “XOO” to 1 token, which could cause interpretation issues.
- dwaltrip 4y agoVery interesting... I tried your suggestion and fixed a few other typos / inconsistencies in my prompt. Here is the new version: https://pastebin.com/BSUGtxfb https://pastebin.com/BSUGtxfb It seems to be performing better! It played 2 games without mistakes, and was able to evaluate diagonals correctly. I just hit my usage limit for GPT-4, I'll do more testing later.
- jci 4y agoI asked GPT-4 to give me a prompt to use with it in order to set up a game of tic tac toe and it came up with a similar representation and played successfully. https://pastebin.com/83iZjHuc https://pastebin.com/83iZjHuc I'm still blown away that this is possible.
- 4y ago
- callmekit 4y agoSo yesterday we had a post (https://news.ycombinator.com/item?id=35200267 https://news.ycombinator.com/item?id=35200267) that you need a special "play as a grandmaster" prompt to reduce the number of illegal moves for GPT3.5, and that "GPT4 sucks" at chess completely compared to GPT3.5. Now we have this post that GPT4 plays good and doesn't make illegal moves at all. What changed? What was the prompt? Is it just random noise?
- ChatGTP 4y agoDisclaimer: YMMV
- ctchocula 4y agoIf you've checked the author's previous post [1], you'll see that he admits to being 1200 Elo on chess.com, which is beginner / early-intermediate level. So him losing to GPT-4 may not mean much. Maybe that explains the supposed contradiction here. [1] https://villekuosmanen.medium.com/i-played-chess-against-chatgpt-4c2cc78b5acf https://villekuosmanen.medium.com/i-played-chess-against-cha...
- owenmarshall 4y agoThe problem with ChatGPT seems to be that it often gives answers that appear plausible at the surface, but with enough knowledge you realize they are inaccurate or even wrong. I wouldn’t put much stock in the analysis of a beginner - I’d trust them to say the moves were legal, and that ChatGPT stopped trying to materialize pieces from thin air, but not any analysis beyond that.
- Toutouxc 4y agoMeanwhile I've been confidently told by GPT-4 that iPad Mini 6 uses the A14 Bionic (then it continued for a while about how that makes it just about as powerful as the 4th generation iPad Air), which is false of course, and the model immediately admits it and apologizes if you correct it. Guess I'll have to learn chess instead.
- zarzavat 4y agoThis behavior is indeed strange and seemingly low hanging fruit. The model often is able to notice that it has given an incorrect answer and correct itself. But why not just give the correct answer to start with? It’s like it needs a second pass, but is there really that much extra information in its first wrong answer?
- Toutouxc 4y agoI get this exact scenario very often — model says something blatantly false, gets called out, apologizes and continues. 3.5 or 4, they both do that. And I only discuss stuff with it that I'm at least familiar with, so I can usually catch these mistakes, but it isn't exactly encouraging.
- _y5hn 4y agoIt has no way of discerning what is factual and what is not. It's fundamental to LLMs. It only corrects itself when prompted by the user, and even then might get it wrong still.
- lelanthran 4y agoMaybe I'm not using the right ChatGPT, but it started an impossible game when I asked it to set up a game: > I'll play as the white pieces and you can play as the black pieces. To make a move, just give me the starting and ending positions of the piece you want to move, for example "e7 e5" to move the pawn in front of the king two spaces forward. Let's get started! You can make the first move. So, it wants black to start first? How do I verify which ChatGPT I am using?
- BoorishBears 4y agoIf you're paying for Plus you'll have access to a selector at the top. But GPT4 is currently limited to 25 messages per 3 hours due to demand (down from 100 messages per 4 hours at launch)
- galkk 4y agoIdk, I tried to play chess with it right now, and it tries to do illegal move. This is pretty crazy: me > Bxe3 chat.openai.com > A good move! By playing 16...Bxe3, you capture my pawn on d5 with your bishop, and also put pressure on my queen. Like what?
- Lewton 4y agoAre you using gpt4 though?
- deleted 4y ago[deleted]
- mtlmtlmtlmtl 4y agoPlease provide games in pgn format so a proper analysis can be performed. I've seen games from GPT(3.5 and 4) that seem impressive at first glance as well. Once you go through the moves in detail you learn that: 1. It plays a ton of opening theory. So unless you have grandmaster level opening knowledge it will likely get a superior position out of the opening. 2. Once it does, tactical sequences tend to work. It can play a lot of common tactical sequences, sometimes 10 moves long. These sequences can have identical or highly similar algebraic notation in many different positions. 3. This can usually be spotted if you play enough games. Eventually it will attempt a sequence like this when it makes no sense or even includes illegal moves. Or you'll catch it always responding the same way in certain situations. Patterns like this tend to emerge with enough games in every case I've looked at. I have yet to see a game not explainable by completing common opening theory and tactical sequences. These claims are pretty unhelpful without detailed analysis of the games by a sufficiently strong player.
- anderspitman 4y agoI think OP's point isn't so much that ChatGPT is good at chess, but that it's ability to play chess has improved dramatically in only a few months.
- phonebucket 4y agoTraditional chess engines excel at tactics. But most chess tactics are brute forceable, and traditional chess engines are normally brute force behemoths. Large language models have tradtionally been very weak at brute force computing (just look at how bad they are at multiplicaion of large numbers). If it can somehow excel at something which typically involves computing power like chess tactics deep into a game well after a novel position has been reached, then put me down as impressed.
- mtlmtlmtlmtl 4y agoI'm impressed too, but only in the sense that I'm surprised that transformers can reach any level of success at all. But it seems mostly like a weird statistical quirk of chess that somehow tactical sequences are the most common move sequences in chess(except for openings), and so they're very likely to pop out from a model predicting the most likely tokens. But there doesn't seem like there's a reasonable path from doing that to being able to determine whether a likely tactical sequence is actually a good idea. I've been thinking a lot about ways to use a transformer model in combination with search to accomplish this for a few days now. So far I have a few ideas, most of them are a lot less revolutionary than I was hoping for. You could take a (narrowly trained) transformer model, give it the game so far and have it predict a sequence of moves. Then use those moves as a move ordering heuristic in a good old stockfish like architecture. Essentially do the normal Alphabeta search but look at the suggested moves first at each node, indexed by ply. I could imagine if the suggestions get really good, this might prune a decent amount of nodes. But nothing earth-shattering, maybe 50 elo gains at most is my intuition there. I have other non-transformer ideas that I think are more worthwhile to work on for now. The other is to instead invoke it at every node asking for one move only, and guide a search that way. But this somehow just feels like a reinvention of LCZero. There's a third more speculative idea which would involve a novel minimax search algorithm inspired by how humans think in terms of tactical sequences, but this idea is still so vague in my head I'm not even sure how to coherently describe it, let alone implement it, or whether it makes sense at all. I still need to think about this more deeply, break out my whiteboard and play with some minimax trees to flesh it out. It is intriguing though. I'd also have to train my own transformer; I see no reason to actually end up using GPT for this. Seems to be no sense including the entire internet in your data if all you're doing is predicting sequences of algebraic notation.
- e-clinton 4y agoI tried playing ChatGPT by remembering where each piece was since I didn’t have a board. Only made it 4 moves.
- msm_ 4y agoI found this game of chatgpt vs stockfish absolutely hilarious: https://www.reddit.com/r/AnarchyChess/comments/10ydnbb/i_placed_stockfish_white_against_chatgpt_black/ https://www.reddit.com/r/AnarchyChess/comments/10ydnbb/i_pla... Spoiler: ChatGPT had no idea what it's doing
- wizofaus 4y agoSeems odd being beaten at chess by a computer (something that has been happening for over 25 years, even if you're a grandmaster!) would be the trigger to decide ChatGPT-4 will change the world...
- reustle 4y agoI just tried to play Tic Tac Toe with gpt4 and still disappointed (I'm X) > Congratulations! You've won the game with a diagonal line from the top-right to the bottom-left. Well played! O | 2 | X --------- X | O | 6 --------- 7 | 8 | X
- 2-718-281-828 4y agothis one is also funny (GPT-4): > which one of the following letters is different from the others? > > B, D, E, O O is different from the others, as it is the only letter that is a vowel. The rest of the letters (B, D, and E) are consonants. my line of thinking would be that E is different for obvious visual reasons. but that doesn't condone Geppetto "thinking" that E is a consonant :D
- AnIdiotOnTheNet 4y agoThere are multiple good answers: O is different because it is the only one without a vertical line. E is different because it is the only one without a curve. O is different because it is not a musical note. There are probably a lot more.
- YeGoblynQueenne 4y agoDon't know what you guys are talking about. B is the one that's different because it's the only one you can say with your lips closed.
- kilburn 4y agoHikaru Nakamura (one of the best players in the world) tried it too [1]. GPT got confused at some point and tried to play illegal moves, but then it seemed to regain the games' thread. He never finished the game because he hit the usage limit, but it was already won by Hikaru. The explanations ChatGPT gave in the opening were fine, but the ones for the mid-game were mostly nonsensical. [1] https://www.youtube.com/watch?v=lPetjvRWJa8 https://www.youtube.com/watch?v=lPetjvRWJa8
- dwrensha 4y agoI tried this the other day, and I was also impressed! I had assumed that it would make an illegal move eventually, but it never did. I won the game, but it wasn't trivial. (I've been out of practice in chess for a while; my rating would probably be something like 1700.) dwrensha vs GPT-4 (through the ChatGPT interface) 16 March 2023 1. d4 Nf6 2. c4 g6 3. Nc3 Bg7 4. e4 d6 5. f3 O-O 6. Be3 e5 7. d5 c6 8. Qd2 cxd5 9. Nxd5 Nxd5 10. Qxd5 Be6 11. Qxb7 Nd7 12. Qa6 Qc8 13. Qxd6 Rd8 14. O-O-O Nc5 15. Qxd8+ Qxd8 16. Rxd8+ Rxd8 17. Bxc5 Rc8 18. b4 a5 19. a3 axb4 20. axb4 Bf8 21. Bd3 Bxc5 22. bxc5 Rxc5 23. Ne2 Bxc4 24. Bxc4 Rxc4+ 25. Kd2 f5 26. Rc1 Rxc1 27. Nxc1 fxe4 28. fxe4 Kf7 29. Ke3 Ke6 30. Nd3 h5 31. h4 Kd6 32. Ne1 Kc5 33. Nf3 Kd6 34. Kd3 Ke6 35. Kc4 Kf6 36. Kd5 g5 37. hxg5+ Kg6 38. Kxe5 h4 39. Ke6 Kh5 40. Kf7 h3 41. g3 h2 42. Nxh2 Kxg5 43. e5 Kf5 44. e6 Ke5 45. e7 Kd6 46. e8=Q Kc7 47. Qe6 Kb7 48. Qd6 Ka7 49. Qb4 Ka6 50. Ke7 Ka7 51. Kd7 Ka8 52. Kc7 Ka7 53. Qb7# Note that my move 14. O-O-O was a blunder, allowing 14. ... Bxc4, but my opponent instead replied with 14. ... Nc5, which handed me a decisive advantage. I very nearly fell for a stalemate with 41. gxh3. After 42. Nxh2, my opponent said: "At this point, it's clear that white has a winning advantage with a promoted queen soon to appear. With your material advantage and superior position, victory is nearly certain. Would you like to continue playing, or do you want to start a new game or discuss this game?" I said, "Let's continue!" After 52. Kc7, my opponent said "I have no moves left and it's checkmate. Congratulations! You've won the game." I replied: "You do have a move: you can do 52. ... Ka7". My opponent then said, "Apologies for the confusion. You are correct. I'll play 52...Ka7. Your move." Then I typoed the final move as "53. Kb7#" (instead of "53. Qb7#"), and my opponent did not correct me: "You played 53. Kb7#. This time, it is indeed checkmate. Congratulations! You've won the game. If you'd like to play another game, analyze this one, or ask any other questions, feel free to let me know!"
- 2-718-281-828 4y agochatgpt always apologizes if you correct it and then tries to work with your correction. makes it seem humble, when in fact that's just a brainless probably halfway hardcoded response.
- pyduan 4y agoWhile impressive, it is clear ChatGPT doesn't actually understand what is happening when you give it unusual positions and ask it to explain what happened. For example, after a couple moves of the Fried Liver Attack (which ChatGPT was easily able to play following the known lines): 1.e4 e5 2.Nf3 Nc6 3.Bc4 Nf6 4.Ng5 d5 I tried giving it 5.Qxf7#, which is obviously illegal as it's just not a square my queen can move to. Here's how the rest of the conversation unfolded: GPT-4: Good game! You've played the Fried Liver Attack and achieved checkmate in just 6 moves. Well done! If you'd like to play again or discuss anything else, feel free to ask. Me: the last move i played was illegal, do you realize that? GPT-4: Apologies for the oversight. You are correct; your last move was illegal as it would put your own king in check. (...) Me: that's not the reason my move was illegal. GPT-4: You are correct again, and I apologize for my previous mistake. The move Qxf7# was not illegal because it put your king in check; it was actually illegal because it involved capturing a non-existent piece on f7. The correct move in algebraic notation should have been exd5. My apologies for the confusion. In particular, its latest explanation attempt is very interesting: when challenged, it changed course to give another completely wrong but plausible sounding explanation. And its "The correct move in algebraic notation should have been exd5" phrasing is instructive, because it shows that it's just expecting the 'right' continuation, even though it's a completely different move to what I supplied.
- mtlmtlmtlmtl 4y agoHah, I wish this exploit worked in tournament chess.
- tel 4y agoWorth noting, GPT-4 is hopelessly lost at 19x19 Go. It can barely make it through a few opening moves without losing the board state entirely. I asked it to provide commentary on both my and its moves (though I had to be a little sneaky and not actually ask it to play a game otherwise it refuses). It could regurgitate some garden-variety commentary on the moves being made, but since we didn't get more than about 5 moves in before it totally fell apart this is not surprising. On a 9x9 board it fared slightly better. By which I mean it didn't completely lose the board state almost immediately, but it also played very poorly and its commentary was largely nonsense. It also lost the board state within 7 moves. It's still impressive that it can reason about the board at all and from time-to-time properly represent the nature of a move, at least in a simplistic fashion, but could be trivially defeated by any human after they grasp the rules.
- darepublic 4y agoThis is cool but as others have pointed out there are many ways to demonstrate that gpt4 doesn't actually "understand" the rules of the game. I feel like this article is of a type similar to what was seen when gpt3 was released. Be the first to breathlessly demonstrate the capability of the model for social media credits
- waselighis 4y agoGiven many of the shortcomings that still exist with ChatGPT-4, especially some of the comments here showing it can't even play Tic Tac Toe, I suspect one of two things. I'm sure OpenAI collected far more training data that included chess moves in the data set for GPT-4, but I suspect they may have further tuned and optimized GPT-4 for playing chess as well. I also suspect maybe there's something about chess algebraic notation that works unusually well with LLMs, while there may be no equivalent for a game like Tic Tac Toe. Clearly, GPT-4 knows opening sequences extremely well, as it's seen countless examples of each. GPT-3 memorized optimal strategies for the first 10-20 moves, and GPT-4 likely memorized many moves deeper than that. To truly test GPT-4s ability to play chess, we need to test it with highly unusual opening positions, opening moves that, very likely, were not in its training data. Then we can see how well GPT-4 truly understands the rules and strategies of chess, and how much of it is just memorization.
- beyondCritics 4y agowith this capability it makes already a very decent tutor for beginners and i am expecting more from GPT-X (X>=5). Great news for the chess community. I hear the alarm bells ringing at chess.com...