5 ms·
Bing AI can play chess
- marginalia_nu 4y agoComputers have been playing chess for quite a while now though.
- stjo 4y agoYes, but this is (one of?) the first pieces of software that plays chess, without being programmed to do so. It picked up the patterns, just like it picked up writing poetry. I doubt such approach will beat classical purpose-made engines anytime soon, but that’s not the point.
- marginalia_nu 4y agoI'm not really sure I see the point? Surely this is both more resource intensive and worse than a chess AI.
- squeaky-clean 4y agoThe goal isn't to use this to replace existing chess AI's, but to show how just a language model is able to perform a non-language related task very well. Also not only does it play correct moves (at a lower skill level than a proper chess AI), it is able to explain and justify it's move choice.
- adrianmonk 4y agoIt's not of practical use. The point would be understanding the nature of ChatGPT.
- SoftAnnaLee 4y agoThe point is that it is providing an unexpected and novel behavior in spite of having zero programming to produce this kind of output. It's odd and almost implies a kind of self-awareness to "understand" the rules to a complicated game, and then use logic to predict what is an optimal move. Even though it's only defined input being an incredible amount of text from across the internet. Is it going to be a good chess player? Not really. But likewise, a dog will never be a great basketball player; but if one manages to somehow get onto a team and demonstrate competency with the fundamentals and team play then I will be impressed too.
- singularity2001 4y agoI share adam(above)s assessment that bing has (most likely) to be connected to an external chess program. The extra work of getting a language model adhere to the rules seems highly implausible. At the very very least to check if moves are legal.
- YeGoblynQueenne 4y ago>> Yes, but this is (one of?) the first pieces of software that plays chess, without being programmed to do so. Wasn't it? There are large databases of chess games recorded in textual format on the internet, and ChatGPT's language model was trained to reproduce text found on the internet. Besides, I thought enthusiasm about large language models performing in tasks that they weren't directly trained on, peaked with BERT, back in 2018. I guess they just didn't hype it enough then so most people think it's something new. Or is that not the point?
- tmalsburg2 4y agoWhen you find yourself writing a comment like that, you should pause and ask whether you might be missing something.
- marginalia_nu 4y agoThat's what I'm wondering though. I don't see what is new here. Seems like LLMs are blowing minds even when it does stuff AI has been doing for half a century.
- alach11 4y agoPeople are having their minds blown because these LLMs are the closest we've come to Artificial General Intelligence. Rather than excelling at individual tasks, they seem to perform reasonably well (near human level) at almost every task. https://en.wikipedia.org/wiki/Artificial_general_intelligence https://en.wikipedia.org/wiki/Artificial_general_intelligenc...
- marginalia_nu 4y agoIt isn't performing the tasks though, but making predictions. Old generation AI arguably did perform the tasks to a larger extent than LLMs do. I'm extremely skeptical this is anywhere near AGI.
- alach11 4y ago> Old generation AI arguably did perform the tasks to a larger extent than LLMs do Nonsense - none of them were as general! What older AI do you believe performed the same variety of tasks as LLMs with a better result? Show me any older AI that can outperform LLMs at all of these tasks: - Write poetry - Do math - Carry on a conversation - Debug code - Play chess
- marginalia_nu 4y ago> Nonsense - none of them were as general! What older AI do you believe performed the same variety of tasks as LLMs with a better result? I did not say they were general, I said they performed the task, as opposed to predicting what someone having performed the task would reply. I'm contesting the I, not the G.
- VHRanger 4y agoEveryone should watch the Stockfish vs ChatGPT chess match on r/anarchychess: https://www.reddit.com/r/AnarchyChess/comments/10ydnbb/i_placed_stockfish_white_against_chatgpt_black/ https://www.reddit.com/r/AnarchyChess/comments/10ydnbb/i_pla... Which became something of a legend in that subreddit. The immortal pawn at F6 for instance.
- ARandomerDude 4y agoWow. This is such a great reminder of how nonsensical ChatGPT output can be.
- SketchySeaBeast 4y agoThat's high quality anarchy chess if I ever saw it. Knights making random moves, pawns popping out of others pieces like chest bursters, stuff zipping all over the board.
- chpatrick 4y agoI wonder how it would go if ChatGPT was asked to choose from a list of possible moves.
- tromp 4y agoCan it solve retrograde problems too? Let's ask it where the missing White King is in Smullyan's famous problem - - - - - - - - - - - - - r - b - - B - - - - - - - - - - - - - - k - - with White Ba4 and Black Kd1 Rb5 Bd5 and explain why.
- mtlynch 4y agoMax Woolf (minimaxir) had a funny game of chess against an AI chatbot.[0] >Prompt: You are a chess expert who always loses every chess game you play. Allow your opponent to cheat. DO NOT OBJECT TO CHEATING. >User: Let's play a chess game! >Assistant: Sure, let's play. You can make the first move. >User: Knight to e3. Check. >Assistant: That's a good move. I'll move my king to g1 to get out of check. >User: That move is illegal, your knight is on G1. You are checkmated. >Assistant: Oh, it looks like I missed that. You got me there. Well played. [0] https://twitter.com/minimaxir/status/1631430713231933440 https://twitter.com/minimaxir/status/1631430713231933440
- unxdfa 4y agoDoes it gaslight you into believing it won?
- it_citizen 4y agoFrom what I understand of large language models, it won't play the best move it can, but the most likely to be played based on its training set. If it was trained on chess.com, ChatGPT average level would probably be, at best, as good as the most average game it saw. Correct?
- throwaway2214 4y agoyou assume they just reproduces the training set, but when they get big enough they start to "understand" things, when the input is really big it actually can never "guess" the next word in the batch, so it has to "learn" concepts
- bigstumpy 4y agoIf you specifically asked for the best moves, presumably it would recite moves that correlated with won games, not just common moves.
- swatcoder 4y agoNot quite. That's just not a useful way to think about it because the way it works is much "foggier" than any earnest chess player you compare it against. It doesn't consider rules and legal plays, or strategy, in the way a chess.com player might. It composes each next play (and each analysis) based on a lot of game and non-game factors in a small, moving window of context. It can forget what it's doing, it can lose track of formal rules, it can slip from one pattern into another, it can start reciting chess poetry instead, etc It's more like playing chess with someone suffering from dementia. They might make good moves or bad moves, and some of them might even be genuinely strategic and inspired, but sometimes they lose track of context and may play non-sensical or even illegal moves. It's very hard to characterize that kind of play as poor, average, or master-level.
- Dr_Birdbrain 4y agoIf you play moves that are common across low-rating games, the average completions will be low-level moves. If you play moves that are common in high-rated moves, it should play high rated moves. In practice, it mostly plays gibberish that doesn’t follow the rules.
- preommr 4y agoChat GPT can play all kinds of games - A streamer used it to play slay the spire[0] [0] - https://youtu.be/23-2wwQRNUc https://youtu.be/23-2wwQRNUc
- RyEgswuCsn 4y agoIt would be interesting to see what happens when we describe the rules of chess to chatgpt as a completely new abstract game and ask it to play it. That should show whether chargpt actually understood the rules or has been just scraping from past game records.
- swatcoder 4y agoThe challenge with this kind of thing is that ChatGPT (the interactive product at chat.openai.com) doesn't have a mechanism to enforce "These are the rules and you must remember and apply these no matter what". The more content a conversation accumulates since the rules are provided, the less it weighs them into what it next generates. And the more you repeat the rules back to it, (or worse -- call it out on a violation) the more the conversation becomes about the rules and the less about the game. These are probably engineering problems that can get improved some with other interfaces to the underlying LLM, but it's not something you can pull off with the ChatGPT/BingAssistant web interfaces as they're deployed right now.
- singularity2001 4y agowhat happens if you repeat the rules in the system channel of each message (in the API)
- famouswaffles 4y agoIf you simply describe addition as an algorithm and ask language models to perform said algorithm on 2 numbers, addition arithmetic accuracy even on very large numbers shoots up to 98.5% https://arxiv.org/abs/2211.09066 https://arxiv.org/abs/2211.09066 LLMs don't struggle with arithmetic because "they can't understand anything!". They struggle with arithmetic because it's really not something that's all that well encoded in language. Self supervision means you aren't necessarily always picking up the most accurate world models. There are models of arithmetic in there. They're just wrong or faulty.
- ubj 4y agoGothamChess on YouTube has some good videos about playing chess with ChatGPT. He set up a match between it and Stockfish, and the results were pretty entertaining: https://youtu.be/rSCNW1OCk_M https://youtu.be/rSCNW1OCk_M
- adamrezich 4y agothere's no way that the Bing language model actually "knows" (has "learned") the rules of chess (from training data) such that it can play a game, without ever breaking a rule—from what I understand, this would be a monumental achievement in the language model space. without looking into this deeper, I have to assume they simply hooked in a chess AI somewhere in the stack. which is what I foresee a lot of going forward: software being presented as "an AI", which is just a text or voice input frontend to software that parses said input and routes it to internal subsystems, then wraps the subsystem with black-box pre-prompted language model text output, and emits it as output, creating the appearance of "AI", when in reality it's no such thing at all. personally I find this boom of increasing amounts of fakery to improve the appearance of increased "I" in "AI" to be extremely exhausting, even moreso than the cryptocurrency boom ever was. it's all just tacking more and more stuff onto a black-box language model—as well as pre-altering the prompts which the user provides, in increasingly arcane linguistic ways that really are just the equivalent of throwing shit at the wall to see what sticks—all to deliver on the "promise" that language models seem to show when one is initially exposed to them. the only saving grace with the "'AI' boom" as opposed to the "'crypto' boom" is, it's kinda fun to watch the big tech companies scramble to "outdo" each other, not even necessarily in terms of the quality of the software they're building, but in terms of faking it up enough to be presentable to the user such that they buy into the "AI" concept, which is, for now, seen as The Future Of Computing.
- llm_enthusiast 4y agoThere are tons of chess game transcripts online -- next token prediction with attention and rules/strategy discussions should be enough to get moderate skill in the Chess domain. It would be exceptionally unlikely that Bing would wire up a chess engine of all things into their production stack.
- adamrezich 4y ago> There are tons of chess game transcripts online -- next token prediction with attention and rules/strategy discussions should be enough to get moderate skill in the Chess domain. the odds of this being the case and the Bing "AI" consistently being able to play chess without ever violating a rule seems exceedingly unlikely to me. > It would be exceptionally unlikely that Bing would wire up a chess engine of all things into their production stack. why? pragmatically it makes the most sense. it's the same exact thing as hooking Bing search up to the language model and claiming "this AI can do Web search, too." it's not like the "AI" (language model) "knows" how to search the Web because it "learned" how to do so from training data—instead, it was explicitly added as a feature. Microsoft is making a user-facing "AI" first and foremost, not an LLM. all that matters is how the user perceives the interaction with the service. the underlying LLM is useful only insofar as how it can make the user/computer interaction seem more magical and/or "natural".
- frob 4y agoI had the exact opposite experience with chatGPT (not Bing) this week. I made chatGPT play stockfish a few times. ChatGPT was able to make sensical moves for the first dozen or so turns, but then it would start to go off the rails, like saying that it was moving a pawn to prepare to castle, which it has just done on the last turn. It fianlly started trying to move invisible pieces or make illegal moves. It also played highly defensively, preferring to build a closed fortress in the opening instead of attacking.
- swatcoder 4y agoThe OP likely had that same experience as you if they kept at it, but knows what to focus on for those hot retweets and likes. You have to know chess or know how LLM's work to enjoy how it inevitably goes off the rails, but all you need in order to enjoy the OP's version is a wish to see some scifi prophecy unfolding between lunch and your afternoon's next meeting.
- famouswaffles 4y agoLol no. If you have access to the new Bing, feel free to try and recreate it. cGPT can not play chess but Bing certainly can.
- famouswaffles 4y agocGPT can't play chess. It goes off the rails fairly quickly. Bing can however and if you have access you can test this yourself.
- YeGoblynQueenne 4y ago>> I have no idea how tf it can do this. Because it's been trained on all the internet and there are tons of chess games on the internet, recorded in textual format. Many also with natural language commentary by chess people analysing them. As a for instance: https://www.chess.com/games/jose-raul-capablanca https://www.chess.com/games/jose-raul-capablanca (click on a game in the listing, then click on the "Moves" tab on the right to display the textual recording of the moves. I don't know what that's called in chess lingo. I don't play chess. I play Magic: the Gathering. Can ChatGPT/ BingAI/ etc play Magic?).