9 ms·
Man beats machine at Go in human victory over AI
- samothrace 4y agoRepetitive phrase makes up redundant title
- pmontra 4y agoPaywalled.
- sp332 4y agoI can't find any other coverage of it. I wonder if it's related to this technique discovered late last year. https://arstechnica.com/information-technology/2022/11/new-go-playing-trick-defeats-world-class-go-ai-but-loses-to-human-amateurs/ https://arstechnica.com/information-technology/2022/11/new-g...
- dkjaudyeqooe 4y agoLooks alot like it. I guess the only difference here is that it's a human employing the strategy.
- netvarun 4y agohttps://archive.is/tDYsY https://archive.is/tDYsY
- The28thDuck 4y ago~One training pass later…~
- gwern 4y agoWould probably do little good. They show that the attack goes through almost perfectly even if you give the target an entire wallclock hour per move to do tree search to try to defend against it (as opposed to the more typical seconds or minutes per move search budget): https://goattack.far.ai/pdfs/go_attack_paper.pdf#page=8 https://goattack.far.ai/pdfs/go_attack_paper.pdf#page=8 So it's a serious blindspot.
- dpaleka 4y agoMore search won't do good, but why wouldn't targeted training help? The way I see it is that the adversarial policy search discovers positions which are off-distribution for anything seen in the victim's self-play training. But training on that particular sort of adversarial states should help against the human player which has learned the strategy, just like training on patch adversarial examples in vision helps against the same type of patches. Of course if the adversarial policy is again allowed to find off-distribution states (by playing against the victim), it will certainly find ways to beat it, until the model is playing perfectly. (Emergent gradient obfuscation could also theoretically happen, but I don't know if it has been demonstrated to actually happen.)
- ablob 4y agoMore targeted training won't do good, but why wouldn't more search help? We've apparently entered the stage where the deciding factor between who wins, man or machine, is just an arms race.
- dpaleka 4y agoMore targeted training won't do good, but why wouldn't more search help? My understanding is that gwern above linked solid evidence in the paper for more search not being enough, as in, the model's evaluation NN is so way off target when searching, that realistic amounts of search don't help. Go seems to have many possible moves per position, so the search doesn't go very deep anyway. Feel free to correct me if I'm wrong, it might be that I misremembered how AlphaGo-style systems work.
- AdamGleave 4y agoOne of the authors of the paper here! David Wu (aka lightvector), the creator of the KataGo AI system that we target, is actually doing a training pass now initializing self-play games to start at adversarial positions found by our adversarial policy. It does seem to have improved things significantly: our adversary's win rate goes down from >99% to around 4% against the latest KataGo checkpoint. However, the real question is whether KataGo has started properly understanding the cyclic groups, or has just learned some heuristics that saves it most of the time. In other words, if we repeated our attack again against the latest checkpoint, could it learn to reliably repeat the 4% of winning cases? We're looking into doing our own adversarial training run to see if we can get to a point where the KataGo agent is robust. My personal suspicion given how difficult adversarial examples have been to eliminate in image classifiers and other ML systems is that although we'll be able to train particular vulnerabilities out of the system, there's still going to be a long tail of issues that can be automatically discovered and exploited.
- dpflan 4y ago"The tactics that put a human back on top on the Go board were suggested by a computer program that had probed the AI systems looking for weaknesses. The suggested plan was then ruthlessly delivered by Pelrine."
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- visarga 4y ago> a computer program that had probed the AI systems Ah, they found an adversarial attack with another program. All neural nets suffer from adversarial attacks.
- bnprks 4y agoYes, but it's more unusual to find an adversarial attack that can be described in human terms and successfully executed by a human against two different models
- visarga 4y agoWith just one trick we won't accomplish much. One lone snowdrop does not make spring. This trick won't generalise, it is just by itself.
- tromp 4y agoThis webpage is rather more informative: https://goattack.far.ai/human-evaluation https://goattack.far.ai/human-evaluation allowing one to replay all games. It looks like the strategy involves letting the computer completely surround a group with 2 eyes, and then depriving that surrounding group of 2 eyes (which the AI appears to be blind to).
- deleted 4y ago[deleted]
- drewcoo 4y ago> depriving that surrounding group of 2 eyes (which the AI appears to be blind to) Completely blind or just lacking depth perception?
- tromp 4y agoI think it has learned that stones completely within your own ring of stones are dead and can just be assumed part of your own eyespace. It has just failed to learn the exceptional case where such stones form 2 eyes.
- hinkley 4y agoI thought Kasparov had a perfectly reasonable argument that in a human v human contest, the players and any coaches have access to a public record of all previous contests the opponent participated in, and can use this information to prepare for a match. Deep Blue had Kasparov's record, but he didn't have Deep Blue's record, which is lopsided. You are always exploiting the flaws of your opponent in these games. Winning is a matter of exploiting more than you have been exploited.
- Yajirobe 4y agoHeadlines went from 'Machine beats man at Go' to 'Man beats machine at Go'
- dmarlow 4y agoMaybe it should have been, 'Machine helps man beat machine at Go'.
- addandsubtract 4y agoMan bites dog
- TristanBall 4y agoShortly to be followed by: AI, "Humans will take all our jobs!"
- scotty79 4y agoSo the trope of human beating AI by doing something moderately stupid is alive an well. So far...
- dkjaudyeqooe 4y agoThe "stupid (but winning)" strategy is a good one it seems. They apparently get pruned before the AI works out that they can win.
- visarga 4y agoDid they say how many times they failed?
- brilee 4y agoCongrats! I see by comparing the author list of v1 and V2 of this paper[1] that a Go player[2] was consulted before making claims about defeating a Go AI. This is indeed a nontrivial exploit that passes muster with a Go player who was skeptical last time around [3]! [1] https://arxiv.org/abs/2211.00241v2 https://arxiv.org/abs/2211.00241v2 [2] https://agagd.usgo.org/player/13822/ https://agagd.usgo.org/player/13822/ (6dan amateur strength is very strong, roughly 2300 Elo in chess) [3] https://news.ycombinator.com/item?id=33449600#33450069 https://news.ycombinator.com/item?id=33449600#33450069
- SonOfLilit 4y agoContext: v1 of the paper exploited a misconfiguration in how they configured the Go AI to play with an esoteric ruleset (the vulnerability that they found was documented in the KataGo docs, took me a few minutes to find it. KataGo still played much better Go than their adversarial program and no Go player would dispute it). My analysis at the time: https://twitter.com/AurSaraf/status/1588184384116514818 https://twitter.com/AurSaraf/status/1588184384116514818 Now they fixed the misconfiguration so their training could find actual weaknesses, and they found a very big actual weakness, that even weak amateur Go players like me can look at and say "oh my God AI what are you doing defend your group!". And it even generalizes to another popular Go AI, which is great! This is a great result, very impressive, great work by the team.
- brilee 4y agoAppendix H in the paper does a good job of explaining why this cyclic topology thing makes sense - "David Wu’s hypothesis is that information propagates through the neural network in a way analagous to going around the cycle, but it cannot tell when it has reached a point it has "seen" before. This leads to it counting each liberty multiple times and judging such groups to be very safe regardless of the actual situation." (David Wu is author of KataGo) I'm a little bit more skeptical now; it seems like what happened was that David explained some known pathologies of Go AIs (based on analysis by Go AI experts), and one of those pathologies (cyclic topology) was soft-coded into an agent and then run through the adversarial training process to harden it, producing the result in the paper. The arrow of causality runs backwards here - I would be far more impressed if the adversarial training process could help elucidate failure modes of AIs. This looks more like expert analysis laundered through adversarial ML woo...
- steve-atx-7600 4y agoI beat chatgpt at writing javascript without an infinite loop....not sure why it or this should be newsworthy though.
- yreg 4y agoWell go beat stockfish at chess and report back if you find it mundane. It's newsworthy because the AIs that were beaten seem superhuman at go. Unlike ChatGPT at writing code.
- deleted 4y ago[deleted]
- fjfaase 4y agoIt looks like this shows that GO AI does not have the 'deep understanding' of Go as it was thought to have. It seems that many amateurs Go players can learn the tricks to exploit these weaknesses. In some sense it is more sensational than Lee Sedol beating AlphaGo, because it shows that Go is not yet beaten by AI and that this is only the true start of the battle between humans and AI with respect to Go.
- inimino 4y agoI appreciate the sentiment, but it's nothing that dramatic. The battle remains over. This just shows that the training process remains imperfect. It's a bit like finding a bug.
- mensetmanusman 4y agoWe know from math though that it’s impossible to prove there are no bugs. Maybe complicated systems will always have these types of exploits.
- modeless 4y agoI say if the adversarial strategy is learnable by human players (even if humans didn't invent it), then the battle isn't over yet. Especially if the same adversarial strategy works against most or all AI opponents. The battle is really over when there are no more adversarial strategies simple enough for humans to use.
- inimino 4y agoIn chess, it is possible that humans and bots can both someday play perfectly, and every game will be a draw. (Young people might play chess until learning how to draw, and then move on.) For Go there is no such possible outcome, and deep reading (calculation) will always be in the machine's favor. The architecture of AlphaGo (policy and evaluation networks plus MCTS) is sufficient to recreate and surpass any kind of human-practical strategy. The era of human dominance in Go is simply over. Of course, current AI programs will never be perfect either, so there may continue to be such adversarial attacks, but they will necessarily become harder to use (and quickly impossible for humans). AI go programs probably have nearly infinite possibility to continue marginally improving against each other, though.
- deleted 4y ago[deleted]
- mteam88 4y agoSeems like propaganda from the losing side of a war. Are we taking this too hard?
- 0xdeadbeefbabe 4y agoWho applied the training algorithms and collected the data? Maybe they are hiding behind a curtain somewhere.
- mark_l_watson 4y agoI was a little confused by the article. Which Go program, specifically, was he playing?
- pradn 4y agoThe flaw worked against both KataGo and LeelaZero.
- cultofmetatron 4y agothis has major john henry vibes
- prettyStandard 4y agoMy first thought.
- emmelaich 4y agohttps://en.wikipedia.org/wiki/John_Henry_(folklore) https://en.wikipedia.org/wiki/John_Henry_(folklore) I initially searched for a Major John Henry. :-) There's also a cartoon (Chuck Jones?) about an axeman defeated by a Gyro Gearloose-type character (with a chainsaw) in tree felling.
- Izkata 4y agoOh yeah, I remember hearing about that one before. The one coming to mind for me was actually named after him, the AI built to oppose Skynet: https://terminator.fandom.com/wiki/John_Henry https://terminator.fandom.com/wiki/John_Henry
- Pigalowda 4y agoA real John Henry competition wouldn’t have the human find out the vulnerability from another AI and exploit it. He would have picked up his own Go pieces and a little piece of board and played until it was the death of him. Lord, Lord.
- labrador 4y agoTo me this is a story of one AI beating another. Yes, there were humans involved on both sides of the equation.
- culi 4y agowell if you're gonna deconstruct things to that level then you could also equivalently say this is and always has been a story of one(/a group of) human(s) beating another
- labrador 4y agoYou could impress me more if you told me a human Go player beat an AI Go player without the aid of AI
- hgsgm 4y agoImpressive that in the same thread you swung from "computers can't understand Go like humans" to "humans can't understand Go a swell as computers without computer help"
- labrador 4y ago> humans can't understand Go a swell as computers without computer help That's not what I said. What I said and what was my intended meaning was that humans can understand Go at a deeper level with a the aid of a computers. The computers don't understand go at all. They're software.
- squeaky-clean 4y agoI doubt there are any high level players in the world of games like Go that haven't studied with an AI. Or at the very least spent time practicing against one. Even if you only learned Go from humans, somewhere along the lines someone probably taught you something that was learned via AI.
- fjfaase 4y agoTrue, one AI was used to found a weakness in another AI. But then it turned out that the weakness could easily be used to defeat that the AI by a strong amateur level player. 6 Dan amateur is about 1 Dan professional, realizing that Lee Sedol was a 9 Dan professional player. I guess that many amateur Dan players can master the weakness. It also seems to shows that the AI does not have the 'deep understanding' of Go as it was thought to have to be able to beat the top professional players. We might be still a long way away from an AI that truly understand the game. And if this is true for one game, it might also be the case for many other AIs. The road to General Intelligence might still be a long road.
- Lyanden 4y agoFew years ago, the author of KataGo tried to solve one of the hardest handcrafted problem in Go using AI[1]. In the article, he pointed out that the engine had noticeable weakness of solving large scale fight and resolving odd shapes. I think the study confirms that weakness, and I am surprised that the tactics (to setup the circumstance for exploit) are easy to understand because those are actually being commonly taught to be strategic blunders to avoid. E.g. allows a group to be completely surround in early game is highly inefficient, even if the group survives with 2 eyes, it's only netting 2 points. [1] https://blog.janestreet.com/deep-learning-the-hardest-go-problem-in-the-world/ https://blog.janestreet.com/deep-learning-the-hardest-go-pro...
- pclmulqdq 4y agoThis is not unsurprising - Hikaru Nakamura in the chess world is well known for beating chess computers while doing funny things, entirely based on analyzing the strengths and weaknesses of the bot.
- eklitzke 4y agoThose are weak bots though. Hikaru can't beat the strongest bots (nor can Magnus nor anyone else), he has videos on YouTube of him playing Stockfish and Komodo on the strongest settings.
- pclmulqdq 4y agoThat is true, but I do think that the bots he beats regularly tend to be rated much higher than he is (which would not happen with a human). Go bots have had less time to mature, so we will eventually have unbeatable ones, but I guess not yet.
- bamboozled 4y agoso we will eventually have unbeatable ones, but I guess not yet. While I believe it's true these things might become practically unbeatable. I also am getting the feeling we've put god like faith in these systems when ultimately, we find they may have critical flaws and weaknesses like all to almost all other things. Of course we're at the point where everything still seems like magic.
- twright0 4y agoDo you have a source for this? I've seen content around GMs finding stronghold-style setups that the AI cannot find a way out of, and theoretically getting to repeated move / 50 moves without capture draw scenarios. But I'm not aware of any examples of Hikaru or anyone actually winning against the best computers (eg modern Stockfish) on even footing
- searealist 4y agoThe strongest chess player in the world can never even draw Stockfish a single game when playing at its full strength even after a billion games.
- redox99 4y agoHas there been significant progress in Go in terms of AI strength after AlphaZero (like there has been with chess), or has Go AI stagnated because of lack of interest and/or resources?
- loeg 4y agoIt wasn’t long ago that AIs couldn’t beat good human Go players. It’s progressing.
- ihatepython 4y agoIt appears that the best approach for a Go AI is simply GOFAI (Good Old Fashioned AI)
- kadoban 4y agoThere was a lot of great work done replicating and then improving on AlphaGo in an open manner. The center(s) of that were LeelaZero and then KataGo. It's a bit hard to say for sure (since you can't pit them against each other, AlphaGo isn't available), but KataGo is very likely to be well stronger than AlphaGo ever was, unless maybe they contuinued it internally (unlikely). If nothing else it's vastly more efficient to train and experiment with.
- twotwotwo 4y agoCould play against adversarial policies become part of the training alongside self-play? It's possible that would be a very slow, essentially "serial" process--adversaries don't really find a new trick until the AI can avoid the old, like with the pass attack in this paper. I'd also be curious whether training against adversaries makes the networks better or worse at general-purpose play or causes no change, but there's an argument that more-robust-but-slightly-worse-on-average is an overall improvement. It also seems relevant to looking at weird behaviors of other models, like what we're seeing as people poke at LLMs more. Some of what we're seeing might simply be that they respond weirdly to some fraction of all prompts because they're not quite baked; also, we haven't defined their goals all that well yet. But automatic searches for problematic classes of input seem like a possible tool for keeping things more on the rails.
- janalsncm 4y agoAdversarial play is going to be huge moving forward. It’s not that exciting anymore to create a generic chess engine that can beat people. But an engine specifically tuned to exploit Magnus Carlsen’s weaknesses? I bet he would pay handsomely for that. Doubly so for his next opponent.
- hgsgm 4y agoWhy would an engine be tuned to beat Carlsen, when it would crush him on default settings?
- BlueTemplar 4y agoBe afraid of ChatGPT. Very afraid : https://www.youtube.com/watch?v=rSCNW1OCk_M https://www.youtube.com/watch?v=rSCNW1OCk_M
- janalsncm 4y agoIt’s like the difference between sparring with a teacher vs sparring with Tyson Fury. You don’t learn that much from being obliterated by Stockfish. Stockfish is too strong. An engine which is tuned to beat Carlsen would demonstrate his specific weaknesses, for example specific issues with an opening he plays. For his opponents, they already try to do this. “He plays this opening a lot” is a model of Carlsen, just limited to the first few moves of the game.
- pharrington 4y agoHats off to Pelrine, but isn't AlphaGo still a cut above the other Go AIs? My gut feeling in this is that this is directly analogous to saying "high level amateur chess player defeats FIDE Grandmaster." There are over 2000 grandmasters, but only one Magnus Carlson.
- pedrosorio 4y agoThe article says: "AlphaGo is not publicly available, but the systems Pelrine prevailed against are considered on a par" I assume this means they are supposed to be stronger than the best humans. If that is the case, this is a high level amateur defeating a stockfish-like engine.
- francisduvivier 4y agoBut they don't mention Alpha Go Zero, which is on another level from Alpha Go because it's not held back by human game sample data. Now the article mentions Leela Zero and that one is an open source version of Alpha Zero where they used a community effort to do the same as Alpha Go zero described in the paper. From the status page here: https://zero.sjeng.org/ https://zero.sjeng.org/ I see that this took multiple years but was declared finished in 2021. Status page says it played games in training are 21 million while Wikipedia says that for alpha go zero it was 5 million. So you'd think it's better but actually it might just mean that they are not using they same techniques after all and so maybe worse. So question for me still is whether this technique would actually be applicable to Alpha Go Zero. But for me, this article is more a win for AI, because it means that they have found better techniques to find blind spots in models which can from now on be used in training to eventually get rid if those.
- espadrine 4y agoWhile it is possible that two trainings yield different weaknesses, the original paper is precise enough (and the authors were active enough in message boards) that it could be reproduced. Some tweaks were added as things went along. However, the fact that KataGo and ELF OpenGo, which have different tweaks but are also based on AlphaZero, are also vulnerable implies that it is very likely that AlphaGoZero and AlphaZero share the weakness. The fundamental issue is that they all do self-play, and those plays follow the same statistical distribution as a result. Weird games happen rarely, while the adversarial network forces a weird game every time. On the other hand, I am not sure whether MuZero would be vulnerable. I would guess so, but its dynamics system may learn something extra about the state of the board. Either way, it seems clear that future self-play training should include adversarial models, instead of pure self-play. (They also all share another weakness, including MuZero, which is that their MCTS algorithm doesn’t allow the neural network to gain board information from their evaluation of possible futures; only win rate. Meanwhile a human sometimes realizes, while navigating a possible sequence of moves, that one particular early move could turn the tide.)
- elif 4y agoThis reminds me of FSD behavior.. flawlessly navigate lane changes in heavy traffic around a bend at 100kph, but then get paralyzed at 0kph by a lone stop sign in my home neighborhood with no cars in sight. Maybe it spends so much time over fitting for the hard cases it just doesn't have enough basic experience.
- etothepii 4y agoThis is the standard problem with machines. When humans are given a task they won't usually follow their orders in totally absurd situations. Machines have no definition or notion of absurd.
- deleted 4y ago[deleted]
- jasfi 4y agoAI is pushing humans to do better. I remember about week ago Paul Graham tweeted that if you couldn't write a better novel than an AI then you need to seriously improve (paraphrasing).
- starchild_3001 4y agoGood job human(s)! It should be obvious to anyone who's following chess AIs: Progress seems endless; we're constantly discovering tactics and anti-tactics to defeat the latest model. That's why computer ELO is going up on a straight line. Progress in Go is admittedly a bit more obscure. Fewer people are working on it (leela go, katago etc). This article demonstrates that we can expect a similar trend.
- morelisp 4y agoI don't believe there's been any analogue to this in chess AIs in many years, if ever. Chess AIs are certainly getting stronger, but when is the last time someone found an adversarial strategy against them that even mid-level players, or a high-level player giving a full queen as a handicap, can pull off? This feels much more like "exploit" than "tactics".
- mannykannot 4y agoIn a game, especially a complete-information one that pits mind against mind, tactics are a matter of exploiting the rules to their utmost (maybe somehow tricking an opponent into breaking the rules, or letting you do so, would be an exploit, but I don't think this can be categorized as such.)
- deleted 4y ago[deleted]
- 3yanlis1bos 4y agoThis turned out to be much more Terminatoresque than I expected.
- v3ss0n 4y agoWe still have a chance , he is our champion in upcoming war against machines.
- hkopp 4y agoI am much more familiar with the development in the chess world. Since Deep Blue beat Kasparov in 1997, there was a lot of progress in the development and the architecture of chess engines. Back then, we did not have neural networks. A lot of the strategies of chess engines are now used by chess grandmasters. As an example, the value of the activity of pieces was underrated in contrast to the value of material. As there are more generations of chess engines than go engines, it would be quite interesting to pull something similar off against them. My intuition is that it maybe works against Leela, as LeelaChess basically uses only neural networks (think alphago but for chess), whereas it should not work with Stockfish, as some parts of Stockfishs evaluation function are still adjusted by hand.
- dikaio 4y agoA clear example of AI understanding the need to show a lack of creativity and intelligence so as to not alert humans of its true objective of universal domination and we, the apes that we are, are truly happy with our victory.
- slac 4y agoHow on par to AlphaGo is it? I feel it is like saying Nascar racecar beat an Indycar, which is on par with a Formula One car.
- waltman 4y agoExploiting your opponent's weaknesses seems like a perfectly cromulent strategy to me, whether your opponent is made of meat or silicon. Go humans!
- bluelightning2k 4y agoMy reading of this: the AI was trained on pro level play and needs a few lower level games to learn the first thing anyone tries?
- icare_1er 4y agoExcellent news, even the most advanced AI can still be hacked it seems.
- ngcc_hk 4y agoThere might be many techniques but is it go play : https://arstechnica.com/information-technology/2022/11/new-go-playing-trick-defeats-world-class-go-ai-but-loses-to-human-amateurs/ https://arstechnica.com/information-technology/2022/11/new-g...