9 ms·

Adversarial policies beat superhuman Go AIs (2023)

306 points by amichail 2y ago

itsnotkami 2y ago
[flagged]
- turboponyy 2y ago
  That summary is nearly identical to the abstract of the paper...
- deleted 2y ago
  [deleted]
BryanLegend 2y ago
It sometimes happens in the go world for complete amateurs to be challenging to play against, because their moves are so unpredictable and their shapes are so far away from being normal. Wildly bizarre play sometimes works.
- PartiallyTyped 2y ago
  Magnus (Carlsen, chess) does this often, he pushes people into unknown territory that they are most certainly underprepared for through new or obscure openings that complicate a position very quickly. The game then turns tactical and they eventually find themselves in a bad endgame, one against Magnus of all people.
  - ozim 2y ago
    Just in case someone thinks Magnus comes up with those openings on the spot. No he has a team that uses computers to find out those plays based on what other player played as all past matches are available. Source: I watched interview with a guy that was hired as a computer scientist consulting gig by Magnus team. It does not take away how good he is as I don’t think many people could learn to remember weird openings and win from that against grand master level players anyway.
    - rybosworld 2y ago
      I remember reading that his memory is unrivaled - so this also isn't a strategy the other top players could simply copy. In chess, there are basically three ways to evaluate moves 1) pure calculation 2) recognize the position (or a very similar one) from a previous game, and remember what the best move was 3) intuition - this one is harder to explain but, I think of it like instinct/muscle memory All the top players are good at all of these things. But some are agreed upon as much better than others. Magnus is widely agreed to have the best memory. The contender for best calculator might be Fabiano. In humans, all else being equal, memory seems to be superior to calculation, because calculation takes time. Chess engines seem to reverse this, with calculation being better than memory, because memory is expensive.
      - zdragnar 2y ago
        This is the reason why I couldn't ever get into chess, despite my dad and brother enjoying it. My intuition was crap (having not developed it) and I lacked the ability or desire to fully visualize multiple steps of the game. All that remained was rote memorization, which makes for a boring game indeed. Despite all of that, I suspect chess will long outlive my preferred entertainment of Unreal Tournament.
        Retric 2y ago
        The magic of chess is in matchmaking. I enjoy using nearly pure intuition when playing so I just use that strategy and see the same ~50/50 win percentage as most players because my ELO is based on how I play past games and there’s millions of online players across a huge range of skill levels. There’s nothing wrong with staying at 1000 or even 300 if that’s what it takes to enjoy the game. It’s only if you want to beat specific people or raise your ELO that forces you to try and optimize play.
        stackghost 2y ago
        I hate ladder systems. Winning is fun and losing is not. Why would I purposely choose to play a game/system where your win rate does not meaningfully improve as you skill up? That sounds frustrating and tedious. If I get better I want to win more often.
  - ANewFormation 2y ago
    Fwiw this is normal in chess nowadays. There was some brief era in chess where everybody was just going down the most critical lines and assuming they could outprepare their opponents, or outplay them if that didn't work out. Kasparov and Fischer are the typical examples of this style. But computers have made this less practical in modern times simply because it's so easy to lose in these sort of positions to the endless number of comp-prepped novelties which may be both objectively mediocre, but also nary impossible to play against without preparation against a prepared opponent. So a lot of preparation now a days is about getting positions that may not be the most critical test of an opening, but that lead to interesting positions and where the first player to spring a novelty isn't going to just steamroll the other guy. So in this brave new world you see things like the Berlin Defense becoming hugely popular while the Najdorf has substantially declined in popularity.
    - deleted 2y ago
      [deleted]
  - Sesse__ 2y ago
    It is true that Magnus usually prefers offbeat lines to get out of the opponent's preparation. However, they're rarely very sharp or otherwise tactically complicated; on the contrary, he excels at slow maneuvering in strategic positions (and, as you say, the endgame).
- giraffe_lady 2y ago
  Challenging in the sense that you have to work through positions you're not very practiced at. Not "challenging" in the sense that you might lose the game though.
- tasuki 2y ago
  No it does not. (Source: I'm European 4 dan. I wipe the go board with weaker players playing whatever unconventional moves they like. Likewise, I get crushed by stronger players, faster than usual if I choose unusual moves. This might work on like the double-digit kyu level...)
  - mafuy 2y ago
    I think you'd have to translate the statement into the appropriate space. Here, the unconventional moves would refer to pro-level trick play. These doubtlessly exist, and have for a long time (e.g. secret house moves in joseki 100 years ago). Once made public, they either lose their surprise advantage and become poor moves, or become standard moves themselves. So I'd say it is correct that there is a benefit to unconventional moves where punishing them requires understanding of unusual patterns. On the same note, it is certainly true that playing unconventional moves against an opponent who does now how to punish them would only bring defeat quicker :)
    - tasuki 2y ago
      Pro-level trick moves only work if you play the right continuation. When a weaker player learns a specific trick sequence, the stronger player is going to fall for it all right, but they'll play it differently than what the weaker player learned to punish. If the weaker player learns all the variations of a trick sequence... perhaps he just became stronger :) Unconventional moves can help if you want to throw off a similarly skilled opponent.
emusan 2y ago
There is hope for us lowly humans!
throwaway81523 2y ago
From 2022, revised 2023, I may have seen it before and forgotten. It is pretty interesting. I wonder how well the approach works against chess engines, at least Leela-style.
thiago_fm 2y ago
I bet that there's a similarity between this and what happens to LLM hallucinations. At some point we will realize that AI will never be perfect, it will just have much better precision than us.
- kachapopopow 2y ago
  I honestly see hallucinations as an absolute win, it's attempting to (predict/'reason') information from the training data it has.
  - voidfunc 2y ago
    I don't think I see them as a win, but they're easily dealt with. AI will need analysts at the latter stage to evaluate the outputs but that will be a relatively short-lived problem.
    - bubaumba 2y ago
      > I don't think I see them as a win Unavoidable, probably > but they're easily dealt with. AI will need analysts at the latter stage to evaluate the outputs but that will be a relatively short-lived problem That solves only to some degree. Hallucinations may happen at this stage too. Then either correct answer can get rejected or false pass through.
  - zdragnar 2y ago
    I think this is a misuse of the term hallucination. When most people talk about AI hallucinating, they're referring to output which violates some desired constraints. In the context of chess, this would be making an invalid move, or upgrading a knight to a queen. In other contexts, some real examples are fabricating court cases and legal precedent (several lawyers have gotten in trouble here), or a grocery store recipe generator recommending mixing bleach and ammonia for a delightful cocktail. None of these hallucinations are an attempt to reason about anything. This is why some people oppose using the term hallucination- it is an anthropomorphizing term that gives too much credit to the AI. We can tighten the band of errors with more data or compute efficiency or power, but in the search for generic AI, this is a dead end.
JKCalhoun 2y ago
"Not chess, Mr. Spock. Poker!"
- white_beach 2y ago
  [flagged]
- zackmorris 2y ago
  You shouldn't have been downvoted for this! http://hardboiledpoker.blogspot.com/2015/02/when-they-introduced-mr-spock-to-poker.html http://hardboiledpoker.blogspot.com/2015/02/when-they-introd... Also I feel like we're aging out of the tech scene. Or maybe that the seemingly antiquated life lessons we've learned might be needed now more than ever.
MaleekHoltkamp 2y ago
[dead]
tantalor 2y ago
> You beat him! >> No sir, it is a stalemate. > What did you do? >> I was playing for a standoff; a draw. While Kolrami was dedicated to winning, I was able to pass up obvious avenues of advancement and settle for a balance. Theoretically, I should be able to challenge him indefinitely. > Then you have beaten him! >> In the strictest sense, I did not win. > (groans) Data! >> I busted him up. > (everybody claps)
- cwillu 2y ago
  You… have made a mockery of me.
  - moffkalast 2y ago
    It is possible to commit no mistakes and still lose. That's not a weakness. That's life.
    - cwillu 2y ago
      But, knowing that he knows that we know that he knows, he might choose to return to his usual pattern.
- taneq 2y ago
  No idea if they did this on purpose but this is exactly what can happen with board game AIs when they know they will win. Unless the evaluation function explicitly promotes winning sooner they will get into an unbeatable position and then just fardle around because they have no reason to win now if they know they can do it later.
  - cjbgkagh 2y ago
    Future payoffs are almost always discounted, even if for no other reason than the future has a greater deal of uncertainty. I.e even if it was not explicit which it almost always is, it would still be implicit. Their conservative style is usually due to having a better fitness function. Humans tend to not be able to model uncertainty as accurately and this results in more aggressive play, a bird in the hand is worth two in the bush.
billforsternz 2y ago
This seems amazing at first sight. It's probably just me, but I find the paper to be very hard to understand even though I know a little bit about Go and Go AI and a lot about chess and chess AI. They seem to expend the absolute minimum amount of effort on describing what they did and how it can possibly work, unnecessarily using unexplained jargon to more or less mask the underlying message. I can almost see through the veil they've surrounded their (remarkable and quite simple?) ideas with, but not quite.
- dragontamer 2y ago
  https://slideslive.com/39006680/adversarial-policies-beat-superhuman-go-ais https://slideslive.com/39006680/adversarial-policies-beat-su... Seems to be a good intro. Go uniquely has long periods of dead-man walking, as I like to call it. Your group might be dead on turn 30, but your opponent won't formally kill the group until turn 150 or later. If your opponent knows the truth all the way back in turn30, while you are led down the wrong path for those hundreds of turns, you will almost certainly lose. This adversarial AI tricks AlphaGo/KataGo into such situations. And instead of capitalizing on it, they focus on the trickery knowing that KataGo reliably fails to understand the situation (aka it's better to make a suboptimal play to keep KataGo tricked / glitched, rather than play an optimal move that may reveal to KataGo the failure of understanding). Even with adversarial training (IE: KataGo training on this flaw), the flaw remains and it's not clear why. ------ It appears that this glitch (the cyclical group) is easy enough for an amateur player to understand (I'm ranked around 10kyu, which is estimated to be the same level of effort as 1500Elo chess. Reasonably practiced but nothing special). So it seems like I as a human (even at 10kyu) could defeat AlphaGo/KataGo with a bit of practice.
  - hinkley 2y ago
    Aji is the concept of essentially making lemonaid from lemons by using the existence of the dead stones to put pressure on the surrounding pieces and claw back some of your losses. Because they haven’t been captured yet they reduce the safety (liberties) of nearby stones. And until those are fully settled an incorrect move could rescue them, and the effort put into preventing that may cost points in the defense.
kizer 2y ago
You’d think the ability to set up elaborate tricks would imply similar knowledge of the game. And also that highly skilled AI would implicitly include adversarial strategies. Interesting result.
- dragontamer 2y ago
  The existence of KataGo and it's super-AlphaGo / AlphaZero strength is because Go players noticed that AlphaGo can't see ladders. A simple formation that even mild amateurs must learn to reach the lowest ranks. KataGo recognizes the flaw and has an explicit ladder solver written in traditional code. It seems like neural networks will never figure out ladders (!!!!!). And it's not clear why such a simple pattern is impossible for deep neural nets to figure out. I'm not surprised that there are other, deeper patterns that all of these AIs have missed.
  - bwfan123 2y ago
    >It seems like neural networks will never figure out ladders (!!!!!). And it's not clear why such a simple pattern is impossible for deep neural nets to figure out. this is very interesting (i dont play go) can you elaborate - what is the characteristic of these formations that elude AIs - is it that they dont appear in the self-training or game databases.
    - earnestinger 2y ago
      https://senseis.xmp.net/?Ladder https://senseis.xmp.net/?Ladder (Kind of like wikipedia for go players)
    - dragontamer 2y ago
      AlphaGo was trained on many human positions, all of which contain numerous ladders. I don't think anyone knows for sure, but ladders are very calculation heavy. Unlike a lot of positions where Go is played by so called instinct, a ladder switches modes into "If I do X opponent does Y so I do Z.....", almost chess like. Except it's very easy because there are only 3 or 4 options per step and really only one of those options continues the ladder. So it's this position where a chess-like tree breaks out in the game of Go but far simpler. You still need to play Go (determining the strength of the overall board and evaluate if the ladder is worth it or if ladder breaker moves are possible/reasonable). But for strictly the ladder it's a simple and somewhat tedious calculation lasting about 20 or so turns on the average. -------- The thing about ladders is that no one actually plays out a ladder. They just sit there on the board because it's rare for it to play to both players advantages (ladders are sharp: they either favor white or black by significant margins). So as, say Black, is losing the ladder, Black will NEVER play the ladder. But needs to remember that the ladder is there for the rest of the game. A ladder breaker is when Black places a piece that maybe in 15 turns (or later) will win the ladder (often while accomplishing something else). So after a ladder breaker, Black is winning the ladder and White should never play the ladder. So the threat of the ladder breaker changes the game and position severely in ways that can only be seen in the far far future, dozens or even a hundred turns from now. It's outside the realm of computer calculations but yet feasible for humans to understand the implications.
kevinwang 2y ago
[2022]
Upvoter33 2y ago
"Our results demonstrate that even superhuman AI systems may harbor surprising failure modes." This is true but really is an empty conclusion. The result has no meaning for future "superintelligences"; they may or may not have these kinds of "failure modes".
- Kapura 2y ago
  On the contrary, this is the most important part of the thesis. They are arguing not only that this AI was vulnerable to this specific attack, but that any AI model is vulnerable to attack vectors that the original builders cannot predict or preemptively guard against. if you say "well, a superintelligence won't be vulnerable" you are putting your faith in magic.
- dragontamer 2y ago
  They developed a system / algorithm that reliably defeats the most powerful Go AI, and is a simple enough system for a trained human to execute. Surely that's important? It was thought that AlphaGo and KataGo were undefeatable by humans.
- aithrowawaycomm 2y ago
  It's more a lesson about the dangers of transferring an objectively true statement: "MuZero can beat any professional Go player" to a squishy but not false statement: "MuZero is an AI which is superhuman at Go" to a statement which is basically wrong: "MuZero has superhuman intelligence in the domain of Go."
deleted 2y ago
[deleted]
vouaobrasil 2y ago
Not so encouraging. This paper will just be used to incorporate defense against adversarial strategies in Go playing AIs. A simple curiosity, but one reflective of the greater state of affairs in AI development which is rather dismal.
- brazzy 2y ago
  According to the abstract, "The core vulnerability uncovered by our attack persists even in KataGo agents adversarially trained to defend against our attack."
  - vouaobrasil 2y ago
    Well, that does not apply to future Go AIs for all of time.
    - kadoban 2y ago
      Okay, how does one protect against it then? Why would this _not_ apply to any future ones?
383toast 2y ago
NOTE: this is a july 2023 paper, the defense paper in september 2024 is https://arxiv.org/abs/2406.12843 https://arxiv.org/abs/2406.12843
- 8n4vidtmkvmk 2y ago
  > We find that though some of these defenses protect against previously discovered attacks, none withstand freshly trained adversaries.
casey2 2y ago
Reminds me of how even after deep blue chess players learned better anti computer strategies. Because the space of Go is so much larger there are likely many more anti computer strategies like this. It exploits the eval function in the same way Like chess more compute will win out, as has already been shown. I will remind everyone that elo is a measure of wins and losses not difficulty, conflating the two will lead to poor reasoning.
- snovv_crash 2y ago
  Elo also takes into account the strength of the opponent, which is a pretty good proxy for difficulty.
nilslindemann 2y ago
Here have some edge cases for chess, fortresses. The first three are "0.0" in the fourth black wins. 8/8/8/1Pk5/2Pn3p/5BbP/6P1/5K1R w - - 0 1 (white can not free the rook) 1B4r1/1p6/pPp5/P1Pp1k2/3Pp3/4Pp1p/5P1P/5K2 b - - 0 1 (the rook can not enter white's position) kqb5/1p6/1Pp5/p1Pp4/P2Pp1p1/K3PpPp/5P1B/R7 b - - 0 1 (Rook to h1. King to g1, Queen can not enter via a6) 2nnkn2/2nnnn2/2nnnn2/8/8/8/3QQQ2/3QKQ2 w - - 0 1 (the knights advance as block, so that attacked knights are protected twice) In the first both Stockfish and Lc0 think white is better (slightly on a deep ply). In the second and in the third they think black wins. Lc0 understands the fourth (applause), Stockfish does not.
- diziet 2y ago
  Links to these fortresses to those without familiarity with chess: https://lichess.org/analysis/standard/8/8/8/1Pk5/2Pn3p/5BbP/6P1/5K1R_w https://lichess.org/analysis/standard/8/8/8/1Pk5/2Pn3p/5BbP/... https://lichess.org/analysis/fromPosition/1B4r1/1p6/pPp5/P1Pp1k2/3Pp3/4Pp1p/5P1P/5K2_b_-_-_0_1 https://lichess.org/analysis/fromPosition/1B4r1/1p6/pPp5/P1P... https://lichess.org/analysis/fromPosition/kqb5/1p6/1Pp5/p1Pp4/P2Pp1p1/K3PpPp/5P1B/R7_b_-_-_0_1 https://lichess.org/analysis/fromPosition/kqb5/1p6/1Pp5/p1Pp... https://lichess.org/analysis/fromPosition/2nnkn2/2nnnn2/2nnnn2/8/8/8/3QQQ2/3QKQ2_w_-_-_0_1 https://lichess.org/analysis/fromPosition/2nnkn2/2nnnn2/2nnn...
- FartyMcFarter 2y ago
  I'm not surprised that engines aren't tuned / haven't learned to evaluate positions like the last one (and probably most of the others) - there's absolutely no way this kind of position shows up in a real chess game.
  - mtlmtlmtlmtl 2y ago
    The last one, for sure won't happen. The two with the crazy pawn chains are unlikely, but these extremely locked structures do occasionally occur. And the first one is actually pretty plausible. The situation with the king on f1 and the rook stuck in the corner is fairly thematic in some opening.It's just not well suited for engine analysis and fairly trivial for humans because we can eliminate large swathes of game tree via logic. I.e. Assuming the black bishop and knight never move, we can see the kingside pawns will never move either. And the king will only ever be able to shuffle between f1 and g1. Therefore we can deduce the rook can never make a useful move. Now the only pieces that can make meaningful moves are the two connected passed pawns on the queenside, and the light-square bishop. Assume there was no bishop. The king can simply shuffle between b6 and c5, and the pawns are contained. Can the white bishop change any of this? No, because those two squares are dark squares, and in fact all of the black pieces are on dark squares. So the white bishop is useless. Ergo, no progress can be made. We've eliminated all the possible continuations based on a very shallow search using constraint based reasoning and basic deduction. Engines can't do any of this. No one has found a generalised algorithm to do this sort of thing(it's something I spend a silly amount of time trying to think up, and I've gotten nowhere with it). All they can do is explore paths to future possible positions, assign them a heuristic evaluation. And choose the best path they find. Although, I haven't actually tried to analyse position 1 with stockfish. I feel like on sufficient depth, it should find a forced repetition. Or the 50 move rule. Though it might waste a ton of time looking at meaningless bishop moves. Naïvely, I'd expect it to do 49 pointless bishop moves and king shuffles, then move a pawn, losing it, then another 49 moves, lose the other pawn. Then finally another 50 moves until running into 50 move rule. So back of the envelope, it would need to search to 150ply before concluding it's a draw. Although pruning and might actually mean it gets there significantly faster.
zahlman 2y ago
Oh, no, not this paper again. Please see https://boardgames.stackexchange.com/questions/58127/ https://boardgames.stackexchange.com/questions/58127/ for reference. The first picture there shows a game supposedly "won by Black", due to a refusal to acknowledge that Black's stones are hopelessly dead everywhere except the top-right of the board. The "exploit" that the adversarial AI has found is, in effect, to convince KataGo to pass in this position, and then claim that White has no territory. It doesn't do this by claiming it could possibly make life with alternating play; it does so, in effect, by citing a ruleset that doesn't include the idea of removing dead stones (https://tromp.github.io/go.html https://tromp.github.io/go.html) and expects everything to be played out (using area scoring) for as long as either player isn't satisfied. Tromp comments: "As a practical shortcut, the following amendment allows dead stone removal" - but this isn't part of the formalization, and anyway the adversarial AI could just not agree, and it's up to KataGo to make pointless moves until it does. To my understanding, the formalization exists in large part because early Go programs often couldn't reliably tell when the position was fully settled (just like beginner players). It's also relevant on a theoretical level for some algorithms - which would like to know with certainty what the score is in any given position, but would theoretically have to already play Go perfectly in order to compute that. (If you're interested in why so many rulesets exist, what kinds of strange situations would make the differences matter, etc., definitely check out the work of Robert Jasiek, a relatively strong amateur European player: https://home.snafu.de/jasiek/rules.html https://home.snafu.de/jasiek/rules.html . Much of this was disregarded by the Go community at the time, because it's incredibly pedantic; but that's exactly what's necessary when it comes to rules disputes and computers.) One of the authors of the paper posted on the Stack Exchange question and argued > Now this does all feel rather contrived from a human perspective. But remember, KataGo was trained with this rule set, and configured to play with it. It doesn't know that the "human" rules of Go are any more important than Tromp-Taylor. But I don't see anything to substantiate that claim. All sorts of Go bots are happy to play against humans in online implementations of the game, under a variety of human-oriented rulesets; and they pass in natural circumstances, and then the online implementation (sometimes using a different AI) proposes group status that is almost always correct (and matches the group status that the human player modeled in order to play that way). As far as I know, if a human player deliberately tries to claim the status is wrong, an AI will either hold its ground or request to resume play and demonstrate the status more clearly. In the position shown at the Stack Exchange link, even in territory scoring without pass stones, White could afford dozens of plays inside the territory (unfairly costing 1 point each) in order to make the White stones all pass-alive and deny any mathematical possibility of the Black stones reaching that status. (Sorry, there really isn't a way to explain that last sentence better without multiple pages of the background theory I linked and/or alluded to above.)
aaron695 2y ago
[dead]
mNovak 2y ago
FYI: discussion [1] of this attack from late 2022, notably including lengthy discussion from the developer (hexahedron / lightvector) of KataGo, probably the most widely used super-human Go AI. Link is mid-thread, because the earlier version of the paper was less interesting than the revision later on. [1] https://forums.online-go.com/t/potential-rank-inflation-on-ogs-or-how-to-beat-katago-with-one-simple-trick-free/45380/118 https://forums.online-go.com/t/potential-rank-inflation-on-o...
leoc 2y ago
Are there or may there be similar “attacks” on the LLM chatbots?
- canjobear 2y ago
  Yes, this is an area of active research. For example https://lilianweng.github.io/posts/2023-10-25-adv-attack-llm/ https://lilianweng.github.io/posts/2023-10-25-adv-attack-llm...

306 points by amichail 2y ago

itsnotkami 2y ago

[flagged]

turboponyy 2y ago
That summary is nearly identical to the abstract of the paper...
deleted 2y ago
[deleted]

BryanLegend 2y ago

It sometimes happens in the go world for complete amateurs to be challenging to play against, because their moves are so unpredictable and their shapes are so far away from being normal. Wildly bizarre play sometimes works.

PartiallyTyped 2y ago
Magnus (Carlsen, chess) does this often, he pushes people into unknown territory that they are most certainly underprepared for through new or obscure openings that complicate a position very quickly. The game then turns tactical and they eventually find themselves in a bad endgame, one against Magnus of all people.
- ozim 2y ago
  Just in case someone thinks Magnus comes up with those openings on the spot. No he has a team that uses computers to find out those plays based on what other player played as all past matches are available. Source: I watched interview with a guy that was hired as a computer scientist consulting gig by Magnus team. It does not take away how good he is as I don’t think many people could learn to remember weird openings and win from that against grand master level players anyway.
  - rybosworld 2y ago
    I remember reading that his memory is unrivaled - so this also isn't a strategy the other top players could simply copy. In chess, there are basically three ways to evaluate moves 1) pure calculation 2) recognize the position (or a very similar one) from a previous game, and remember what the best move was 3) intuition - this one is harder to explain but, I think of it like instinct/muscle memory All the top players are good at all of these things. But some are agreed upon as much better than others. Magnus is widely agreed to have the best memory. The contender for best calculator might be Fabiano. In humans, all else being equal, memory seems to be superior to calculation, because calculation takes time. Chess engines seem to reverse this, with calculation being better than memory, because memory is expensive.
    - zdragnar 2y ago
      This is the reason why I couldn't ever get into chess, despite my dad and brother enjoying it. My intuition was crap (having not developed it) and I lacked the ability or desire to fully visualize multiple steps of the game. All that remained was rote memorization, which makes for a boring game indeed. Despite all of that, I suspect chess will long outlive my preferred entertainment of Unreal Tournament.
      - Retric 2y ago
        The magic of chess is in matchmaking. I enjoy using nearly pure intuition when playing so I just use that strategy and see the same ~50/50 win percentage as most players because my ELO is based on how I play past games and there’s millions of online players across a huge range of skill levels. There’s nothing wrong with staying at 1000 or even 300 if that’s what it takes to enjoy the game. It’s only if you want to beat specific people or raise your ELO that forces you to try and optimize play.
        stackghost 2y ago
        I hate ladder systems. Winning is fun and losing is not. Why would I purposely choose to play a game/system where your win rate does not meaningfully improve as you skill up? That sounds frustrating and tedious. If I get better I want to win more often.
- ANewFormation 2y ago
  Fwiw this is normal in chess nowadays. There was some brief era in chess where everybody was just going down the most critical lines and assuming they could outprepare their opponents, or outplay them if that didn't work out. Kasparov and Fischer are the typical examples of this style. But computers have made this less practical in modern times simply because it's so easy to lose in these sort of positions to the endless number of comp-prepped novelties which may be both objectively mediocre, but also nary impossible to play against without preparation against a prepared opponent. So a lot of preparation now a days is about getting positions that may not be the most critical test of an opening, but that lead to interesting positions and where the first player to spring a novelty isn't going to just steamroll the other guy. So in this brave new world you see things like the Berlin Defense becoming hugely popular while the Najdorf has substantially declined in popularity.
  - deleted 2y ago
    [deleted]
- Sesse__ 2y ago
  It is true that Magnus usually prefers offbeat lines to get out of the opponent's preparation. However, they're rarely very sharp or otherwise tactically complicated; on the contrary, he excels at slow maneuvering in strategic positions (and, as you say, the endgame).
giraffe_lady 2y ago
Challenging in the sense that you have to work through positions you're not very practiced at. Not "challenging" in the sense that you might lose the game though.
tasuki 2y ago
No it does not. (Source: I'm European 4 dan. I wipe the go board with weaker players playing whatever unconventional moves they like. Likewise, I get crushed by stronger players, faster than usual if I choose unusual moves. This might work on like the double-digit kyu level...)
- mafuy 2y ago
  I think you'd have to translate the statement into the appropriate space. Here, the unconventional moves would refer to pro-level trick play. These doubtlessly exist, and have for a long time (e.g. secret house moves in joseki 100 years ago). Once made public, they either lose their surprise advantage and become poor moves, or become standard moves themselves. So I'd say it is correct that there is a benefit to unconventional moves where punishing them requires understanding of unusual patterns. On the same note, it is certainly true that playing unconventional moves against an opponent who does now how to punish them would only bring defeat quicker :)
  - tasuki 2y ago
    Pro-level trick moves only work if you play the right continuation. When a weaker player learns a specific trick sequence, the stronger player is going to fall for it all right, but they'll play it differently than what the weaker player learned to punish. If the weaker player learns all the variations of a trick sequence... perhaps he just became stronger :) Unconventional moves can help if you want to throw off a similarly skilled opponent.

emusan 2y ago

There is hope for us lowly humans!

throwaway81523 2y ago

From 2022, revised 2023, I may have seen it before and forgotten. It is pretty interesting. I wonder how well the approach works against chess engines, at least Leela-style.

thiago_fm 2y ago

I bet that there's a similarity between this and what happens to LLM hallucinations. At some point we will realize that AI will never be perfect, it will just have much better precision than us.

kachapopopow 2y ago
I honestly see hallucinations as an absolute win, it's attempting to (predict/'reason') information from the training data it has.
- voidfunc 2y ago
  I don't think I see them as a win, but they're easily dealt with. AI will need analysts at the latter stage to evaluate the outputs but that will be a relatively short-lived problem.
  - bubaumba 2y ago
    > I don't think I see them as a win Unavoidable, probably > but they're easily dealt with. AI will need analysts at the latter stage to evaluate the outputs but that will be a relatively short-lived problem That solves only to some degree. Hallucinations may happen at this stage too. Then either correct answer can get rejected or false pass through.
- zdragnar 2y ago
  I think this is a misuse of the term hallucination. When most people talk about AI hallucinating, they're referring to output which violates some desired constraints. In the context of chess, this would be making an invalid move, or upgrading a knight to a queen. In other contexts, some real examples are fabricating court cases and legal precedent (several lawyers have gotten in trouble here), or a grocery store recipe generator recommending mixing bleach and ammonia for a delightful cocktail. None of these hallucinations are an attempt to reason about anything. This is why some people oppose using the term hallucination- it is an anthropomorphizing term that gives too much credit to the AI. We can tighten the band of errors with more data or compute efficiency or power, but in the search for generic AI, this is a dead end.

JKCalhoun 2y ago

"Not chess, Mr. Spock. Poker!"

white_beach 2y ago
[flagged]
zackmorris 2y ago
You shouldn't have been downvoted for this! http://hardboiledpoker.blogspot.com/2015/02/when-they-introduced-mr-spock-to-poker.html http://hardboiledpoker.blogspot.com/2015/02/when-they-introd... Also I feel like we're aging out of the tech scene. Or maybe that the seemingly antiquated life lessons we've learned might be needed now more than ever.

MaleekHoltkamp 2y ago

[dead]

tantalor 2y ago

> You beat him! >> No sir, it is a stalemate. > What did you do? >> I was playing for a standoff; a draw. While Kolrami was dedicated to winning, I was able to pass up obvious avenues of advancement and settle for a balance. Theoretically, I should be able to challenge him indefinitely. > Then you have beaten him! >> In the strictest sense, I did not win. > (groans) Data! >> I busted him up. > (everybody claps)

cwillu 2y ago
You… have made a mockery of me.
- moffkalast 2y ago
  It is possible to commit no mistakes and still lose. That's not a weakness. That's life.
  - cwillu 2y ago
    But, knowing that he knows that we know that he knows, he might choose to return to his usual pattern.
taneq 2y ago
No idea if they did this on purpose but this is exactly what can happen with board game AIs when they know they will win. Unless the evaluation function explicitly promotes winning sooner they will get into an unbeatable position and then just fardle around because they have no reason to win now if they know they can do it later.
- cjbgkagh 2y ago
  Future payoffs are almost always discounted, even if for no other reason than the future has a greater deal of uncertainty. I.e even if it was not explicit which it almost always is, it would still be implicit. Their conservative style is usually due to having a better fitness function. Humans tend to not be able to model uncertainty as accurately and this results in more aggressive play, a bird in the hand is worth two in the bush.

billforsternz 2y ago

This seems amazing at first sight. It's probably just me, but I find the paper to be very hard to understand even though I know a little bit about Go and Go AI and a lot about chess and chess AI. They seem to expend the absolute minimum amount of effort on describing what they did and how it can possibly work, unnecessarily using unexplained jargon to more or less mask the underlying message. I can almost see through the veil they've surrounded their (remarkable and quite simple?) ideas with, but not quite.

dragontamer 2y ago
https://slideslive.com/39006680/adversarial-policies-beat-superhuman-go-ais https://slideslive.com/39006680/adversarial-policies-beat-su... Seems to be a good intro. Go uniquely has long periods of dead-man walking, as I like to call it. Your group might be dead on turn 30, but your opponent won't formally kill the group until turn 150 or later. If your opponent knows the truth all the way back in turn30, while you are led down the wrong path for those hundreds of turns, you will almost certainly lose. This adversarial AI tricks AlphaGo/KataGo into such situations. And instead of capitalizing on it, they focus on the trickery knowing that KataGo reliably fails to understand the situation (aka it's better to make a suboptimal play to keep KataGo tricked / glitched, rather than play an optimal move that may reveal to KataGo the failure of understanding). Even with adversarial training (IE: KataGo training on this flaw), the flaw remains and it's not clear why. ------ It appears that this glitch (the cyclical group) is easy enough for an amateur player to understand (I'm ranked around 10kyu, which is estimated to be the same level of effort as 1500Elo chess. Reasonably practiced but nothing special). So it seems like I as a human (even at 10kyu) could defeat AlphaGo/KataGo with a bit of practice.
- hinkley 2y ago
  Aji is the concept of essentially making lemonaid from lemons by using the existence of the dead stones to put pressure on the surrounding pieces and claw back some of your losses. Because they haven’t been captured yet they reduce the safety (liberties) of nearby stones. And until those are fully settled an incorrect move could rescue them, and the effort put into preventing that may cost points in the defense.

kizer 2y ago

You’d think the ability to set up elaborate tricks would imply similar knowledge of the game. And also that highly skilled AI would implicitly include adversarial strategies. Interesting result.

dragontamer 2y ago
The existence of KataGo and it's super-AlphaGo / AlphaZero strength is because Go players noticed that AlphaGo can't see ladders. A simple formation that even mild amateurs must learn to reach the lowest ranks. KataGo recognizes the flaw and has an explicit ladder solver written in traditional code. It seems like neural networks will never figure out ladders (!!!!!). And it's not clear why such a simple pattern is impossible for deep neural nets to figure out. I'm not surprised that there are other, deeper patterns that all of these AIs have missed.
- bwfan123 2y ago
  >It seems like neural networks will never figure out ladders (!!!!!). And it's not clear why such a simple pattern is impossible for deep neural nets to figure out. this is very interesting (i dont play go) can you elaborate - what is the characteristic of these formations that elude AIs - is it that they dont appear in the self-training or game databases.
  - earnestinger 2y ago
    https://senseis.xmp.net/?Ladder https://senseis.xmp.net/?Ladder (Kind of like wikipedia for go players)
  - dragontamer 2y ago
    AlphaGo was trained on many human positions, all of which contain numerous ladders. I don't think anyone knows for sure, but ladders are very calculation heavy. Unlike a lot of positions where Go is played by so called instinct, a ladder switches modes into "If I do X opponent does Y so I do Z.....", almost chess like. Except it's very easy because there are only 3 or 4 options per step and really only one of those options continues the ladder. So it's this position where a chess-like tree breaks out in the game of Go but far simpler. You still need to play Go (determining the strength of the overall board and evaluate if the ladder is worth it or if ladder breaker moves are possible/reasonable). But for strictly the ladder it's a simple and somewhat tedious calculation lasting about 20 or so turns on the average. -------- The thing about ladders is that no one actually plays out a ladder. They just sit there on the board because it's rare for it to play to both players advantages (ladders are sharp: they either favor white or black by significant margins). So as, say Black, is losing the ladder, Black will NEVER play the ladder. But needs to remember that the ladder is there for the rest of the game. A ladder breaker is when Black places a piece that maybe in 15 turns (or later) will win the ladder (often while accomplishing something else). So after a ladder breaker, Black is winning the ladder and White should never play the ladder. So the threat of the ladder breaker changes the game and position severely in ways that can only be seen in the far far future, dozens or even a hundred turns from now. It's outside the realm of computer calculations but yet feasible for humans to understand the implications.

kevinwang 2y ago

[2022]

Upvoter33 2y ago

"Our results demonstrate that even superhuman AI systems may harbor surprising failure modes." This is true but really is an empty conclusion. The result has no meaning for future "superintelligences"; they may or may not have these kinds of "failure modes".

Kapura 2y ago
On the contrary, this is the most important part of the thesis. They are arguing not only that this AI was vulnerable to this specific attack, but that any AI model is vulnerable to attack vectors that the original builders cannot predict or preemptively guard against. if you say "well, a superintelligence won't be vulnerable" you are putting your faith in magic.
dragontamer 2y ago
They developed a system / algorithm that reliably defeats the most powerful Go AI, and is a simple enough system for a trained human to execute. Surely that's important? It was thought that AlphaGo and KataGo were undefeatable by humans.
aithrowawaycomm 2y ago
It's more a lesson about the dangers of transferring an objectively true statement: "MuZero can beat any professional Go player" to a squishy but not false statement: "MuZero is an AI which is superhuman at Go" to a statement which is basically wrong: "MuZero has superhuman intelligence in the domain of Go."

deleted 2y ago

[deleted]

vouaobrasil 2y ago

Not so encouraging. This paper will just be used to incorporate defense against adversarial strategies in Go playing AIs. A simple curiosity, but one reflective of the greater state of affairs in AI development which is rather dismal.

brazzy 2y ago
According to the abstract, "The core vulnerability uncovered by our attack persists even in KataGo agents adversarially trained to defend against our attack."
- vouaobrasil 2y ago
  Well, that does not apply to future Go AIs for all of time.
  - kadoban 2y ago
    Okay, how does one protect against it then? Why would this _not_ apply to any future ones?

383toast 2y ago

NOTE: this is a july 2023 paper, the defense paper in september 2024 is https://arxiv.org/abs/2406.12843 https://arxiv.org/abs/2406.12843

8n4vidtmkvmk 2y ago
> We find that though some of these defenses protect against previously discovered attacks, none withstand freshly trained adversaries.

casey2 2y ago

Reminds me of how even after deep blue chess players learned better anti computer strategies. Because the space of Go is so much larger there are likely many more anti computer strategies like this. It exploits the eval function in the same way Like chess more compute will win out, as has already been shown. I will remind everyone that elo is a measure of wins and losses not difficulty, conflating the two will lead to poor reasoning.

snovv_crash 2y ago
Elo also takes into account the strength of the opponent, which is a pretty good proxy for difficulty.

nilslindemann 2y ago

Here have some edge cases for chess, fortresses. The first three are "0.0" in the fourth black wins. 8/8/8/1Pk5/2Pn3p/5BbP/6P1/5K1R w - - 0 1 (white can not free the rook) 1B4r1/1p6/pPp5/P1Pp1k2/3Pp3/4Pp1p/5P1P/5K2 b - - 0 1 (the rook can not enter white's position) kqb5/1p6/1Pp5/p1Pp4/P2Pp1p1/K3PpPp/5P1B/R7 b - - 0 1 (Rook to h1. King to g1, Queen can not enter via a6) 2nnkn2/2nnnn2/2nnnn2/8/8/8/3QQQ2/3QKQ2 w - - 0 1 (the knights advance as block, so that attacked knights are protected twice) In the first both Stockfish and Lc0 think white is better (slightly on a deep ply). In the second and in the third they think black wins. Lc0 understands the fourth (applause), Stockfish does not.

diziet 2y ago
Links to these fortresses to those without familiarity with chess: https://lichess.org/analysis/standard/8/8/8/1Pk5/2Pn3p/5BbP/6P1/5K1R_w https://lichess.org/analysis/standard/8/8/8/1Pk5/2Pn3p/5BbP/... https://lichess.org/analysis/fromPosition/1B4r1/1p6/pPp5/P1Pp1k2/3Pp3/4Pp1p/5P1P/5K2_b_-_-_0_1 https://lichess.org/analysis/fromPosition/1B4r1/1p6/pPp5/P1P... https://lichess.org/analysis/fromPosition/kqb5/1p6/1Pp5/p1Pp4/P2Pp1p1/K3PpPp/5P1B/R7_b_-_-_0_1 https://lichess.org/analysis/fromPosition/kqb5/1p6/1Pp5/p1Pp... https://lichess.org/analysis/fromPosition/2nnkn2/2nnnn2/2nnnn2/8/8/8/3QQQ2/3QKQ2_w_-_-_0_1 https://lichess.org/analysis/fromPosition/2nnkn2/2nnnn2/2nnn...
FartyMcFarter 2y ago
I'm not surprised that engines aren't tuned / haven't learned to evaluate positions like the last one (and probably most of the others) - there's absolutely no way this kind of position shows up in a real chess game.
- mtlmtlmtlmtl 2y ago
  The last one, for sure won't happen. The two with the crazy pawn chains are unlikely, but these extremely locked structures do occasionally occur. And the first one is actually pretty plausible. The situation with the king on f1 and the rook stuck in the corner is fairly thematic in some opening.It's just not well suited for engine analysis and fairly trivial for humans because we can eliminate large swathes of game tree via logic. I.e. Assuming the black bishop and knight never move, we can see the kingside pawns will never move either. And the king will only ever be able to shuffle between f1 and g1. Therefore we can deduce the rook can never make a useful move. Now the only pieces that can make meaningful moves are the two connected passed pawns on the queenside, and the light-square bishop. Assume there was no bishop. The king can simply shuffle between b6 and c5, and the pawns are contained. Can the white bishop change any of this? No, because those two squares are dark squares, and in fact all of the black pieces are on dark squares. So the white bishop is useless. Ergo, no progress can be made. We've eliminated all the possible continuations based on a very shallow search using constraint based reasoning and basic deduction. Engines can't do any of this. No one has found a generalised algorithm to do this sort of thing(it's something I spend a silly amount of time trying to think up, and I've gotten nowhere with it). All they can do is explore paths to future possible positions, assign them a heuristic evaluation. And choose the best path they find. Although, I haven't actually tried to analyse position 1 with stockfish. I feel like on sufficient depth, it should find a forced repetition. Or the 50 move rule. Though it might waste a ton of time looking at meaningless bishop moves. Naïvely, I'd expect it to do 49 pointless bishop moves and king shuffles, then move a pawn, losing it, then another 49 moves, lose the other pawn. Then finally another 50 moves until running into 50 move rule. So back of the envelope, it would need to search to 150ply before concluding it's a draw. Although pruning and might actually mean it gets there significantly faster.

zahlman 2y ago

Oh, no, not this paper again. Please see https://boardgames.stackexchange.com/questions/58127/ https://boardgames.stackexchange.com/questions/58127/ for reference. The first picture there shows a game supposedly "won by Black", due to a refusal to acknowledge that Black's stones are hopelessly dead everywhere except the top-right of the board. The "exploit" that the adversarial AI has found is, in effect, to convince KataGo to pass in this position, and then claim that White has no territory. It doesn't do this by claiming it could possibly make life with alternating play; it does so, in effect, by citing a ruleset that doesn't include the idea of removing dead stones (https://tromp.github.io/go.html https://tromp.github.io/go.html) and expects everything to be played out (using area scoring) for as long as either player isn't satisfied. Tromp comments: "As a practical shortcut, the following amendment allows dead stone removal" - but this isn't part of the formalization, and anyway the adversarial AI could just not agree, and it's up to KataGo to make pointless moves until it does. To my understanding, the formalization exists in large part because early Go programs often couldn't reliably tell when the position was fully settled (just like beginner players). It's also relevant on a theoretical level for some algorithms - which would like to know with certainty what the score is in any given position, but would theoretically have to already play Go perfectly in order to compute that. (If you're interested in why so many rulesets exist, what kinds of strange situations would make the differences matter, etc., definitely check out the work of Robert Jasiek, a relatively strong amateur European player: https://home.snafu.de/jasiek/rules.html https://home.snafu.de/jasiek/rules.html . Much of this was disregarded by the Go community at the time, because it's incredibly pedantic; but that's exactly what's necessary when it comes to rules disputes and computers.) One of the authors of the paper posted on the Stack Exchange question and argued > Now this does all feel rather contrived from a human perspective. But remember, KataGo was trained with this rule set, and configured to play with it. It doesn't know that the "human" rules of Go are any more important than Tromp-Taylor. But I don't see anything to substantiate that claim. All sorts of Go bots are happy to play against humans in online implementations of the game, under a variety of human-oriented rulesets; and they pass in natural circumstances, and then the online implementation (sometimes using a different AI) proposes group status that is almost always correct (and matches the group status that the human player modeled in order to play that way). As far as I know, if a human player deliberately tries to claim the status is wrong, an AI will either hold its ground or request to resume play and demonstrate the status more clearly. In the position shown at the Stack Exchange link, even in territory scoring without pass stones, White could afford dozens of plays inside the territory (unfairly costing 1 point each) in order to make the White stones all pass-alive and deny any mathematical possibility of the Black stones reaching that status. (Sorry, there really isn't a way to explain that last sentence better without multiple pages of the background theory I linked and/or alluded to above.)

aaron695 2y ago

[dead]

mNovak 2y ago

FYI: discussion [1] of this attack from late 2022, notably including lengthy discussion from the developer (hexahedron / lightvector) of KataGo, probably the most widely used super-human Go AI. Link is mid-thread, because the earlier version of the paper was less interesting than the revision later on. [1] https://forums.online-go.com/t/potential-rank-inflation-on-ogs-or-how-to-beat-katago-with-one-simple-trick-free/45380/118 https://forums.online-go.com/t/potential-rank-inflation-on-o...

leoc 2y ago

Are there or may there be similar “attacks” on the LLM chatbots?

canjobear 2y ago
Yes, this is an area of active research. For example https://lilianweng.github.io/posts/2023-10-25-adv-attack-llm/ https://lilianweng.github.io/posts/2023-10-25-adv-attack-llm...