4 ms·

A human just defeated an AI in Go. Here's why that matters

66 points by amadeuspagel 3y ago

version_five 3y ago
Good article. This is a great example of "AI"'s inability to extrapolate. Yes, it's able to find amazing shortcuts that no human could working in-distribution, but it has no understanding beyond that, and so can fail as outlined. I wish people would bear this in mind more as they pontificate about how we're near AGI and all that implies. It's all Eliza with just a really big set of phrases it can respond to. Doesn't mean it isn't useful but it's so much different than all the silly musings about sentience and whatnot.
- pipo234 3y ago
  +1 Arguably "AI" is already plenty general enough, it's just not very intelligent.
- worrycue 3y ago
  Guess this is similar to ChatGPT and “ davidjl” - https://news.ycombinator.com/item?id=36242914 https://news.ycombinator.com/item?id=36242914 - and tricking image categorisers with specially designed noise. Today’s AI seem to have all sort of unexpected failure modes which can be exploited by adversarial attacks.
  - DemeterFarm 3y ago
    [dead]
  - Zetobal 3y ago
    Like humans?
    - worrycue 3y ago
      The good thing is we have a lot of knowledge on human failure modes though and have planned around most of them. AIs are currently an unknown quality when it comes to their failure modes* and we are often caught by surprise when they fail at something that we expect they should be able to do because we don't fail at it. * Given how little we understand about how it all actually works in practice, I have a feeling we will never be able to fully grasp their failure modes unlike traditional computing.
- rcme 3y ago
  I think it depends on what you mean by understanding. KataGo taught itself to play go by explicitly deprioritizing “losing” strategies. This means it didn’t play many amateur strategies because they were lost early in the training. This is hard for a human to understand because humans all generally share a learning curve going from beginning to amateur to expert. So all humans have more experience with “losing” techniques. Basically what I’m saying is, it might be that the training scheme of this AI explicitly prioritized having little understanding of these specific tactics, which is different than not having any understanding.
karmakaze 3y ago
Quick info: > This is where Kellin Pelrine steps in. Pelrine is a good player, but an amateur. Specifically, he's one level below the top amateur ranking. He's also one of the study authors, so he was well aware of the vulnerabilities of KataGo, so he thought why not try his own hand? > Apparently, it was surprisingly easy to find a way to defeat AI by exploiting its weakness. Pelrine managed to beat KataGo 14 out of 15 times. For comparison, KataGo beat AlphaGo 100 times out of 100, and AlphaGo beat mankind's best player 4-1. One caveat is that this is vs AlphaGo which was trained on pro games, so has biases in what it's been trained on. I wonder if it would do as well against AlphaZero which is only from self-play. The article says KataGo is the strongest, so stronger than AlphaZero. Because of non-transitivity, it's also not known if the same weaknesses exist in AlphaZero. The warning for weaknesses in AI/ML is still of great importance when applying the tech.
- firstlink 3y ago
  Analogous sui generis weaknesses in human cognition are the far greater threat.
- pritambaral 3y ago
  From TFA: "... KataGo, a machine-learning algorithm that taught itself how to play, ...".
  - karmakaze 3y ago
    My question wasn't about KataGo. It was about it beating AlphaGo and could it also defeat AlphaZero 100 times out of 100. The fact that both KataGo and AlphaGo were self-taught doesn't answer if KataGo would defeat AlphaZero. I'm only taking TFA's assertion that it's the strongest with only a reference to AlphaGo.
- paulddraper 3y ago
  Yes the trick used against KataGo would likely not have worked on AlphaGo. But then again ... It took humans not AlphaGo to find the exploit. Still a victory for the meatbags.
PakG1 3y ago
This reminds me of the anime Hikaru no Go. In one episode, prodigy named Touya Akira (the main rival of the main character) is being bullied by his schoolmates. They challenge him to a 3-board match (he has to play three games simultaneously, each against a different person) with his eyes closed. Because of his reputation, he accepts the challenge. He calls out his moves and the opposing player places the stone. The opposing players also call out their own moves so that Akira can know what's happening in each of the games. he has to construct and track what the three boards look like constantly using only his memory. The one game of the three where he starts losing is the one where the opposing player is the least experienced. It's not the skill that starts defeating Akira. Rather, it's the idiocy of the amateur who doesn't know what he's doing, making it more difficult for Akira to track what the game actually looks like, because it's harder to remember and track nonsensical moves.
- Someone 3y ago
  In “Thought And Choice In Chess”, A.D. de Groot (https://en.wikipedia.org/wiki/Adriaan_de_Groot https://en.wikipedia.org/wiki/Adriaan_de_Groot) noticed that for chess. He asked good chess players (including Euwe and Alekhine) to take a short look at a board and then replicate it from memory. They were better at doing that for real games than for artificial setups. Also, their errors tended to be non-essential. When they misplaced pawns, for example, that wouldn’t affect the plan to pick. He also asked amateur-level players. I don’t remember whether they made more errors, but they did make more essential errors, and for them, the difference between ‘logical’ and ‘illogical’ board setups was smaller.
- aio2 3y ago
  It reminds me of one of Murphy's Law saying that an amateur is more dangerous than a professional since they are more unpredictable.
- aleksiy123 3y ago
  When I used to play a lot of CS:GO this effect was very obvious. Ranking up was essentially learning to counter strategies at a certain skill level. But sometimes when you played much weaker players your playstyle would be over optimized to predict/counter a different style of play. The solution was usually to stop overthinking it and just rely on your raw aim mechanics to beat them.
ngcc_hk 3y ago
I thought both alpha go and its zero ver. retired. Is that an issue only with KataGo?
alan-crowe 3y ago
The Go book The Treasure Chest Enigma by Noriyuki Nakayama starts with an essay that gives the book its title. The Treasure Chest Enigma is a life-and-death problem too hard for professional Go players, but easy for amateurs. A real life problem that caused much consternation and merriment in the Nihon Ki-in in 1958. It relies on the technique that is at the heart of this paper. There is a small dead group completely surrounded by a larger group. But the larger group runs out of liberties and the small dead group comes back to life when the big group is captured. This kind of thing never comes up in professional play. (The game that it came from was part of the qualifying tournament for professional shodan; Mizuno Hiroshi and Noriyuki Nakayama were not yet professional players.) All the top players had drilled so hard on the life and death problems that actually arise in professional play that they had developed a peculiar blind spot around capturing races with one group completely surrounded by another. My expectation is that the first fifty years of Artificial Intelligence disasters will be dominated by humans being over-impressed by the intelligence of AI systems and giving them responsibilities that they are not read for. This article https://www.gbnews.com/news/ai-latest-news-judge-legal-decisions https://www.gbnews.com/news/ai-latest-news-judge-legal-decis... already speculating that "AI machines 'could soon make legally binding decisions' in UK, says senior judge" is the kind of thing I fear. I don't know what to make of The Treasure Chest Enigma. Theory one: it tells us that we are already coping successfully with this problem in human intelligence. Yes, AI has weird mental blind spots, but people do too and there is no need to worry. Theory two: Human mental blind spots are smaller and more obscure that the mental blind spots of current AI. There is plenty of quantitative difference so that we can legitimately worry that AI blind spots will cause unexpected disasters. We will be taken by surprise because we are used to human intelligence with its small, benign blind spots. The significance of The Treasure Chest Enigma is that it exists, which tells us that mental blind spots are inherent to intelligence. We will not solve the problem of AI having large, dangerous mental blind spots easily. We can only hope to shrink the problem down to an ineradicable core.
aaron695 3y ago
[dead]
_8j50 3y ago
> ...there could be fringe situations where it behaves badly I think the more important takeaway is that we humans don't know those fringe situations or corner cases beforehand. No way around finding the problem in prod.

66 points by amadeuspagel 3y ago

Good article. This is a great example of "AI"'s inability to extrapolate. Yes, it's able to find amazing shortcuts that no human could working in-distribution, but it has no understanding beyond that, and so can fail as outlined. I wish people would bear this in mind more as they pontificate about how we're near AGI and all that implies. It's all Eliza with just a really big set of phrases it can respond to. Doesn't mean it isn't useful but it's so much different than all the silly musings about sentience and whatnot.

pipo234 3y ago
+1 Arguably "AI" is already plenty general enough, it's just not very intelligent.
worrycue 3y ago
Guess this is similar to ChatGPT and “ davidjl” - https://news.ycombinator.com/item?id=36242914 https://news.ycombinator.com/item?id=36242914 - and tricking image categorisers with specially designed noise. Today’s AI seem to have all sort of unexpected failure modes which can be exploited by adversarial attacks.
- DemeterFarm 3y ago
  [dead]
- Zetobal 3y ago
  Like humans?
  - worrycue 3y ago
    The good thing is we have a lot of knowledge on human failure modes though and have planned around most of them. AIs are currently an unknown quality when it comes to their failure modes* and we are often caught by surprise when they fail at something that we expect they should be able to do because we don't fail at it. * Given how little we understand about how it all actually works in practice, I have a feeling we will never be able to fully grasp their failure modes unlike traditional computing.
rcme 3y ago
I think it depends on what you mean by understanding. KataGo taught itself to play go by explicitly deprioritizing “losing” strategies. This means it didn’t play many amateur strategies because they were lost early in the training. This is hard for a human to understand because humans all generally share a learning curve going from beginning to amateur to expert. So all humans have more experience with “losing” techniques. Basically what I’m saying is, it might be that the training scheme of this AI explicitly prioritized having little understanding of these specific tactics, which is different than not having any understanding.

karmakaze 3y ago

Quick info: > This is where Kellin Pelrine steps in. Pelrine is a good player, but an amateur. Specifically, he's one level below the top amateur ranking. He's also one of the study authors, so he was well aware of the vulnerabilities of KataGo, so he thought why not try his own hand? > Apparently, it was surprisingly easy to find a way to defeat AI by exploiting its weakness. Pelrine managed to beat KataGo 14 out of 15 times. For comparison, KataGo beat AlphaGo 100 times out of 100, and AlphaGo beat mankind's best player 4-1. One caveat is that this is vs AlphaGo which was trained on pro games, so has biases in what it's been trained on. I wonder if it would do as well against AlphaZero which is only from self-play. The article says KataGo is the strongest, so stronger than AlphaZero. Because of non-transitivity, it's also not known if the same weaknesses exist in AlphaZero. The warning for weaknesses in AI/ML is still of great importance when applying the tech.

firstlink 3y ago
Analogous sui generis weaknesses in human cognition are the far greater threat.
pritambaral 3y ago
From TFA: "... KataGo, a machine-learning algorithm that taught itself how to play, ...".
- karmakaze 3y ago
  My question wasn't about KataGo. It was about it beating AlphaGo and could it also defeat AlphaZero 100 times out of 100. The fact that both KataGo and AlphaGo were self-taught doesn't answer if KataGo would defeat AlphaZero. I'm only taking TFA's assertion that it's the strongest with only a reference to AlphaGo.
paulddraper 3y ago
Yes the trick used against KataGo would likely not have worked on AlphaGo. But then again ... It took humans not AlphaGo to find the exploit. Still a victory for the meatbags.

PakG1 3y ago

This reminds me of the anime Hikaru no Go. In one episode, prodigy named Touya Akira (the main rival of the main character) is being bullied by his schoolmates. They challenge him to a 3-board match (he has to play three games simultaneously, each against a different person) with his eyes closed. Because of his reputation, he accepts the challenge. He calls out his moves and the opposing player places the stone. The opposing players also call out their own moves so that Akira can know what's happening in each of the games. he has to construct and track what the three boards look like constantly using only his memory. The one game of the three where he starts losing is the one where the opposing player is the least experienced. It's not the skill that starts defeating Akira. Rather, it's the idiocy of the amateur who doesn't know what he's doing, making it more difficult for Akira to track what the game actually looks like, because it's harder to remember and track nonsensical moves.

Someone 3y ago
In “Thought And Choice In Chess”, A.D. de Groot (https://en.wikipedia.org/wiki/Adriaan_de_Groot https://en.wikipedia.org/wiki/Adriaan_de_Groot) noticed that for chess. He asked good chess players (including Euwe and Alekhine) to take a short look at a board and then replicate it from memory. They were better at doing that for real games than for artificial setups. Also, their errors tended to be non-essential. When they misplaced pawns, for example, that wouldn’t affect the plan to pick. He also asked amateur-level players. I don’t remember whether they made more errors, but they did make more essential errors, and for them, the difference between ‘logical’ and ‘illogical’ board setups was smaller.
aio2 3y ago
It reminds me of one of Murphy's Law saying that an amateur is more dangerous than a professional since they are more unpredictable.
aleksiy123 3y ago
When I used to play a lot of CS:GO this effect was very obvious. Ranking up was essentially learning to counter strategies at a certain skill level. But sometimes when you played much weaker players your playstyle would be over optimized to predict/counter a different style of play. The solution was usually to stop overthinking it and just rely on your raw aim mechanics to beat them.

ngcc_hk 3y ago

I thought both alpha go and its zero ver. retired. Is that an issue only with KataGo?

alan-crowe 3y ago

The Go book The Treasure Chest Enigma by Noriyuki Nakayama starts with an essay that gives the book its title. The Treasure Chest Enigma is a life-and-death problem too hard for professional Go players, but easy for amateurs. A real life problem that caused much consternation and merriment in the Nihon Ki-in in 1958. It relies on the technique that is at the heart of this paper. There is a small dead group completely surrounded by a larger group. But the larger group runs out of liberties and the small dead group comes back to life when the big group is captured. This kind of thing never comes up in professional play. (The game that it came from was part of the qualifying tournament for professional shodan; Mizuno Hiroshi and Noriyuki Nakayama were not yet professional players.) All the top players had drilled so hard on the life and death problems that actually arise in professional play that they had developed a peculiar blind spot around capturing races with one group completely surrounded by another. My expectation is that the first fifty years of Artificial Intelligence disasters will be dominated by humans being over-impressed by the intelligence of AI systems and giving them responsibilities that they are not read for. This article https://www.gbnews.com/news/ai-latest-news-judge-legal-decisions https://www.gbnews.com/news/ai-latest-news-judge-legal-decis... already speculating that "AI machines 'could soon make legally binding decisions' in UK, says senior judge" is the kind of thing I fear. I don't know what to make of The Treasure Chest Enigma. Theory one: it tells us that we are already coping successfully with this problem in human intelligence. Yes, AI has weird mental blind spots, but people do too and there is no need to worry. Theory two: Human mental blind spots are smaller and more obscure that the mental blind spots of current AI. There is plenty of quantitative difference so that we can legitimately worry that AI blind spots will cause unexpected disasters. We will be taken by surprise because we are used to human intelligence with its small, benign blind spots. The significance of The Treasure Chest Enigma is that it exists, which tells us that mental blind spots are inherent to intelligence. We will not solve the problem of AI having large, dangerous mental blind spots easily. We can only hope to shrink the problem down to an ineradicable core.

aaron695 3y ago

[dead]

_8j50 3y ago

> ...there could be fringe situations where it behaves badly I think the more important takeaway is that we humans don't know those fringe situations or corner cases beforehand. No way around finding the problem in prod.