7 ms·

The fact that they only used self play with no outside input here is really interesting. I wonder if this system produced more new styles of play. While I am no

by aeleos 9y ago

The fact that they only used self play with no outside input here is really interesting. I wonder if this system produced more new styles of play. While I am not that familiar with Go, I know in some of the other articles they talk about things like Chinese starts that are specific to certain cultures. I wonder if the fact that it had no outside reinforcement made it produce movements that we have already seen that are somehow inherent to the game, or if it produced many more new moves that were a result of it learning without any possibility of cultural interference. According to the article it did invent some unconventional and creative moves, but I also wonder how much it rediscovered. I also wonder how much it’s style of play changes if it were re trained, due to the random start that it is given. Maybe that would produce something like seeds for procedurally generated worlds in games. Like if they could find a seed for a Chinese or japanese players, or ones that more aggressive styles. This is some pretty cool work and may open up even more doors for pure reinforcement learning

shmageggy 9y ago
The key part from the paper: > To assess the merits of self-play reinforcement learning, compared to learning from human data, we trained a second neural network (using the same architecture) to predict expert moves in the KGS Server data set; this achieved state-of-the-art prediction accuracy compared to pre vious work 12,30–33 (see Extended Data Tables 1 and 2 for current and previous results, respectively). Supervised learning achieved a better initial performance, and was better at predicting human professional moves (Fig. 3). Notably, although supervised learning achieved higher move prediction accuracy, the self-learned player performed much better overall, defeating the human-trained player within the first 24 h of training. This suggests that AlphaGo Zero may be learning a strategy that is qualitatively different to human play.
- aeleos 9y ago
  That is really interesting. Given a neural network that solely exist to play Go, one that is influenced by the human mind is limited compared to the exact same set of neurons that doesn't have that influence. EDIT: changed a set of neurons to neural network per andbbergers comments
  - andbberger 9y ago
    Please don't refer to it as 'a set of neurons' - it only serves to fuel the (IMO) absolutely ridiculous AI winter fearmongering, and is also just a bad description. Neural nets are linear algebra blackboxes, the connections to biology are tenuous at best. Sorry to be that guy, but the AI hype is getting out hand. COSYNE this year was packed with papers comparing deep learning to the brain... it drives me nutty. Convnets can be reasonably put into analogy with the visual system.... because they were inspired from it. But that's about it. To address your actual comment: I would argue that this is not really interesting or surprising (at least to the ML practitioner), it is very well known that neural nets are incredibly sensitive to initialization. Think of it like this: as training progresses, parameters of neural nets move along manifolds in parameter space, but they can get nudged off of the "right" manifold and will never be able to recover. Sorry for the rant, the AI hype is just getting really out of hand recently. Machine learning is specifically not magic. Blackboxes are not useful. Convnets work so well because they build the symmetries of natural scenes directly into the model - natural scenes are translation invariant (as well as a couple of other symmetries), anything that models them sure as hell better have those symmetries too, or you're just crippling your model with extra superfluous parameters.
    - etiam 9y ago
      Please don't refer to them as black boxes. The internals are fully observable.
      - fragsworth 9y ago
        It's a conceptual black box. There's no way for us to understand what each individual neuron is doing.
        etiam 9y ago
        Indeed? https://arxiv.org/pdf/1312.6034.pdf https://arxiv.org/pdf/1312.6034.pdf
        blt 9y ago
marcosscriven 9y ago
Possibly a dumb q, but is ‘self play’ in any way related to ‘adversarial’ learning? I don’t see it mentioned in the article, but it reminds me of the principle.
- aeleos 9y ago
  In some ways it is, but the main difference is that adversarial learning (usually) produces a second neural network whose purpose is to exploit weakness is the first. Whereas reinforcement learning does not produce a second neural network to beat the first, it uses what it learned to solely improve the original. As a side note, the main application I have seen with adversarial learning research is with photo recognition, but I guess you could have an adversarial network exist to help help improve an object recognition network. At that point it would probably become something between adversarial and reinforcement learning. However, with game based reinforcement learning, it doesn't require a second specific network as the adversary, it can easily just be paired against itself. It isn't a dumb question, they are very similar in some ways. They mainly differ in what exactly the goal of the opponent is. In this case, it is to help improve itself, however in typical adversarial situations it is solely to exploit (become its adversary).
jasonwatkinspdx 9y ago
> I wonder if this system produced more new styles of play. One thing Alpha go has told us clearly is that it thinks human players over value the margin of victory vs the probability of victory.
- edanm 9y ago
  I'm not 100% sure I agree. It values probability of victory because that's it's goal. For humans, aiming only for probability of victory might not be as good, because we're much worse at estimating probabilities. So aiming for maintaining a large margin at all times is conceivably the best proxy that we can use in practice.
  - pmontra 9y ago
    Agreed. I know I'm winning by 4 points but I have no idea about my probability of winning. However if I'm winning I know that I should play low risk moves and refrain from starting complicated fights. That increases the probability of winning. IMHO the exact value is out of reach for human beings.
    - derefr 9y ago
      It would be interesting to play human go, assisted by a go computer that doesn't say anything about moves, but rather just spits out, for each player, their current likelihood of victory if all further moves by both players were "what it would do." That way, each player could know, at all times, (one major factor that goes into) their probability of winning. They'd still have to mentally adjust it for the likelihood of them and their opponent making an error, and how that can be controlled by making intimidating moves, etc. But it could lead to much tighter control on the abstract flow of the game. It'd almost be like the computer was the general, issuing strategy, picking battles; and the human player the tactician, fighting those battles.
      - pmontra 9y ago
        There is computer go program (maybe Crazy Stone?) that analyzes a game record and annotates it with the winning percentage for every move. Knowing that the opponent's winning probability changed from 52 to 57 was interesting only because it hints at a mistake. In case of such a large change the program suggests the move it would have played. I saw an annotated game record and there were no variations: I remember a suggested move that made me wonder "why!?". Another benefit of seeing the value of the winning probability is an assessment of who's ahead. However that's already possible with the score estimation that programs and go servers provide. Sometimes is crude, sometimes is good, but it's the score, not the winning probability that humans can estimate when playing. The best probability estimate I can make is: if the score is close and the game is still complicated, it's 50-50; if the score is close but the game is almost over, it's 95-5 for who's ahead. If the score is not close, the player with more points will probably win.
kndyry 9y ago
I don't think it's an overstatement to say that, since playing Lee Sedol in 2016, AlphaGo has completely revolutionized professional and amateur go. It's certainly not unprecedented — the last major revolution happened in the early 20th century (often called the 'Shin Fuseki' era [0]) — but AlphaGo has demonstrably surpassed any previous high-water mark. > I wonder if this system produced more new styles of play. Absolutely. One such innovation has been the use of early 3-3 invasions [1]. There are many more, and indeed AlphaGo's games are still being analyzed by professional players. Michael Redmond, a 9-dan professional, has been working with the American Go Association on one such series [2]. > I wonder if the fact that it had no outside reinforcement made it produce movements that we have already seen that are somehow inherent to the game... Interestingly, yes. Strong players have commented that AlphaGo seems to agree with things that players like Go Seigen [3] have suggested in the past, but that were never fully developed or understood [4]. Very, very interesting work indeed. [0] https://senseis.xmp.net/?ShinFuseki https://senseis.xmp.net/?ShinFuseki [1] https://www.eurogofed.org/index.html?id=127 https://www.eurogofed.org/index.html?id=127 [2] http://www.usgo.org/news/category/go-news/computer-goai/masteralphago-commentaries/ http://www.usgo.org/news/category/go-news/computer-goai/mast... [3] https://senseis.xmp.net/?GoSeigen https://senseis.xmp.net/?GoSeigen [4] https://lifein19x19.com/forum/viewtopic.php?f=13&t=14129 https://lifein19x19.com/forum/viewtopic.php?f=13&t=14129
- alex_stoddard 9y ago
  I have only skimmed the paper but one thing I don't see any discussion of is whether komi (the handicap given to white for going second) is correct. They do say the rules used for all games, including self-play, set komi consistently to 7.5 . If the strongest AI was consistently winning predominantly with one color it would be an indication that komi isn't fair for the best play. Of the 20 games released for the strongest play it appears white won 14 times and black 6. I don't think that is enough to be conclusive but maybe komi is too high. I wonder if different "correct" play at the strongest levels would be learned with a 6.5 komi.
  - gizmo686 9y ago
    You can only change komi by full point increments. There is a .5 to break ties, but a komi of 7.5 is identical to one of 7.4. From a theoretical standpoint, any non-integer komi should lead to one player winning 100% of the time. So even if the actual win ratio is 14:6 at komi=7.5 that might still be the best value.
autarch 9y ago
> things like Chinese starts that are specific to certain cultures While it's true that there are national styles of play, the Chinese opening is not called that because it's really popular among Chinese people. It's called that because a particular Chinese pro helped popularize it, even though it was invented by a Japanese amateur. See https://en.wikipedia.org/wiki/Chinese_opening https://en.wikipedia.org/wiki/Chinese_opening for some more info. FWIW, I (a caucasian American) use this opening all the time. It's just a generally good opening if you like a certain style of play.
- igravious 9y ago
  > talk about things like Chinese starts that are specific to certain cultures Came here to make this point. It's Chinese Opening, not Chinese start – similarly recall that you have the French Defense / Italian Defense / Scandinavian Defense among chess opening variations and none of these implies that that opening variation is specific to that culture or nation.

by aeleos 9y ago

shmageggy 9y ago

The key part from the paper: > To assess the merits of self-play reinforcement learning, compared to learning from human data, we trained a second neural network (using the same architecture) to predict expert moves in the KGS Server data set; this achieved state-of-the-art prediction accuracy compared to pre vious work 12,30–33 (see Extended Data Tables 1 and 2 for current and previous results, respectively). Supervised learning achieved a better initial performance, and was better at predicting human professional moves (Fig. 3). Notably, although supervised learning achieved higher move prediction accuracy, the self-learned player performed much better overall, defeating the human-trained player within the first 24 h of training. This suggests that AlphaGo Zero may be learning a strategy that is qualitatively different to human play.

aeleos 9y ago
That is really interesting. Given a neural network that solely exist to play Go, one that is influenced by the human mind is limited compared to the exact same set of neurons that doesn't have that influence. EDIT: changed a set of neurons to neural network per andbbergers comments
- andbberger 9y ago
  Please don't refer to it as 'a set of neurons' - it only serves to fuel the (IMO) absolutely ridiculous AI winter fearmongering, and is also just a bad description. Neural nets are linear algebra blackboxes, the connections to biology are tenuous at best. Sorry to be that guy, but the AI hype is getting out hand. COSYNE this year was packed with papers comparing deep learning to the brain... it drives me nutty. Convnets can be reasonably put into analogy with the visual system.... because they were inspired from it. But that's about it. To address your actual comment: I would argue that this is not really interesting or surprising (at least to the ML practitioner), it is very well known that neural nets are incredibly sensitive to initialization. Think of it like this: as training progresses, parameters of neural nets move along manifolds in parameter space, but they can get nudged off of the "right" manifold and will never be able to recover. Sorry for the rant, the AI hype is just getting really out of hand recently. Machine learning is specifically not magic. Blackboxes are not useful. Convnets work so well because they build the symmetries of natural scenes directly into the model - natural scenes are translation invariant (as well as a couple of other symmetries), anything that models them sure as hell better have those symmetries too, or you're just crippling your model with extra superfluous parameters.
  - etiam 9y ago
    Please don't refer to them as black boxes. The internals are fully observable.
    - fragsworth 9y ago
      It's a conceptual black box. There's no way for us to understand what each individual neuron is doing.
      - etiam 9y ago
        Indeed? https://arxiv.org/pdf/1312.6034.pdf https://arxiv.org/pdf/1312.6034.pdf
      - blt 9y ago

marcosscriven 9y ago

Possibly a dumb q, but is ‘self play’ in any way related to ‘adversarial’ learning? I don’t see it mentioned in the article, but it reminds me of the principle.

aeleos 9y ago
In some ways it is, but the main difference is that adversarial learning (usually) produces a second neural network whose purpose is to exploit weakness is the first. Whereas reinforcement learning does not produce a second neural network to beat the first, it uses what it learned to solely improve the original. As a side note, the main application I have seen with adversarial learning research is with photo recognition, but I guess you could have an adversarial network exist to help help improve an object recognition network. At that point it would probably become something between adversarial and reinforcement learning. However, with game based reinforcement learning, it doesn't require a second specific network as the adversary, it can easily just be paired against itself. It isn't a dumb question, they are very similar in some ways. They mainly differ in what exactly the goal of the opponent is. In this case, it is to help improve itself, however in typical adversarial situations it is solely to exploit (become its adversary).

jasonwatkinspdx 9y ago

> I wonder if this system produced more new styles of play. One thing Alpha go has told us clearly is that it thinks human players over value the margin of victory vs the probability of victory.

edanm 9y ago
I'm not 100% sure I agree. It values probability of victory because that's it's goal. For humans, aiming only for probability of victory might not be as good, because we're much worse at estimating probabilities. So aiming for maintaining a large margin at all times is conceivably the best proxy that we can use in practice.
- pmontra 9y ago
  Agreed. I know I'm winning by 4 points but I have no idea about my probability of winning. However if I'm winning I know that I should play low risk moves and refrain from starting complicated fights. That increases the probability of winning. IMHO the exact value is out of reach for human beings.
  - derefr 9y ago
    It would be interesting to play human go, assisted by a go computer that doesn't say anything about moves, but rather just spits out, for each player, their current likelihood of victory if all further moves by both players were "what it would do." That way, each player could know, at all times, (one major factor that goes into) their probability of winning. They'd still have to mentally adjust it for the likelihood of them and their opponent making an error, and how that can be controlled by making intimidating moves, etc. But it could lead to much tighter control on the abstract flow of the game. It'd almost be like the computer was the general, issuing strategy, picking battles; and the human player the tactician, fighting those battles.
    - pmontra 9y ago
      There is computer go program (maybe Crazy Stone?) that analyzes a game record and annotates it with the winning percentage for every move. Knowing that the opponent's winning probability changed from 52 to 57 was interesting only because it hints at a mistake. In case of such a large change the program suggests the move it would have played. I saw an annotated game record and there were no variations: I remember a suggested move that made me wonder "why!?". Another benefit of seeing the value of the winning probability is an assessment of who's ahead. However that's already possible with the score estimation that programs and go servers provide. Sometimes is crude, sometimes is good, but it's the score, not the winning probability that humans can estimate when playing. The best probability estimate I can make is: if the score is close and the game is still complicated, it's 50-50; if the score is close but the game is almost over, it's 95-5 for who's ahead. If the score is not close, the player with more points will probably win.

kndyry 9y ago

I don't think it's an overstatement to say that, since playing Lee Sedol in 2016, AlphaGo has completely revolutionized professional and amateur go. It's certainly not unprecedented — the last major revolution happened in the early 20th century (often called the 'Shin Fuseki' era [0]) — but AlphaGo has demonstrably surpassed any previous high-water mark. > I wonder if this system produced more new styles of play. Absolutely. One such innovation has been the use of early 3-3 invasions [1]. There are many more, and indeed AlphaGo's games are still being analyzed by professional players. Michael Redmond, a 9-dan professional, has been working with the American Go Association on one such series [2]. > I wonder if the fact that it had no outside reinforcement made it produce movements that we have already seen that are somehow inherent to the game... Interestingly, yes. Strong players have commented that AlphaGo seems to agree with things that players like Go Seigen [3] have suggested in the past, but that were never fully developed or understood [4]. Very, very interesting work indeed. [0] https://senseis.xmp.net/?ShinFuseki https://senseis.xmp.net/?ShinFuseki [1] https://www.eurogofed.org/index.html?id=127 https://www.eurogofed.org/index.html?id=127 [2] http://www.usgo.org/news/category/go-news/computer-goai/masteralphago-commentaries/ http://www.usgo.org/news/category/go-news/computer-goai/mast... [3] https://senseis.xmp.net/?GoSeigen https://senseis.xmp.net/?GoSeigen [4] https://lifein19x19.com/forum/viewtopic.php?f=13&t=14129 https://lifein19x19.com/forum/viewtopic.php?f=13&t=14129

alex_stoddard 9y ago
I have only skimmed the paper but one thing I don't see any discussion of is whether komi (the handicap given to white for going second) is correct. They do say the rules used for all games, including self-play, set komi consistently to 7.5 . If the strongest AI was consistently winning predominantly with one color it would be an indication that komi isn't fair for the best play. Of the 20 games released for the strongest play it appears white won 14 times and black 6. I don't think that is enough to be conclusive but maybe komi is too high. I wonder if different "correct" play at the strongest levels would be learned with a 6.5 komi.
- gizmo686 9y ago
  You can only change komi by full point increments. There is a .5 to break ties, but a komi of 7.5 is identical to one of 7.4. From a theoretical standpoint, any non-integer komi should lead to one player winning 100% of the time. So even if the actual win ratio is 14:6 at komi=7.5 that might still be the best value.

autarch 9y ago

> things like Chinese starts that are specific to certain cultures While it's true that there are national styles of play, the Chinese opening is not called that because it's really popular among Chinese people. It's called that because a particular Chinese pro helped popularize it, even though it was invented by a Japanese amateur. See https://en.wikipedia.org/wiki/Chinese_opening https://en.wikipedia.org/wiki/Chinese_opening for some more info. FWIW, I (a caucasian American) use this opening all the time. It's just a generally good opening if you like a certain style of play.

igravious 9y ago
> talk about things like Chinese starts that are specific to certain cultures Came here to make this point. It's Chinese Opening, not Chinese start – similarly recall that you have the French Defense / Italian Defense / Scandinavian Defense among chess opening variations and none of these implies that that opening variation is specific to that culture or nation.