4 ms·
> Previous versions of AlphaGo initially trained on thousands of human amateur and professional games to learn how to play Go. AlphaGo Zero skips this step and
by davidkuhta 9y ago
> Previous versions of AlphaGo initially trained on thousands of human amateur and professional games to learn how to play Go. AlphaGo Zero skips this step and learns to play simply by playing games against itself, starting from completely random play.
So technically this version has lost every game it's ever won.
Jokes aside, it's pretty interesting to note that they were able to combine the "policy" and "value" networks. Good SO answers on the difference (https://datascience.stackexchange.com/questions/10932/difference-between-alphagos-policy-network-and-value-network https://datascience.stackexchange.com/questions/10932/differ...)
> accumulating thousands of years of human knowledge during a period of just a few days
It'd be interesting for what this would mean when things like a neural lace become a reality.
As an aside, anyone have any other links or references to others investigating learning algorithms with a 'tabula rasa' approach?
- milofeynman 9y agohttps://blog.openai.com/ https://blog.openai.com/ I believe their DotA 2 AI uses that approach
- mappingbabeljc 9y agoCorrect - we saw a similar phenomenon of rapid capability gain via self-play in our Dota 2 work: https://blog.openai.com/more-on-dota-2/ https://blog.openai.com/more-on-dota-2/
- davidkuhta 9y agoThanks!
- apetresc 9y ago> So technically this version has lost every game it's ever won. No, they've also played it against AlphaGo Lee and AlphaGo Master. The SGFs are available at: https://www.nature.com/nature/journal/v550/n7676/extref/nature24270-s2.zip https://www.nature.com/nature/journal/v550/n7676/extref/natu...
- davidkuhta 9y agoI meant that tongue-in-cheek based on the "by playing games against itself" during training. Nonetheless, thanks for clarifying that in case it's unclear for others (and for the SGFs).
- greysphere 9y agoTD-gammon is a well known version of this technique (with 2 ply lookahead, vs a 1600 deep mcts) https://en.m.wikipedia.org/wiki/TD-Gammon https://en.m.wikipedia.org/wiki/TD-Gammon Temporal difference learning was previously consider weak at 'tactical' games, ie ones with gamestates that require long chains of precise moves to improve position (like many checkmate scenarios in chess) . For anyone more familiar with this technique, is it clear how the mcts/checkpoint system overcomes this? How sensative is the system to the tuning params for those parts of the alg. Like is Go a particularly good candidate because of the ~400 play positions resulting in a (relatively) small tree seach requirement? (I kinda cant believe im saying that go has 'a small search tree'!) We us td learning for the ai in our game Race for the Galaxy, so it's neat to hear about possible avenues for improvement!
- shmageggy 9y agoYeah it's not clear to me why temporal difference learning all of a sudden works so well here? Is it the case that nobody had really tried it for learning a policy for Go with a strong NN architecture? In the Methods they mention TD learning for value functions but I don't see anything about policies. edit: OK, they're calling it policy iteration as opposed to TD learning. I guess I don't get the difference.
- psb217 9y agoTD learning is, in some sense, a component of policy iteration. TD learning is about learning the value function for a given policy. In policy iteration you use a value function to decide how to update the policy for which the value function was estimated, and you iterate between the "learn value" and "update policy" steps. https://www.cs.cmu.edu/afs/cs/project/jair/pub/volume4/kaelbling96a-html/node20.html https://www.cs.cmu.edu/afs/cs/project/jair/pub/volume4/kaelb...
- davidkuhta 9y agoWow, that was a really deep and enjoyable Wikipedia rabbit hole journey. I hadn't heard of Temporal Difference before (though I was familiar with Q-learning). It was interesting to note that TD-Gammon improved with expert designed features. I wonder if this was simply related to the technology of the field as it stood over 20 years ago or some underlying categorization or complexity associated with the games themselves (backgammon being more favorable to human comprehension than Go in this case). > Even though TD-Gammon discovered insightful features on its own, Tesauro wondered if its play could be improved by using hand-designed features like Neurogammon's. Indeed, the self-training TD-Gammon with expert-designed features soon surpassed all previous computer backgammon programs. It stopped improving after about 1,500,000 games (self-play) using 80 hidden units. For others: Richard Sutton, one of the pioneers of TD makes his Reinforcement Learning: An Introduction textbook available for free on his website: http://incompleteideas.net/sutton/ http://incompleteideas.net/sutton/ (MIT Press also links to it)
- JoeDaDude 9y agoI remember reading about Blondie24, a program that learned to play checkers at a high level without human input. It was based on neural network and genetic algorithm technology. From the Wikipedia entry: "The significance of the Blondie24 program is that its ability to play checkers did not rely on any human expertise of the game. Rather, it came solely from the total points earned by each player and the evolutionary process itself." [1]. In addition to numerous journal articles, the creators wrote a lay-person book on their creation: Blondie24: playing at the edge of AI, by David B. Fogel [2]. [1]. https://en.wikipedia.org/wiki/Blondie24 https://en.wikipedia.org/wiki/Blondie24 [2]. https://dl.acm.org/citation.cfm?id=501597 https://dl.acm.org/citation.cfm?id=501597