6 ms·
The article glosses over why the 4 hours was possible. Firstly, a major challenge in training an AI of this sort is getting enough labelled data. They played
by rstuart4133 9y ago
The article glosses over why the 4 hours was possible.
Firstly, a major challenge in training an AI of this sort is getting enough labelled data. They played 300,000 games from memory. Under normal circumstances, that requires access to 300,000 games played by experts so the AI can learn to copy what the export does. That is how Alpha Go did it.
AlphaZero neatly side steps this by generating it's own training data by playing itself. If how to do this was "obvious", it would have been done a long time ago.
Secondly, they parallelised things. Alpha Go trained the AI from the results of each game as it is played. AlphaZero played 1,250 games simultaneously, and feed the results into the AI as they became available. The result is it took well over an order of magnitude less elapsed time to train AlphaZero than Alpha Go, even though the CPU cycles may have been roughly similar.
Finally, he overstates how hard it is to customise the engine (Markov algorithm + AI) to a game. There are two pointers to this. Firstly, it took them over 2 years to create Alpha Go. It became the world champion on 23 May. Now, 7 months later we have AlphaZero. But AlphaZero didn't to play just one game in those 7 months: is the best player on the planet for 3 games: Go, Chess, and Shogi.
I don't know whether they customised the AI for each game, but I suspect if the input and output layers were wide enough to accommodate the largest game they could use the same one for each. The Markov engine does have to know how to make all legal moves from any game position, but coding that isn't rocket science or particularly time consuming. The AI does _not_ start out knowing those rules - it learns them from the Markov engine. It's all very DRY.
This sort of engine only works are a particular style of game - one where their is only a smallish set of well known moves at each step, and the playing board is also smallish (19x19 in the case of Go, with three possible states for each position: empty, black, white). Most board and card games fit this description. AlphaZero can teach itself to play any of them to a standard higher than any human can play them, and do it within a few hours, not the decades it takes to create a human grand master. The net result is Homo sapiens reign of supremacy at playing this style of game is now over. For this entire niche our brains have been firmly relegated to a 2nd class intelligence.
I don't know whether you would call pulling this off a breakthrough, but I do know the techniques they applied will be copied by man+dog for years if not decades to come.
- sanxiyn 9y agoI agree, but: > Most board and card games fit this description. Most card games don't, because they are not perfect information games.
- joelthelion 9y agoIs it really impossible to adapt Deepmind's approach to, say, poker? I don't see anything fundamentally impossible, but I'm no RL expert.
- sanxiyn 9y agoI would consider a successful application of AlphaZero algorithm to imperfect information games to be a separate achievement, not a done deal. David Silver himself, who led AlphaGo project, wrote in 2016 in "Deep Reinforcement Learning from Self-Play in Imperfect-Information Games" https://arxiv.org/abs/1603.01121 https://arxiv.org/abs/1603.01121 "While many machine learning methods have achieved near-optimal solutions to classical, perfect-information games, these methods fail to converge in imperfect-information games".
- joelthelion 9y agoThanks for that paper, very interesting.
- tialaramex 9y agoWhereas the "correct" way to play chess will consist of knowing always exactly which move to make, most Poker variants will require a statistical model that says things like "In this situation we will fold 4% of the time, call 89% of the time and raise 7% of the time". Then the actual choices you make are random with a specific weighting. So that's hugely different. This truth is why strong poker analysts are always so equivocal. "All options are good here" annoys people but it's often true. For Heads Up (two players) Limit (bet sizes are fixed) Hold Em (a popular modern variant with two hole cards and five community cards) such a strategy was developed and you can view it online, it's essentially unbeatable. Its name is Cepheus.