6 ms·
"It uses one neural network rather than two." and "AlphaGo Zero only uses the black and white stones from the Go board as its input, whereas previous versions o
by Jyaif 9y ago
"It uses one neural network rather than two."
and
"AlphaGo Zero only uses the black and white stones from the Go board as its input, whereas previous versions of AlphaGo included a small number of hand-engineered features."
This is amazing! The technology they came up with must be super generic.
- bluetwo 9y agoIt sounds like that's what they are going for. Minimal tuning, generally adaptable. Very interesting.
- samfriedman 9y agoSuper, super impressive work. I'd love to see how hard it is to apply the architecture to other games/problems that work well with self-play.
- gordon_freeman 9y agoI will be interested to see what kind of algorithms they have used to allow AlphaGo to learn from its own moves. Are these pretty generics algos or are these very customized and specific ones that only apply to AlphaGo and the game of Go?
- Scaevolus 9y agoThey have a new reinforcement learning algorithm that should be generically applicable to anything where a long sequence of moves results in a specifically gradable outcome. > The neural network in AlphaGo Zero is trained from games of selfplay by a novel reinforcement learning algorithm. In each position s, an MCTS search is executed, guided by the neural network fθ. The MCTS search outputs probabilities π of playing each move. These search probabilities usually select much stronger moves than the raw move probabilities p of the neural network fθ(s); MCTS may therefore be viewed as a powerful policy improvement operator. Self-play with search—using the improved MCTS-based policy to select each move, then using the game winner z as a sample of the value—may be viewed as a powerful policy evaluation operator. The main idea of our reinforcement learning algorithm is to use these search operators repeatedly in a policy iteration procedure: the neural network’s parameters are updated to make the move probabilities and value (p, v)= fθ(s) more closely match the improved search probabilities and selfplay winner (π, z); these new parameters are used in the next iteration of self-play to make the search even stronger.
- romaniv 9y ago> They have a new reinforcement learning algorithm that should be generically applicable to anything where a long sequence of moves results in a specifically gradable outcome. Statements like these always make me wonder why certain obvious things weren't tried. If it's so generic, why wasn't it tried on Chess? Or was it tried, failed to impress and thus didn't make it into the press release? This is a big problem with all these public discussion on AI. Almost no one speaks about algorithm failures. I haven't seen a single research paper that said "oh, and we also tried algorithm in X domain and it totally sucked".
- namelost 9y agoThe conventional wisdom for Chess engines is that aggressive pruning doesn't work well. Chess is much more tactical than Go, selective algorithms tend to lead to some crucial tactic being missed, and the greater the search depth, the more likely that is. Modern Chess engines are designed to brute-force the search tree as efficiently as possible. I will go out on a limb here and say they would wipe the floor with AlphaGo, because AlphaGo's hardware would be more of a liability than an asset against a CPU. See also: https://chessprogramming.wikispaces.com/Type+A+Strategy https://chessprogramming.wikispaces.com/Type+A+Strategy https://chessprogramming.wikispaces.com/Type+B+Strategy https://chessprogramming.wikispaces.com/Type+B+Strategy
- nojvek 9y agoUntil I see AlphaGo zero defeating StockFish 100-0 and with same algorithm defeating best Go AI and killing the Atari games including montezuma’s revenge, I call this hype bullshit. Give me your results on OpenAI gym in a variety of different styles of games including GTA and WoW. I will believe you if a generic unsupervised algorithm running on a single machine is absolutely destroying the best players. Until then ...
- goatlover 9y agoAdd Pacman and Pitfall to the list. Humans have played perfect games of both. My understanding is DeepMind performed poorly on those games.
- deleted 9y ago[deleted]
- amelius 9y agoBut I suppose they still do the searching/pruning with a separate piece of code (not a neural network).
- hacker_9 9y agoFrom the paper: "it uses a single neural network, rather than separate policy and value networks ... it uses a simpler tree search that relies upon this single neural network to evaluate positions and sample moves, without performing any MonteCarlo rollouts. To achieve these results, we introduce a new reinforcement learning algorithm that incorporates lookahead search inside the training loop, resulting in rapid improvement and precise and stable learning."
- jtolmar 9y agoYes, but tree search + neural net is still pretty generic. It only assumes that you can enumerate branches.
- mannigfaltig 9y agoIt also presumes that one can simulate the world at low cost. In AlphaGo Zero it takes 0.4 s for 1.600 node extensions, but in this case the cost of the world is negligible. Anyway, assuming you need that many node extensions to get decent quality updates, that puts a rather a tight limit on the cost of simulating the world.
- gwern 9y agoDM has already done a bunch of work on 'deep models' of environments to plan over. Use them and you have 'model-predictive control' and planning, and this tree extension to policy gradients would work as well (probably). It could be pretty interesting to see what would happen if you tried that sort of hybrid on ALE.
- mannigfaltig 9y agoI guess deep world models are still severely riddled by all sorts of problems: vanishing gradients, BPTT being O(T), poor generalization ability of NNs (which likely is due to the lack of attractor state associative recall, as well as concept composability), lack of probabilistic message passing to deal with uncertainty, and perhaps some priors about the world are necessary to make learning tractable (such as spatial maps and fine-tuning for time scales that contain interesting information).
- skinner_ 9y agoAlso, unsupervised. Also, no rollouts. They got rid of a lot of complexity. At this point it looks like a reasonable challenge to write a superhuman Go AI in 500 lines of unobfuscated python.