5 ms·

I don't get what is new in the set of attributes that this article describes. Monte Carlo was already used in 2005 in AIs playing on KGS. Gradient Descent is a

• gwern 9y ago
  > The curious thing about Zero is that usually with Gradient Descent you run a huge risk of running into a local maximum and then stop evolving because every evolution makes you not better than the current step. No. The curious thing is that you can train a godawful huge NN with 40 layers via pure self-play with no checkpoints or baselines or pretraining or library of hard problems or any kind of stabilization mechanism, and it won't diverge but will learn incredibly rapidly and well and stably. As Silver says in the AmA, all their attempts at pure self-play ran into the usual divergence problems where the training explodes and engages in catastrophic forgetting, which is what the RL folklore predicts will happen if you try to do that. Local maximums are not the problem - the problem is the self-play can't even find a local maximum much less maintain it or improve it. > So one question is actually how they used these same old algorithms so much more efficiently, and the second question is how did they overcame the local maximum problem. Er, this is exactly what OP is all about: the Monte Carlo tree search supervision. That's how they used them.
  • psb217 9y ago
    If you read the AGZ paper closely, they actually use checkpoints during training. Specifically, during training they only perform updates to the "stable" set of parameters when the current "learning" set of parameters produces a policy which beats the stable set at least 55% of the time. The current stable parameters are what they use for generating the self-play data which they use to update the current "learning" parameters. I believe this is only mentioned in the supplementary material...
    • gwern 9y ago
      I did and I would point out that while they use checkpoints, the training curves indicate this is not necessary, and what I meant is that they do not use the usual self-play (and evolutionary) mechanism of checkpoints from throughout the training history which is necessary to combat catastrophic forgetting (and which apparently wasn't enough to stabilize Zero on its own as it is the single most obvious thing to do but Silver notes all the pre-Zero self-plays diverged until they finally came up with that of MCTS supervision). The checkpoint mechanism there appears no more necessary than checkpoints in training any NN - it's critical to avoid a random error or bug wasting weeks of time but does not affect the training dynamics in any important way.
      • gwern 9y ago
        (And Anthony et al 2017 don't use checkpoints at all, noting that it slows things down a lot for no benefit in their Hex agent.)
  • erikb 9y ago
    > divergence problems where the training explodes and engages in catastrophic forgetting This is a new problem I haven't heard about. Thanks for adding it to the discussion. > Er, this is exactly what OP is all about: the Monte Carlo tree search supervision. And here, I don't know. I think I exclaimed confidently that this is a super old algorthim. What is new about it, that you need to mention it again? Really, may be that I'm misinterpreting something, or forgetting something, but the algorithm itself I'm quite sure I discussed with Go AI scientists in 2005 who had them in all their bots, afaik. Please correct if you believe my memory is cheating me here.
    • gwern 9y ago
      > but the algorithm itself I'm quite sure I discussed with Go AI scientists in 2005 who had them in all their bots, afaik You did not because first, MCTS was only published by Coulom in 2006 (so they couldn't 'all' have been using it in 2005), and second because you are missing the crucial iteration between the heavy heuristic (NN) for playouts and the refined estimates from the playouts: https://news.ycombinator.com/item?id=15627834 https://news.ycombinator.com/item?id=15627834 Any MCTS in 2006 for Go would either be using random light playouts or simple unchanging hand-engineered heavy heuristics, hence, no possible recursive self-improvement in self-play.
• nilkn 9y ago
  > The curious thing about Zero is that usually with Gradient Descent you run a huge risk of running into a local maximum and then stop evolving because every evolution makes you not better than the current step. Deep networks have many parameters and represent extremely high-dimensional cost functions. Some basic heuristics suggest that sufficiently random high-dimensional functions have relatively few local extrema, but lots of saddle points. (Look at the signs of the eigenvalues of the Hessian matrix and apply a coin flipping argument. This suggests that local extrema become exponentially rarer as the dimension increases.) Moreover, stochastic gradient descent is actually pretty good at escaping saddle points.