10 ms·

This is an incredible demonstration that the AG Zero expert iteration method is a general method. If you go back to the discussions of AG Zero lo a month ago, t

• Houshalter 9y ago
  They aren't the first to apply NNs to chess though. What are they doing differently? And does anyone else smell smoke?
  • gwern 9y ago
    It's certainly not the first NN chess program. You may remember one of OP author's Giraffe NN (https://arxiv.org/abs/1509.01549 https://arxiv.org/abs/1509.01549) which was essentially 'AlphaGo for chess'. But like the original AG, it struggles to learn and Lai had a lot less computation as a student than he does now at DM. What they're doing is applying AlphaGo Zero expert iteration with some simplifications and TPUs. And that pwns previous work like Giraffe the way AlphaGo Zero pwns AlphaGo. Quantity becomes a quality all its own.
  • evandijk70 9y ago
    Look at Figure 2, and remember that DM has access to a lot of hardware. At short thinking times, AlphaZero is weaker than Stockfish. This is equivalent to longer thinking times with weaker hardware, and it is likely that the former applications of NNs to chess had hardware that was a 1000-fold slower than what DM has access to. This means that even if the approach was identical to DMs, they would not have seen a better performance of NNs than the classical alpha beta approach.
    • nojvek 9y ago
      In essence MCTS + NN is just another way of tree search just like AlphaBeta or its brute force cousin Minimax. AlphaZero just tries to be smarter about which branches to evaluate so it can go deeper. But I would love to see AlphaZero (trained) run side by side with stockfish on an iPhone hardware and defeat it. That would be a more apples to apples comparison.
  • make3 9y ago
    they are a huge company (Google) with access to top top top talent (experts) and infinite hardware resources. I don't know why it would be surprising if they acheived performance that hadn't been acheived before
    • deleted 9y ago
      [deleted]
• deepnotderp 9y ago
  See the thing is though, Giraffe's evaluation actually was better than Stockfish's evaluation function, but it took much longer, and thus wasn't able to search as deep as Stockfish et al. So in a way, the real triumph of the AlphaGo series was the TPU and GPU army.
  • gwern 9y ago
    > So in a way, the real triumph of the AlphaGo series was the TPU and GPU army. Eh. It's still searching many fewer positions than Stockfish is.
    • deepnotderp 9y ago
      Right right, but my comparison was between giraffe and AlphaGo , not neural networks and Stockfish.
      • joshuamorton 9y ago
        But it looks like AlphaGo is searching fewer positions per second than Giraffe did. AlphaZero evaluates 80K positions per second, according to this paper, and the Giraffe paper says that Giraffe averaged 258570 evaluations per second when running STS. While we can't directly compare the computer power, this implies that AZ has learned a better representation.
        • allenz 9y ago
          It's unclear how much of the "better representation" was due to better algorithm vs. more compute/deeper NN.
          • joshuamorton 9y ago
            Giraffe was trained until convergence. Maybe if there was more compute power then, a different model would have been used, but that's deep into the world of silly hypotheticals.
• backpropaganda 9y ago
  While I'm glad to see you're excited about this, take note that this is still an approach which requires that an exact model is known, the state is fully visible, and the reward is perfectly define-able and known. Progress in this setup isn't necessarily correlated with the kind of AI for which we'd need a fire alarm.
  • visarga 9y ago
    It's easy for many to think that solving Go and chess means we can also solve household work like cleaning, cooking and washing dishes but it's actually harder.
    • backpropaganda 9y ago
      Next up: Google's Deepmind AI learns to perform arithmetic tabula rasa. More seriously, it seems Deepmind and the AI community in general is having a Streetlight effect problem, i.e. looking for AI in what works now, rather than coming to terms with the hard challenges. This explains why there are so many papers on GANs. People are just doubling down on what works (where the streetlight is), rather than acknowledging that where we need to look for AI is dark. Since it's become such a cut-throat race to be the next one to say "we made a breakthrough!", it makes much more economic sense to solve simple problems and advertise them as huge challenges.
      • visarga 9y ago
        I wouldn't dismiss GANs so easily. Yann LeCun was singing odes to GANs - as the most interesting idea in the last decade. The interesting thing about GANs is that they don't use a predefined loss, but instead the discriminator acts as the loss function for the generator - thus, it is learning a loss fn instead of using human guesswork to create it. That's quite a powerful new idea. Applications of GANs include making simulated images look more real, which is essential for RL, generating 'artificial' training images for other tasks and using the discriminator as an image embedding generator or classifier.
      • skybrian 9y ago
• grumpopotamus 9y ago
  >it only takes 4 hours of training to beat Stockfish In that time I figure they used the equivalent of about 1000 cpu-years. Imagine the things we'll be able to achieve as we can do more and more computation in less and less time.
  • cdancette 9y ago
    But how many "cpu hours" of human work were used to design stockfish? You can't really compare that. Some scientist say the brain has a power of several petaflops, so if you use this, I guess the design of stockfish was way less efficient. You can't really compare things to cpu years, it doesn't make sense. Power consumption would be a better metric I think.
    • CydeWeys 9y ago
      The best metric is total cost, including the cost of the hardware as well as the electricity. It might be worth prorating the hardware by the amount of time it spends on the task, too, assuming the hardware is general enough for many purposes (like TPUs are), vs say something like EFF's DES cracker which was not.
    • gwern 9y ago
      A ton of CPU has gone into Stockfish, if only for their distributed computing project fishnet: http://tests.stockfishchess.org/tests http://tests.stockfishchess.org/tests
      • gwern 9y ago
        To be a little more precise: Stockfish has used >5,667,382 CPU-hours (5.6 million CPU-hours) adding up just the participants who contributed >10,000 CPU-hours according to https://github.com/mcostalba/Stockfish/blob/master/Top%20CPU%20Contributors.txt https://github.com/mcostalba/Stockfish/blob/master/Top%20CPU...
        • nojvek 9y ago
• Cybiote 9y ago
  Let's break this down and consider things carefully. To informed researchers, what is most surprising here is not that the AlphaGo Zero algorithm beat stockfish but that MCTS managed to outperform Alpha-beta search. I'll venture a hypothesis as to why this was. Informed skepticism would have discounted MCTS against alpha-beta search but wouldn't have put much stock into the idea that Neural Networks couldn't learn better features than what has been painstakingly handcrafted. We know that given sufficient data and an appropriate architecture, neural nets have achieved better local minima than humans. This shouldn't be surprising anymore. A structurally adapted searcher will always do better in its adapted to domain. A Cat is so good at being a cat, it doesn't even have to think about how to cat. Choice of optimization method, input pre-processing, loss function, hyper-parameters and architecture together define a search space, a structural prior and how to navigate. Returning to alpha-beta vs MCTS, my view is that earlier work on the chess search space being ill-suited to MCTS has not been invalidated once you account for the synergy between the neural net and search method brought about by the imitation learning approach. What might be happening here is the neural net not only learns to correct when it goes out of bounds, it also learns to account for missteps of MCTS! The AlphaGo Zero Chess Program is clearly smarter than stockfish from the perspective of its ability to better navigate the search space but before talking about fire alarms there are some things to note. Assuming the paper, AlphaGo zero does well if you hold compute fixed and adjust time, but how does it do as you move along both compute and time? This is of relevance to the general community, especially if AlphaGoZero skill degrades gracefully enough to allow it to be a better tutor than current engines. Contrary to the no fire alarm claim, we should see sudden improvements everywhere due to how close joint, structured prediction, reinforcement and imitation learning are to each other. Unexpected improvement across a broad class of problems is a fire alarm. Right now, POMDP or games with hidden information and multiple interacting agents are still very difficult. Structured prediction is still difficult. Granted, this was before AGZ, but Neural Nets+MCTS had to be modified to Neural Self-Play before it could work just ok in poker-like games. What we should take away is the power of combining searching and learning. I'll argue that what is now being called expert iteration was presaged in an antique 2006 paper [1] where Hal Daume et al discuss the power of a learning algorithm trained to imitate a search computed policy. Even with limited compute and data, you can use similar ideas under the learning to search framework. The imitation approach is what's consistently yielded great results, whether applied to neural nets or logistic regression. [1] http://www.umiacs.umd.edu/~hal/docs/daume06searn-practice.pdf http://www.umiacs.umd.edu/~hal/docs/daume06searn-practice.pd... https://link.springer.com/content/pdf/10.1007/s10994-009-5106-x.pdf https://link.springer.com/content/pdf/10.1007/s10994-009-510...
• tantalor 9y ago
  > doesn't work as well in chess: it works better That's not how "as well" works.
  • dragonwriter 9y ago
    Yes, it is; while the idiom standing on its own implicitly includes a leading “at least”, it is also idiomatic to use it in exactly the way used by the grandparent post, in an explicit contrast with better, where it comes with an implicit (or sometimes explicit) leading “merely” instead of “at least”.
    • T-hawk 9y ago
      It's unnecessary, though, and makes the point harder to read. "It works even better" would be a perfectly sufficient description. "It works not as well as but better" is an unnecessary rhetorical flourish.
  • suchow 9y ago
    The misdirection is being used as a rhetorical device — you're supposed to feel a brief confusion when you get to the colon; it's then quickly resolved.