37 ms·

Mastering Chess and Shogi by Self-Play with General Reinforcement Learning

539 points by dennybritz 9y ago

stretchwithme 9y ago
See, Mom? Self play is a good thing.
ericand 9y ago
Certainly a significant achievement. Also, kind of interesting that the AlphaGo team spent a lot of energy to convince us Go is much harder than Chess, only to turn around and tell us that it is amazing that it can also win at Chess.
- 2bitencryption 9y ago
  > only to turn around and tell us that it is amazing that it can also win at Chess. What they're demoing here is a single, general formula for mastering multiple games. Start with empty AG0, then teach it chess from scratch until it is the strongest player on the planet. Go back to an empty slate, with the same exactly "untrained" AG0, and now teach it Go, to the same result. No fine-tuning for the domain of the game you are training -- it is general(ized). That's the gist I'm getting from this. question for someone who has time to read the paper: can you train it to master chess and go at the same time? or is it one or the other? I'm assuming the latter. edit: check out the graph on the 4th page. AlphaZero, which can master chess and shogi, can beat AlphaGo Zero, the implementation specifically designed for Go, at its own game.
  - visarga 9y ago
    Question: do you think you are using the same parts of the brain to play chess and Go? What counts is not using the same neurons, but using the same neural algorithm.
  - gwern 9y ago
    > question for someone who has time to read the paper: can you train it to master chess and go at the same time? or is it one or the other? I'm assuming the latter. I'm sure you could with a multi-headed NN. But what would be the point? There's very little transfer of knowledge between the games, especially once you get past the very most basics.
    - thanatropism 9y ago
      The point is that real problem domains are not neatly partitioned and labeled. I don't know what kind of input the NN itself gets, but computer vision is enough to translate a photo of a chessboard to a usable symbolic representation. But it would be nice to already have a black box-ish computer program that figures out what's the game at hand and how to play it. The next variation is have the adversary start playing a chess variant and have the machine recognize it (assuming honesty) and play it to significant skill. Then "real life Pong" where the size and aerodynamics of the ball are unknown to it. This is the gist of human intelligence: answering questions is significantly easier than figuring out what the question is.
  - jmmcd 9y ago
    > Go back to an empty slate, with the same exactly "untrained" AG0, and now teach it Go, to the same result. No fine-tuning for the domain of the game you are training -- it is general(ized). Not quite -- different input features, which implies slightly different network architecture at least at the front.
gallerdude 9y ago
I wonder if being an expert at one game makes it easier to be an expert at another. If so, then maybe the examples are datasets, and convergence would be able to complete new tasks after a few examples.
- vinchuco 9y ago
  Really interesting question. Some strategic concepts may transfer, say, from chess to chess variants. However, a simple change in the rules can have a huge impact in the game mechanics as anyone who has tried chess variants [1] knows. [1] https://en.wikipedia.org/wiki/List_of_chess_variants https://en.wikipedia.org/wiki/List_of_chess_variants The answer may be that it is hard enough to become an expert at anything, but there may be some serendipitous (how to make this precise?) overlap.
- foota 9y ago
  It would be very interesting to see someone try something like transfer learning from one game to another.
- jmmcd 9y ago
  Well, it's not doing anything like that for now. Even though the algorithm, in an abstract sense, is the same for all three games, in fact it's a new network for each of the three games, with architecture and input features adapted to the game, and then trained from scratch.
Invictus0 9y ago
I'm not sure it's really fair to compare Stockfish to AlphaZero; AlphaZero used 24h of 5000 TPUs in compute time, and still needed 4 TPUs in real play, while Stockfish ran on just 64 threads and 1GB RAM. Nonetheless, still an impressive achievement.
- nandemo 9y ago
  Wait, how's the 24h x 5000 TPUs relevant? That is training time, and that training corresponds to years and years of hardcoding evaluations in Stockfish, not to compute time during the match.
- Recursing 9y ago
  Only 1GB RAM? Really?
  - bluecalm 9y ago
    Yes, this is really strange. Hash table size is a major contributing factor for strength of chess programs. It looks like a very artificial limitation.
partycoder 9y ago
If you have seen the Stockfish project you will see many hardcoded weights in the configuration, found through experimentation. All these adjustments took probably years to achieve... and now Alpha Go Zero just self-learns everything and surpasses it. Would be good to see Deepmind's solution play Arimaa and Stratego, and see what kind of strategy it comes up with. Or weird variations of Go. Eventually this tech will make it into military strategy simulators and that's where things will get really messed up. 4 star generals will be replaced by bots.
- gamegoblin 9y ago
  I don't think this technique immediately applies to Stratego because it's not a perfect information game. I suspect it would exceed the state of the art in Arimaa, since Arimaa is specifically designed to have a high branching factor (17281 -- compared to 35 for chess), and this technique was designed to work well in high-branching factor games (since Go is a high-branching factor game, though much lower than Arimaa).
  - partycoder 9y ago
    In that regard then Stratego would share some aspects with Starcraft, another incomplete information game. Deepmind is actively working in a StarCraft bot. It would be interesting to see if they can be put together a supraintelligent StarCraft bot and then translate those results to Stratego.
ericand 9y ago
Two things to note: 1) Alpha Zero beats AlphaGo Zero and AlphaGo Lee and starts tabla rasa 2) "Shogi is a significantly harder game, in terms of computational complexity, than chess (2, 14): it is played on a larger board, and any captured opponent piece changes sides and may subsequently be dropped anywhere on the board. The strongest shogi programs, such as Computer Shogi Association (CSA) world-champion Elmo, have only recently defeated human champions (5)"
- pacaro 9y ago
  Shogi is a fun game, it always feels a little sad that it doesn't get more exposure outside of Japan (and my understanding is that, by and large, in Japan it is considered an "old persons" game) Because captured pieces change sides, there is less of an "endgame" scenario, and as a beginner (like me) it is very easy to put too many captured pieces back into play, which makes it hard to defend everything and essentially you end up giving them back to your opponent
  - foobaw 9y ago
    It briefly became popular in the otaku culture from an anime called Hunter X Hunter.
    - mikekchar 9y ago
      I'm curious to see if "San Gatsu no Lion" (the Lion of March) will spark interest. I highly recommend it to anyone interested in more slice-of-life/drama kinds of things. It's quite a beautiful anime/manga, even if the shogi isn't quite centre stage.
      - sanxiyn 9y ago
        Recommendation seconded, Sangatsu no Lion is a lovely work. On the other hand, it has been running for 10 years (!), if it could spark interest like Hikaru no Go, it would have happened already.
naveen99 9y ago
Very happy to see this result. It's like a moral victory for humans, as alphago is more human like (discounting montecarlo search) than stockfish. Maybe deep learning will give us the next Euler, Newton, or Einstein.
- visarga 9y ago
  Shogi, chess and Go are "perfect information games", meaning you can see the whole game state. It's a whole different thing to be able to solve games where you don't see everything (based on uncertainty).
  - jacobkg 9y ago
    In 2015 Heads-Up Limit Hold'em was solved: http://science.sciencemag.org/content/347/6218/145.full http://science.sciencemag.org/content/347/6218/145.full
  - ucaetano 9y ago
    You mean, like poker? https://www.cmu.edu/news/stories/archives/2017/january/AI-beats-poker-pros.html https://www.cmu.edu/news/stories/archives/2017/january/AI-be... Actually machines can have an even higher advantage in those cases, because they can be much better at estimating probabilities than humans. Think of card counting, for example.
    - grumpopotamus 9y ago
      I disagree. Computers have been outplaying the best humans at chess for two decades, but they only recently beat the top players at 2-player NLHE and only with the aid of massive computational power during training. Furthermore, techniques like monte-carlo tree search used in AlphaGo don't work very well for poker - You can't just try and find the "best move" from the current game state, or you will end up playing a highly-exploitable strategy. You essentially have to solve the entire game every time (or completely in advance) to make sure you are playing a balanced strategy. Only the Counter-Factual Regret Minimization algorithm has been able to achieve this level of play in Heads Up, and right now it looks hard to scale to poker games with more players, like the full-ring games you see at the World Series of Poker, for example. We still have a ways to go in Poker AI.
magoghm 9y ago
"We also analysed the relative performance of AlphaZero’s MCTS search compared to the state-of-the-art alpha-beta search engines used by Stockfish and Elmo. AlphaZero searches just 80 thousand positions per second in chess and 40 thousand in shogi, compared to 70 million for Stockfish and 35 million for Elmo. AlphaZero compensates for the lower number of evaluations by using its deep neural network to focus much more selectively on the most promising variations – arguably a more “human-like” approach to search, as originally proposed by Shannon." <- Amazing!
- maxander 9y ago
  Meanwhile a human player considers <1 position per second, so there’s a few orders of magnitude left to go in that direction. But unsettlingly few, nonetheless.
  - nopinsight 9y ago
    Humans are also much weaker than AlphaZero in these three games. The difference in the numbers of positions searched might be responsible for a substantial part of that.
    - yters 9y ago
      It'd be interesting to weaken AZ until it is on par with a human, and then compare moves evaluated. I'd suspect humans still evaluate significantly fewer moves.
  - aidenn0 9y ago
    Strong human players consider a lot more than 1 position per second in Chess...
gwern 9y ago
This is an incredible demonstration that the AG Zero expert iteration method is a general method. If you go back to the discussions of AG Zero lo a month ago, there was a lot of skepticism that NNs would ever challenge Stockfish et al - they are just too good, too close to perfection, and chess not well suited for MCTS and NNs. Well, it turns out that AG Zero doesn't work as well in chess: it works better as it only takes 4 hours of training to beat Stockfish. This is going to be an impetus for researchers to explore solving many more MDPs than just chess or Go using expert iteration... ("There is no fire alarm.")
- Houshalter 9y ago
  They aren't the first to apply NNs to chess though. What are they doing differently? And does anyone else smell smoke?
  - gwern 9y ago
    It's certainly not the first NN chess program. You may remember one of OP author's Giraffe NN (https://arxiv.org/abs/1509.01549 https://arxiv.org/abs/1509.01549) which was essentially 'AlphaGo for chess'. But like the original AG, it struggles to learn and Lai had a lot less computation as a student than he does now at DM. What they're doing is applying AlphaGo Zero expert iteration with some simplifications and TPUs. And that pwns previous work like Giraffe the way AlphaGo Zero pwns AlphaGo. Quantity becomes a quality all its own.
  - evandijk70 9y ago
    Look at Figure 2, and remember that DM has access to a lot of hardware. At short thinking times, AlphaZero is weaker than Stockfish. This is equivalent to longer thinking times with weaker hardware, and it is likely that the former applications of NNs to chess had hardware that was a 1000-fold slower than what DM has access to. This means that even if the approach was identical to DMs, they would not have seen a better performance of NNs than the classical alpha beta approach.
    - nojvek 9y ago
deleted 9y ago
[deleted]
firebones 9y ago
A lot of the graphs in the paper seem to level out as they hit the level of the opponent. It makes me wonder to what extent AlphaGo Zero is merely optimizing to beat flaws in existing opponents' current implementations (even if "existing opponents" == all available opponents' data and algorithms today) rather than generalizable insights into the underlying game. Because wouldn't you expect that unless we are at the theoretical limit of perfect chess that a tabula rasa approach might exceed existing best practice significantly, especially with the massive computation advantage it has? Not that there's anything wrong with that; AlphaGo Zero supposedly optimized for the "just enough" win rather than the crushing win. It doesn't even mean Stockfish is doomed--I suspect Stockfish could beat it in a future heads up match provided that Zero didn't have time to retrain, but that a retrained Zero (having the benefit of optimizing against a new Stockfish) would be able to supersede it once again.
- Houshalter 9y ago
  It's not. It learns entirely through self play and never learns from playing it's opponent. Diminishing returns isn't unusual and happens in every domain. These AIs are probably playing close to the limit of what is possible, just not quite there yet.
  - tlb 9y ago
    Are there popular games where the best human players are not near the limit of what is possible? Obviously you can construct one to be hard for humans (large 3SAT problems, or even big arithmetic problems), but I wonder if there is one that people enjoy.
    - throw_away_777 9y ago
      Humans are nowhere near the limit of what is possible in chess, as evidenced by how much better computers are at the game.
      - nandemo 9y ago
        Presumably tlb meant what is humanly possible...
hyperpape 9y ago
This paper compares AlphaZero to the 20 block version of AlphaGo Zero that was trained for 3 days. Am I right in thinking that this version was significantly less strong than the 40 block version? If so, does it matter?
lern_too_spel 9y ago
What is its win percentage against itself on each side of the board in each game? Is chess a draw for its style of play? Is there a first move advantage for the other games with its play style?
xianshou 9y ago
One impressive statistic from the paper: AlphaZero analyzes 80,000 chess positions per second, while Stockfish looks at 70,000,000. Seventy million, three orders of magnitude higher. Yet AG0 beats Stockfish half the time as White and never loses with either color. A stunning demonstration of generality indeed.
- 1024core 9y ago
  So ... what if you combined Stockfish and AG0, and let AG0 explore 70M positions instead of 80K? Would it improve even faster?
  - thom 9y ago
    The issue is you can’t evaluate positions that fast in AlphaZero (currently).
    - thom 9y ago
      It would be interesting to see if there were some way to extract a couple of new heuristics from AlphaZero that could be implemented fast enough to incorporate in Stockfish's evaluator though. I suppose this is the age old problem of black-box models: _why_ does it think this?
      - bhouston 9y ago
        I think that it is almost always possible to extract optimized models from nn and implement them faster. I wonder if this can be generalized. Nn to optimized fix algo for Max speed? This has to already exist as it is very obvious.
        thom 9y ago
        I dunno, seems like Google would just do this instead of keep around the pesky neural net at runtime. There's an _awful_ lot of computation going on inside, and it's necessarily hugely interconnected. I'd be impressed if someone had already done it, but it seems a great avenue of research if not. I suppose it goes hand in hand with models for which you can actually _explain_ their results, which certainly is an active area of research.
luckyt 9y ago
It doesn't seem to like the Sicilian Defense (1.e4 c5), which is the most popular opening by human players. I wonder if this will change opening theory?
- SubiculumCode 9y ago
  That's stunning. I thought that was one of the strongest openings for black.
  - SubiculumCode 9y ago
    It seemed to play a lot of English Opening...that also seems strange to me.
  - thom 9y ago
    I think in a way it’s an opening that rewards preparation and theory. With near perfect play expected on both sides, what seem like sharp games to humans are quite easily navigable.
  - Isofarro 9y ago
    Not strongest opening, but because it's an asymmetric opening system, which introduces imbalance into the position, thus it tends to have less drawish tendencies than a symmetric opening system. This creates the psychological effect of slightly turning the knob of "Black is playing for equality", to "Black is playing for counter-play".
- dnsva 9y ago
  Combinations appear in Sicilian often. If you're playing black, that's a good thing depending on your rating.
- dnsva 9y ago
  A list of openings in recent world champions: https://www.chess.com/blog/ih8sens/world-championship-openings https://www.chess.com/blog/ih8sens/world-championship-openin... Queen's gambit is there.
- 9y ago
Sukotto 9y ago
Is this a library or something I can download and try training myself (on a small scale)? I'm not in a position to read the paper right now, so my apologies if that's covered in there. I want to ask just in case it's not, while this is still on the front page.
- gwern 9y ago
  No. DM only occasionally releases software. Expert iteration is simple enough that someone can code it up on their own and there's already a few clones, so if anyone cares to train their own, it's doable, although it may take a while.
  - chillee 9y ago
    "a while" is a bit of an understatement. Leela zero (the main alphago zero replication project) is a crowd sourced computation effort that's going to take a fairly long time to get anywhere. And from this paper: > "Training proceeded for 700,000 steps (mini-batches of size 4,096) starting from randomly initialised parameters, using 5,000 first-generation TPUs (15) to generate self-play games and 64 second-generation TPUs to train the neural networks."
    - Houshalter 9y ago
      You don't have to start from zero though. It's cool that it works with google scale resources. But it seems like it would be faster to initialize with a neural net first trained to mimic the moves of an existing chess or Go AI. And then improve it from there. >"Why is the net wired randomly?", asked Minsky. "I do not want it to have any preconceptions of how to play", Sussman said. Minsky then shut his eyes. "Why do you close your eyes?", Sussman asked his teacher. "So that the room will be empty." At that moment, Sussman was enlightened.
      - nandemo 9y ago
        I'm pretty sure starting from zero is the point of the Leela-Zero. If they started from Stockfish, it wouldn't be a replication of AlphaZero.
asdfologist 9y ago
While this sounds impressive, I'll believe it when AlphaZero wins TCEC.
- computerphage 9y ago
  It beat the winner of TCEC-2016, Stockfish, with a record of 28-72-0. That's zero losses.
  - namelost 9y ago
    They didn't demonstrate that AlphaGo Zero can beat Stockfish in a fair contest: i.e. take the amount of money they spent on Stockfish's CPU and RAM, buy a commodity GPU for AlphaGo and then see.
    - vinchuco 9y ago
      https://news.ycombinator.com/item?id=2232163 https://news.ycombinator.com/item?id=2232163
      - namelost 9y ago
        I'm sorry, I thought we were discussing the paper.
  - Scarblac 9y ago
    On completely different hardware.
    - deleted 9y ago
      [deleted]
  - bluecalm 9y ago
    If I run SF on my desktop computer it will kill SF run on my phone. It doesn't prove anything. Comparing TPUs and CPUs is hard but they could've at least let SF run on what is considered top of the line setup and sensible settings (1GB hash memory is very limited, 8GB is standard for rapid games on a quad core CPU, let alone 64core one).
tboerstad 9y ago
Stockfish plays like an ambitious amateur in the first game, giving away a piece for two pawns on move 13. Perhaps this move was justified though, as later in the same game Stockfish gets a position which is at worst drawn, likely winning. Moves later however, around move 40, Stockfish gets its own knight trapped and the game is over. This is not the kind of chess we normally see from Stockfish.
- rozim 9y ago
  Yeah, that game was kind of different from the others - in the other games the feeling I got was that over time AphaGo's pieces got increasingly effective while Stockfish's pieces would get bottled up and lose their mobility.
foobaw 9y ago
Did Magnus play against this? Is there a way we can see the game?
- NegatioN 9y ago
  No he didn't play it. As far as I know, computers are already far ahead of humans in chess, so a further progress in this wouldn't really make a difference.
  - foobaw 9y ago
    You're right that it's pointless. The paper has the game with Stockfish so that's good enough for me.
  - thomasahle 9y ago
    It would be interesting in one way though: Magnus says he hates playing against computers, because "it's like being beaten by an idiot". Modern chess engines still make moves that are somewhat strategically weak, but they make up for it with amazing tactics. It would be interesting to hear if Magnus thought AlphaZero played less like an idiot.
zwischenzug 9y ago
I smell a rat. The paper says: 'AlphaZero achieved within 24 hours a superhuman level of play in the games of chess and shogi' In the first game Stockfish's, 9. Qe1 is one of the strangest moves I've ever seen, which would never be considered by a human, let alone a superhuman. 11. Kh1 also makes little sense, but is not as bad. My Stockfish sees it as losing 0.2 pawns, which makes it highly suspect in such a position. 35. Nc4 is also a deeply puzzling move that my Stockfish sees as losing half a pawn immediately, and a whole pawn soon after. 50. g4 also suspect 52. e5 is insane. This is bullshit. Edit: bullshit is too much - see comments below. Edit: Oh dear. We're doomed. https://lichess.org/study/qiwMCyNQ https://lichess.org/study/qiwMCyNQ
- allenz 9y ago
  > which would never be considered by ... a superhuman How would you know?
  - zwischenzug 9y ago
    Ask any GM.
    - gamegoblin 9y ago
      How would a GM (human) know what would be played by a superhuman?
  - thom 9y ago
    I don’t see it in my database, and it’s never been played on Lichess, even in bullet games.
    - allenz 9y ago
      Which, of course, is not evidence that a superhuman wouldn't consider such a move. AlphaGo also made unusual moves that looked like mistakes, but turned out to be insights.
thom 9y ago
I can’t see any reference to whether Stockfish was configured with an endgame tablebase. It’d be interesting to see results then, as you’d expect AlphaZero’s superior evaluation to give it an advantage out of the opening, but later in the game Stockfish would have access to perfect evaluations. Obviously there’s nothing stopping you from plugging a tablebase into AlphaZero but that feels wrong.
- thomasahle 9y ago
  It's not clear that it had an opening book either. In any case it's not specified which one.
Aissen 9y ago
Serious question: how does one evaluate the results reproducibility of this paper ? Maybe I'm missing some things but: - Are 1st gen TPUs even accessible ? You have to fill out a form to learn more about those second generation TPUs: https://cloud.google.com/tpu/ https://cloud.google.com/tpu/ - I can't find the source code This does not look like a scientific paper, but a (very impressive) tech demo.
- gcp 9y ago
  There are too many details missing for the results to be reproducible. Does it even qualify as a tech demo if the result only exists in DeepMind's lab?
  - bawana 9y ago
    This raises an interesting concept. If you cannot reproduce an experiment because of lack of resources, can you believe it? Or is this the equivalent of 'photoshopping your results'? A similar problem exists in cosmology. Can you verify the multiverse model if you only have one universe to experiment in? As data storage requirements in RAM and TPU power requirements increase to run certain models/algorithms, machine learning is becoming more obscure. Not only can we not understand how an AI is reaching its conclusions (inscrutability), we cannot even probe it (by tweaking parameters, etc) to find weak points (inaccesibility). This is actually a good thing. Where humans cannot tread, there can be no evil?
    - gcp 9y ago
      At least in computer chess such experiments were typically demonstrated by winning the World Championship. (And sometimes they failed...cough Deep Blue cough)
- 317070 9y ago
  How do you replicate CERN experiments? The LHC? Hubble? LIGO? LISA? At least this paper is reproducible by people who have the compute, and many universities have super computers. Even at home, you can verify the results by replaying the games against stockfish. You might not be able to replicate the setup at home, but that does not mean it is not science.
nl 9y ago
For those complaining about the TPU resources used during self training it is worth noting that Stockfish has used over 10,000 CPU hours for tuning its parameters. See https://github.com/mcostalba/Stockfish/blob/master/Top%20CPU%20Contributors.txt https://github.com/mcostalba/Stockfish/blob/master/Top%20CPU...
- dmurray 9y ago
  This understates it a bit. More like 10 million CPU hours according to that link.
elcapitan 9y ago
What would be a good starting point to learn about the AI behind that for a "normal" programmer? There seem to be so many resources now that it's hard to choose. Combination of hands-on plus theory would be good.
- thanatropism 9y ago
  The keyword is "reinforcement learning".
  - elcapitan 9y ago
    I know the names of the general concepts, I was wondering if someone has concrete recommendations on where to start and which books/frameworks are sort of beginner-friendly.
    - AlexCoventry 9y ago
      Try Hands-On Machine Learning with Scikit-Learn and TensorFlow Concepts, Tools, and Techniques to Build Intelligent Systems for the fundamentals. http://shop.oreilly.com/product/0636920052289.do http://shop.oreilly.com/product/0636920052289.do For reinforcement learning, I hear Barto and Sutton is very readable, but I haven't read it myself. You can just pick the concepts up by reading papers. The introduction in the Deep Q-Learning paper is not great, but it's how I first learned the concept. http://ufal.mff.cuni.cz/~straka/courses/npfl114/2016/sutton-bookdraft2016sep.pdf http://ufal.mff.cuni.cz/~straka/courses/npfl114/2016/sutton-... https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf
    - magoghm 9y ago
      You can check this Reinforcement Learning Course by David Silver on YouTube: https://www.youtube.com/watch?v=2pWv7GOvuf0&t=836s https://www.youtube.com/watch?v=2pWv7GOvuf0&t=836s By the way, I believe David Silver was the lead programmer for AlphaZero.
- tom_wilde 9y ago
  Coursera - Andrew NG's course => Classic starting point, very thorough and digestable introduction to Neural Networks. I found he covered the 'how the heck do I use this?' rather well.. :) From there, Coursera has a paid(?) DL course by Andrew NG or there's Fast.ai which looks good. Good luck!
narrator 9y ago
So when are they going to apply this to Atari Games or well anything? The next step is they have one AI figure out the rules by making a GAN that imitates player behavior and the other AI be Alpha Go which tweaks the GAN inputs to generate different moves to win. Voila...Almost General Purpose AI that can learn to play any game.
- eref 9y ago
  The main problem is that we still lack good generative models and good ways of interrogating them. GANs are unstable and difficult to apply to time series, VAEs suffer from posterior collapse, WaveNet/PixelRNN grow with the input size and overemphasize the details, RNNs are hard to train because we lack good training algorithms. Generally, small errors tend to compound in step-wise predictions because NNs do not generalize very well and gradients tend to vanish and shatter. If you just regard computation time to roll out the future, modeling domains in which the rules are simple enough to be hand-coded and evaluated quickly (such as Go and Chess) probably makes MCTS a million times more suitable compared to domains in which you need a complex model.
  - gwern 9y ago
    To expand on eref's comment a little: you absolutely could apply this or MCTS to ALE (and Guo et al 2014 did it very nicely). After all, the ALE is deterministic and simulatable by definition, so of course you can explore the game tree and reset the simulation as necessary. But people aren't much interested in this approach because using the ALE as a 'simulator' is cheating as far as testing full-strength AI techniques (we don't have simulators of the real world, after all), and the ALE games themselves (unlike Go) are of little intrinsic interest so there's no real benefit to engaging in cheating.
- namarie 9y ago
  Didn't they start it all by playing atari games?
110011 9y ago
What an amazing result! Evaluating fewer (by a factor of 1000) positions AlphaZero still beats Stockfish. In the figure on its preferred openings I find it very interesting that it doesn't like the Ruy Lopez very much over training time (there is a small bump but that is transient). I am hardly a chess expert but I know that it was very favored at the world championships so maybe the chess world will be turned upside down by this result now? Positing that the chess world is bigger than the Go world (in terms of interest and finances) there is probably going to be a race to replicate these results "at home" and train yourself before your competitors :)
soveran 9y ago
The ten sample games: Sample game 1 https://lichess.org/VMe0gfa2 https://lichess.org/VMe0gfa2 Sample game 2 https://lichess.org/Zqwn4Gzk https://lichess.org/Zqwn4Gzk Sample game 3 https://lichess.org/G2fPHci8 https://lichess.org/G2fPHci8 Sample game 4 https://lichess.org/LLt8wyYp https://lichess.org/LLt8wyYp Sample game 5 https://lichess.org/3r6CXx3H https://lichess.org/3r6CXx3H Sample game 6 https://lichess.org/sbdyUYS4 https://lichess.org/sbdyUYS4 Sample game 7 https://lichess.org/88vsAftE https://lichess.org/88vsAftE Sample game 8 https://lichess.org/1uvCwaeB https://lichess.org/1uvCwaeB Sample game 9 https://lichess.org/743quCXj https://lichess.org/743quCXj Sample game 10 https://lichess.org/SkCjxXkb https://lichess.org/SkCjxXkb
- EvgeniyZh 9y ago
  So, 1. d4 for White, Berlin for Black. I got it
- sireat 9y ago
  Time and again Alpha shows it is much better at eval than Stockfish. Alpha play feels "human" at least to this FM. This is fantastic news! It is what I would imagine a good correspondence GM would play like with engine assistance. I already commented on Game 1 where Stockfish played extremely aggressively with 13. Ncxe5 ??! and 31. Qxc7 ?! Game 3 is a positional masterpiece. Alpha is willing to play pawns + exchange down when it correctly evaluates that Black queen and rooks will be tied down. This kind of long term thinking is beyond what regular engines perform. Game 10 is also an impressive showing by Alpha. Alpha is willing to play down a piece and a pawn for 15 (30 ply) moves in a middle game beyond the reach of Stockfish's raw calculations. If one could only get access to Alpha evals :) When do mere mortals get access to TPUs on Google Compute Engine?
  - forgot-my-pw 9y ago
    Thanks for the analysis. There's a project currently that emulates AlphaGo Zero using distributed computing / crowdsourcing: https://github.com/gcp/leela-zero https://github.com/gcp/leela-zero . You can run it on the browser too and it will submit the games after: https://ntt123.github.io/leela-zero/ https://ntt123.github.io/leela-zero/ Hope such a project will be available soon for the chess variant. Or maybe Deepmind will release this as a SaaS product?
bfirsh 9y ago
Here's an HTML version of the paper: https://www.arxiv-vanity.com/papers/1712.01815/ https://www.arxiv-vanity.com/papers/1712.01815/ Table 2 is broken, but the rest is much more readable if you're on a phone.
Scarblac 9y ago
As a chess player I find the win rate astonishing. Given the drawish tendency at top level, among human players, in correspondence chess and also in the TCEC final, I thought that even absolutely perfect play wouldn't score so well against a decent Stockfish setup (which 64 cores and 1 minute per move should be).
thomasahle 9y ago
Discussion at the Computer Chess Club (CCC) forum: http://www.talkchess.com/forum/viewtopic.php?topic_view=threads&p=741214&t=65910 http://www.talkchess.com/forum/viewtopic.php?topic_view=thre... and http://www.talkchess.com/forum/viewtopic.php?topic_view=threads&p=741211&t=65909 http://www.talkchess.com/forum/viewtopic.php?topic_view=thre...
hmate9 9y ago
So AlphaGo Zero used 4 TPUs while AlphaZero used 1500. It’s not immediately obvious to me why there is this massive difference. Can anyone elaborate?
- sanxiyn 9y ago
  Both used 4 TPUs at playing time. At training time, AlphaGo Zero used unspecified amount of computing resource, AlphaZero used 5000 TPUs for self-play.
  - hmate9 9y ago
    Ah, thanks for clearing that up! Makes sense.
imrehg 9y ago
As a Shogi enthusiast (but complete beginner), I'd like to have seen more Shogi details in the article. Nevertheless there's plenty of other things to geek out on...
naveen99 9y ago
Is there an sdk or compiler for using the google tpu's beyond just using tensorflow ? Is the tpu backend of tensorflow based on cuda, opencl, plain c or something else ?
deleted 9y ago
[deleted]
k2xl 9y ago
Chess.com forum thread https://www.chess.com/forum/view/general/stockfish-dethroned https://www.chess.com/forum/view/general/stockfish-dethroned
auggierose 9y ago
Great result, but without access to source code this is not a scientific paper.
plg 9y ago
source code?
cdelsolar 9y ago
I wanted to contact the authors directly but can't seem to find contact info at the moment, with a question. I hope some of you might know enough to answer it. I'm interested in applying this method, or a similar neural-network / tabula rasa based method to the game of Scrabble. I read the original AlphaGo Zero paper and they mentioned that this method works best for games of perfect information. The standard Scrabble AI right now is quite good and can definitely beat top experts close to 50% of the time, but it uses simple Monte Carlo simulations to evaluate positions and just picks the ones that perform better. It doesn't quite account for defensive considerations or other subtleties of the game. I was wondering if anyone who had more insight into MCTS and NN would be able to talk me through how to apply this to Scrabble, or if it even makes sense. One of the issues I can see currently would be very slow convergence; as it has a luck factor, the algorithm could make occasional terrible moves and still win games, and thus be "wrongly trained".
- bo1024 9y ago
  Step 1: millions (?) of dollars of hardware.
SubiculumCode 9y ago
There is only one way for a human to win at chess against these computers; and it involves violence against the chess board.
wskish 9y ago
The more interesting metric going forward is performance at a given power budget (not unlike with motorsports). The TPUs are consuming sooo much power here! Most interesting real-world problems are power-limited, including in nature (e.g. metabolic limits).
- orasis 9y ago
  When a lot of money is on the line you can use a lot of resources.
skc 9y ago
I'm only a fairly pedestrian chess player, but I looked at one of these games between AGZ and SF and aside from the endgame, AGZ played in a manner that almost seemed alien. It seemed to completely ignore various little rules of thumb which is to be expected in hindsight but fairly mind-blowing when you actually watch a game.
TwoBit 9y ago
Wasn't Stockfish gimped for this competition? No openings, no endgame tables, low RAM, etc? If that's so then this AI did not in fact beat the computer chess champ.

539 points by dennybritz 9y ago

See, Mom? Self play is a good thing.

Certainly a significant achievement. Also, kind of interesting that the AlphaGo team spent a lot of energy to convince us Go is much harder than Chess, only to turn around and tell us that it is amazing that it can also win at Chess.

2bitencryption 9y ago
> only to turn around and tell us that it is amazing that it can also win at Chess. What they're demoing here is a single, general formula for mastering multiple games. Start with empty AG0, then teach it chess from scratch until it is the strongest player on the planet. Go back to an empty slate, with the same exactly "untrained" AG0, and now teach it Go, to the same result. No fine-tuning for the domain of the game you are training -- it is general(ized). That's the gist I'm getting from this. question for someone who has time to read the paper: can you train it to master chess and go at the same time? or is it one or the other? I'm assuming the latter. edit: check out the graph on the 4th page. AlphaZero, which can master chess and shogi, can beat AlphaGo Zero, the implementation specifically designed for Go, at its own game.
- visarga 9y ago
  Question: do you think you are using the same parts of the brain to play chess and Go? What counts is not using the same neurons, but using the same neural algorithm.
- gwern 9y ago
  > question for someone who has time to read the paper: can you train it to master chess and go at the same time? or is it one or the other? I'm assuming the latter. I'm sure you could with a multi-headed NN. But what would be the point? There's very little transfer of knowledge between the games, especially once you get past the very most basics.
  - thanatropism 9y ago
    The point is that real problem domains are not neatly partitioned and labeled. I don't know what kind of input the NN itself gets, but computer vision is enough to translate a photo of a chessboard to a usable symbolic representation. But it would be nice to already have a black box-ish computer program that figures out what's the game at hand and how to play it. The next variation is have the adversary start playing a chess variant and have the machine recognize it (assuming honesty) and play it to significant skill. Then "real life Pong" where the size and aerodynamics of the ball are unknown to it. This is the gist of human intelligence: answering questions is significantly easier than figuring out what the question is.
- jmmcd 9y ago
  > Go back to an empty slate, with the same exactly "untrained" AG0, and now teach it Go, to the same result. No fine-tuning for the domain of the game you are training -- it is general(ized). Not quite -- different input features, which implies slightly different network architecture at least at the front.

gallerdude 9y ago

I wonder if being an expert at one game makes it easier to be an expert at another. If so, then maybe the examples are datasets, and convergence would be able to complete new tasks after a few examples.

vinchuco 9y ago
Really interesting question. Some strategic concepts may transfer, say, from chess to chess variants. However, a simple change in the rules can have a huge impact in the game mechanics as anyone who has tried chess variants [1] knows. [1] https://en.wikipedia.org/wiki/List_of_chess_variants https://en.wikipedia.org/wiki/List_of_chess_variants The answer may be that it is hard enough to become an expert at anything, but there may be some serendipitous (how to make this precise?) overlap.
foota 9y ago
It would be very interesting to see someone try something like transfer learning from one game to another.
jmmcd 9y ago
Well, it's not doing anything like that for now. Even though the algorithm, in an abstract sense, is the same for all three games, in fact it's a new network for each of the three games, with architecture and input features adapted to the game, and then trained from scratch.

Invictus0 9y ago

I'm not sure it's really fair to compare Stockfish to AlphaZero; AlphaZero used 24h of 5000 TPUs in compute time, and still needed 4 TPUs in real play, while Stockfish ran on just 64 threads and 1GB RAM. Nonetheless, still an impressive achievement.

nandemo 9y ago
Wait, how's the 24h x 5000 TPUs relevant? That is training time, and that training corresponds to years and years of hardcoding evaluations in Stockfish, not to compute time during the match.
Recursing 9y ago
Only 1GB RAM? Really?
- bluecalm 9y ago
  Yes, this is really strange. Hash table size is a major contributing factor for strength of chess programs. It looks like a very artificial limitation.

partycoder 9y ago

If you have seen the Stockfish project you will see many hardcoded weights in the configuration, found through experimentation. All these adjustments took probably years to achieve... and now Alpha Go Zero just self-learns everything and surpasses it. Would be good to see Deepmind's solution play Arimaa and Stratego, and see what kind of strategy it comes up with. Or weird variations of Go. Eventually this tech will make it into military strategy simulators and that's where things will get really messed up. 4 star generals will be replaced by bots.

gamegoblin 9y ago
I don't think this technique immediately applies to Stratego because it's not a perfect information game. I suspect it would exceed the state of the art in Arimaa, since Arimaa is specifically designed to have a high branching factor (17281 -- compared to 35 for chess), and this technique was designed to work well in high-branching factor games (since Go is a high-branching factor game, though much lower than Arimaa).
- partycoder 9y ago
  In that regard then Stratego would share some aspects with Starcraft, another incomplete information game. Deepmind is actively working in a StarCraft bot. It would be interesting to see if they can be put together a supraintelligent StarCraft bot and then translate those results to Stratego.

ericand 9y ago

Two things to note: 1) Alpha Zero beats AlphaGo Zero and AlphaGo Lee and starts tabla rasa 2) "Shogi is a significantly harder game, in terms of computational complexity, than chess (2, 14): it is played on a larger board, and any captured opponent piece changes sides and may subsequently be dropped anywhere on the board. The strongest shogi programs, such as Computer Shogi Association (CSA) world-champion Elmo, have only recently defeated human champions (5)"

pacaro 9y ago
Shogi is a fun game, it always feels a little sad that it doesn't get more exposure outside of Japan (and my understanding is that, by and large, in Japan it is considered an "old persons" game) Because captured pieces change sides, there is less of an "endgame" scenario, and as a beginner (like me) it is very easy to put too many captured pieces back into play, which makes it hard to defend everything and essentially you end up giving them back to your opponent
- foobaw 9y ago
  It briefly became popular in the otaku culture from an anime called Hunter X Hunter.
  - mikekchar 9y ago
    I'm curious to see if "San Gatsu no Lion" (the Lion of March) will spark interest. I highly recommend it to anyone interested in more slice-of-life/drama kinds of things. It's quite a beautiful anime/manga, even if the shogi isn't quite centre stage.
    - sanxiyn 9y ago
      Recommendation seconded, Sangatsu no Lion is a lovely work. On the other hand, it has been running for 10 years (!), if it could spark interest like Hikaru no Go, it would have happened already.

naveen99 9y ago

Very happy to see this result. It's like a moral victory for humans, as alphago is more human like (discounting montecarlo search) than stockfish. Maybe deep learning will give us the next Euler, Newton, or Einstein.

visarga 9y ago
Shogi, chess and Go are "perfect information games", meaning you can see the whole game state. It's a whole different thing to be able to solve games where you don't see everything (based on uncertainty).
- jacobkg 9y ago
  In 2015 Heads-Up Limit Hold'em was solved: http://science.sciencemag.org/content/347/6218/145.full http://science.sciencemag.org/content/347/6218/145.full
- ucaetano 9y ago
  You mean, like poker? https://www.cmu.edu/news/stories/archives/2017/january/AI-beats-poker-pros.html https://www.cmu.edu/news/stories/archives/2017/january/AI-be... Actually machines can have an even higher advantage in those cases, because they can be much better at estimating probabilities than humans. Think of card counting, for example.
  - grumpopotamus 9y ago
    I disagree. Computers have been outplaying the best humans at chess for two decades, but they only recently beat the top players at 2-player NLHE and only with the aid of massive computational power during training. Furthermore, techniques like monte-carlo tree search used in AlphaGo don't work very well for poker - You can't just try and find the "best move" from the current game state, or you will end up playing a highly-exploitable strategy. You essentially have to solve the entire game every time (or completely in advance) to make sure you are playing a balanced strategy. Only the Counter-Factual Regret Minimization algorithm has been able to achieve this level of play in Heads Up, and right now it looks hard to scale to poker games with more players, like the full-ring games you see at the World Series of Poker, for example. We still have a ways to go in Poker AI.

magoghm 9y ago

"We also analysed the relative performance of AlphaZero’s MCTS search compared to the state-of-the-art alpha-beta search engines used by Stockfish and Elmo. AlphaZero searches just 80 thousand positions per second in chess and 40 thousand in shogi, compared to 70 million for Stockfish and 35 million for Elmo. AlphaZero compensates for the lower number of evaluations by using its deep neural network to focus much more selectively on the most promising variations – arguably a more “human-like” approach to search, as originally proposed by Shannon." <- Amazing!

maxander 9y ago
Meanwhile a human player considers <1 position per second, so there’s a few orders of magnitude left to go in that direction. But unsettlingly few, nonetheless.
- nopinsight 9y ago
  Humans are also much weaker than AlphaZero in these three games. The difference in the numbers of positions searched might be responsible for a substantial part of that.
  - yters 9y ago
    It'd be interesting to weaken AZ until it is on par with a human, and then compare moves evaluated. I'd suspect humans still evaluate significantly fewer moves.
- aidenn0 9y ago
  Strong human players consider a lot more than 1 position per second in Chess...

gwern 9y ago

This is an incredible demonstration that the AG Zero expert iteration method is a general method. If you go back to the discussions of AG Zero lo a month ago, there was a lot of skepticism that NNs would ever challenge Stockfish et al - they are just too good, too close to perfection, and chess not well suited for MCTS and NNs. Well, it turns out that AG Zero doesn't work as well in chess: it works better as it only takes 4 hours of training to beat Stockfish. This is going to be an impetus for researchers to explore solving many more MDPs than just chess or Go using expert iteration... ("There is no fire alarm.")

Houshalter 9y ago
They aren't the first to apply NNs to chess though. What are they doing differently? And does anyone else smell smoke?
- gwern 9y ago
  It's certainly not the first NN chess program. You may remember one of OP author's Giraffe NN (https://arxiv.org/abs/1509.01549 https://arxiv.org/abs/1509.01549) which was essentially 'AlphaGo for chess'. But like the original AG, it struggles to learn and Lai had a lot less computation as a student than he does now at DM. What they're doing is applying AlphaGo Zero expert iteration with some simplifications and TPUs. And that pwns previous work like Giraffe the way AlphaGo Zero pwns AlphaGo. Quantity becomes a quality all its own.
- evandijk70 9y ago
  Look at Figure 2, and remember that DM has access to a lot of hardware. At short thinking times, AlphaZero is weaker than Stockfish. This is equivalent to longer thinking times with weaker hardware, and it is likely that the former applications of NNs to chess had hardware that was a 1000-fold slower than what DM has access to. This means that even if the approach was identical to DMs, they would not have seen a better performance of NNs than the classical alpha beta approach.
  - nojvek 9y ago

deleted 9y ago

[deleted]

firebones 9y ago

A lot of the graphs in the paper seem to level out as they hit the level of the opponent. It makes me wonder to what extent AlphaGo Zero is merely optimizing to beat flaws in existing opponents' current implementations (even if "existing opponents" == all available opponents' data and algorithms today) rather than generalizable insights into the underlying game. Because wouldn't you expect that unless we are at the theoretical limit of perfect chess that a tabula rasa approach might exceed existing best practice significantly, especially with the massive computation advantage it has? Not that there's anything wrong with that; AlphaGo Zero supposedly optimized for the "just enough" win rather than the crushing win. It doesn't even mean Stockfish is doomed--I suspect Stockfish could beat it in a future heads up match provided that Zero didn't have time to retrain, but that a retrained Zero (having the benefit of optimizing against a new Stockfish) would be able to supersede it once again.

Houshalter 9y ago
It's not. It learns entirely through self play and never learns from playing it's opponent. Diminishing returns isn't unusual and happens in every domain. These AIs are probably playing close to the limit of what is possible, just not quite there yet.
- tlb 9y ago
  Are there popular games where the best human players are not near the limit of what is possible? Obviously you can construct one to be hard for humans (large 3SAT problems, or even big arithmetic problems), but I wonder if there is one that people enjoy.
  - throw_away_777 9y ago
    Humans are nowhere near the limit of what is possible in chess, as evidenced by how much better computers are at the game.
    - nandemo 9y ago
      Presumably tlb meant what is humanly possible...

hyperpape 9y ago

This paper compares AlphaZero to the 20 block version of AlphaGo Zero that was trained for 3 days. Am I right in thinking that this version was significantly less strong than the 40 block version? If so, does it matter?

lern_too_spel 9y ago

What is its win percentage against itself on each side of the board in each game? Is chess a draw for its style of play? Is there a first move advantage for the other games with its play style?

xianshou 9y ago

One impressive statistic from the paper: AlphaZero analyzes 80,000 chess positions per second, while Stockfish looks at 70,000,000. Seventy million, three orders of magnitude higher. Yet AG0 beats Stockfish half the time as White and never loses with either color. A stunning demonstration of generality indeed.

1024core 9y ago
So ... what if you combined Stockfish and AG0, and let AG0 explore 70M positions instead of 80K? Would it improve even faster?
- thom 9y ago
  The issue is you can’t evaluate positions that fast in AlphaZero (currently).
  - thom 9y ago
    It would be interesting to see if there were some way to extract a couple of new heuristics from AlphaZero that could be implemented fast enough to incorporate in Stockfish's evaluator though. I suppose this is the age old problem of black-box models: _why_ does it think this?
    - bhouston 9y ago
      I think that it is almost always possible to extract optimized models from nn and implement them faster. I wonder if this can be generalized. Nn to optimized fix algo for Max speed? This has to already exist as it is very obvious.
      - thom 9y ago
        I dunno, seems like Google would just do this instead of keep around the pesky neural net at runtime. There's an _awful_ lot of computation going on inside, and it's necessarily hugely interconnected. I'd be impressed if someone had already done it, but it seems a great avenue of research if not. I suppose it goes hand in hand with models for which you can actually _explain_ their results, which certainly is an active area of research.

luckyt 9y ago

It doesn't seem to like the Sicilian Defense (1.e4 c5), which is the most popular opening by human players. I wonder if this will change opening theory?

SubiculumCode 9y ago
That's stunning. I thought that was one of the strongest openings for black.
- SubiculumCode 9y ago
  It seemed to play a lot of English Opening...that also seems strange to me.
- thom 9y ago
  I think in a way it’s an opening that rewards preparation and theory. With near perfect play expected on both sides, what seem like sharp games to humans are quite easily navigable.
- Isofarro 9y ago
  Not strongest opening, but because it's an asymmetric opening system, which introduces imbalance into the position, thus it tends to have less drawish tendencies than a symmetric opening system. This creates the psychological effect of slightly turning the knob of "Black is playing for equality", to "Black is playing for counter-play".
dnsva 9y ago
Combinations appear in Sicilian often. If you're playing black, that's a good thing depending on your rating.
dnsva 9y ago
A list of openings in recent world champions: https://www.chess.com/blog/ih8sens/world-championship-openings https://www.chess.com/blog/ih8sens/world-championship-openin... Queen's gambit is there.
9y ago

Sukotto 9y ago

Is this a library or something I can download and try training myself (on a small scale)? I'm not in a position to read the paper right now, so my apologies if that's covered in there. I want to ask just in case it's not, while this is still on the front page.

gwern 9y ago
No. DM only occasionally releases software. Expert iteration is simple enough that someone can code it up on their own and there's already a few clones, so if anyone cares to train their own, it's doable, although it may take a while.
- chillee 9y ago
  "a while" is a bit of an understatement. Leela zero (the main alphago zero replication project) is a crowd sourced computation effort that's going to take a fairly long time to get anywhere. And from this paper: > "Training proceeded for 700,000 steps (mini-batches of size 4,096) starting from randomly initialised parameters, using 5,000 first-generation TPUs (15) to generate self-play games and 64 second-generation TPUs to train the neural networks."
  - Houshalter 9y ago
    You don't have to start from zero though. It's cool that it works with google scale resources. But it seems like it would be faster to initialize with a neural net first trained to mimic the moves of an existing chess or Go AI. And then improve it from there. >"Why is the net wired randomly?", asked Minsky. "I do not want it to have any preconceptions of how to play", Sussman said. Minsky then shut his eyes. "Why do you close your eyes?", Sussman asked his teacher. "So that the room will be empty." At that moment, Sussman was enlightened.
    - nandemo 9y ago
      I'm pretty sure starting from zero is the point of the Leela-Zero. If they started from Stockfish, it wouldn't be a replication of AlphaZero.

asdfologist 9y ago

While this sounds impressive, I'll believe it when AlphaZero wins TCEC.

computerphage 9y ago
It beat the winner of TCEC-2016, Stockfish, with a record of 28-72-0. That's zero losses.
- namelost 9y ago
  They didn't demonstrate that AlphaGo Zero can beat Stockfish in a fair contest: i.e. take the amount of money they spent on Stockfish's CPU and RAM, buy a commodity GPU for AlphaGo and then see.
  - vinchuco 9y ago
    https://news.ycombinator.com/item?id=2232163 https://news.ycombinator.com/item?id=2232163
    - namelost 9y ago
      I'm sorry, I thought we were discussing the paper.
- Scarblac 9y ago
  On completely different hardware.
  - deleted 9y ago
    [deleted]
- bluecalm 9y ago
  If I run SF on my desktop computer it will kill SF run on my phone. It doesn't prove anything. Comparing TPUs and CPUs is hard but they could've at least let SF run on what is considered top of the line setup and sensible settings (1GB hash memory is very limited, 8GB is standard for rapid games on a quad core CPU, let alone 64core one).

tboerstad 9y ago

Stockfish plays like an ambitious amateur in the first game, giving away a piece for two pawns on move 13. Perhaps this move was justified though, as later in the same game Stockfish gets a position which is at worst drawn, likely winning. Moves later however, around move 40, Stockfish gets its own knight trapped and the game is over. This is not the kind of chess we normally see from Stockfish.

rozim 9y ago
Yeah, that game was kind of different from the others - in the other games the feeling I got was that over time AphaGo's pieces got increasingly effective while Stockfish's pieces would get bottled up and lose their mobility.

foobaw 9y ago

Did Magnus play against this? Is there a way we can see the game?

NegatioN 9y ago
No he didn't play it. As far as I know, computers are already far ahead of humans in chess, so a further progress in this wouldn't really make a difference.
- foobaw 9y ago
  You're right that it's pointless. The paper has the game with Stockfish so that's good enough for me.
- thomasahle 9y ago
  It would be interesting in one way though: Magnus says he hates playing against computers, because "it's like being beaten by an idiot". Modern chess engines still make moves that are somewhat strategically weak, but they make up for it with amazing tactics. It would be interesting to hear if Magnus thought AlphaZero played less like an idiot.

zwischenzug 9y ago

I smell a rat. The paper says: 'AlphaZero achieved within 24 hours a superhuman level of play in the games of chess and shogi' In the first game Stockfish's, 9. Qe1 is one of the strangest moves I've ever seen, which would never be considered by a human, let alone a superhuman. 11. Kh1 also makes little sense, but is not as bad. My Stockfish sees it as losing 0.2 pawns, which makes it highly suspect in such a position. 35. Nc4 is also a deeply puzzling move that my Stockfish sees as losing half a pawn immediately, and a whole pawn soon after. 50. g4 also suspect 52. e5 is insane. This is bullshit. Edit: bullshit is too much - see comments below. Edit: Oh dear. We're doomed. https://lichess.org/study/qiwMCyNQ https://lichess.org/study/qiwMCyNQ

allenz 9y ago
> which would never be considered by ... a superhuman How would you know?
- zwischenzug 9y ago
  Ask any GM.
  - gamegoblin 9y ago
    How would a GM (human) know what would be played by a superhuman?
- thom 9y ago
  I don’t see it in my database, and it’s never been played on Lichess, even in bullet games.
  - allenz 9y ago
    Which, of course, is not evidence that a superhuman wouldn't consider such a move. AlphaGo also made unusual moves that looked like mistakes, but turned out to be insights.

thom 9y ago

I can’t see any reference to whether Stockfish was configured with an endgame tablebase. It’d be interesting to see results then, as you’d expect AlphaZero’s superior evaluation to give it an advantage out of the opening, but later in the game Stockfish would have access to perfect evaluations. Obviously there’s nothing stopping you from plugging a tablebase into AlphaZero but that feels wrong.

thomasahle 9y ago
It's not clear that it had an opening book either. In any case it's not specified which one.

Aissen 9y ago

Serious question: how does one evaluate the results reproducibility of this paper ? Maybe I'm missing some things but: - Are 1st gen TPUs even accessible ? You have to fill out a form to learn more about those second generation TPUs: https://cloud.google.com/tpu/ https://cloud.google.com/tpu/ - I can't find the source code This does not look like a scientific paper, but a (very impressive) tech demo.

gcp 9y ago
There are too many details missing for the results to be reproducible. Does it even qualify as a tech demo if the result only exists in DeepMind's lab?
- bawana 9y ago
  This raises an interesting concept. If you cannot reproduce an experiment because of lack of resources, can you believe it? Or is this the equivalent of 'photoshopping your results'? A similar problem exists in cosmology. Can you verify the multiverse model if you only have one universe to experiment in? As data storage requirements in RAM and TPU power requirements increase to run certain models/algorithms, machine learning is becoming more obscure. Not only can we not understand how an AI is reaching its conclusions (inscrutability), we cannot even probe it (by tweaking parameters, etc) to find weak points (inaccesibility). This is actually a good thing. Where humans cannot tread, there can be no evil?
  - gcp 9y ago
    At least in computer chess such experiments were typically demonstrated by winning the World Championship. (And sometimes they failed...cough Deep Blue cough)
317070 9y ago
How do you replicate CERN experiments? The LHC? Hubble? LIGO? LISA? At least this paper is reproducible by people who have the compute, and many universities have super computers. Even at home, you can verify the results by replaying the games against stockfish. You might not be able to replicate the setup at home, but that does not mean it is not science.

nl 9y ago

For those complaining about the TPU resources used during self training it is worth noting that Stockfish has used over 10,000 CPU hours for tuning its parameters. See https://github.com/mcostalba/Stockfish/blob/master/Top%20CPU%20Contributors.txt https://github.com/mcostalba/Stockfish/blob/master/Top%20CPU...

dmurray 9y ago
This understates it a bit. More like 10 million CPU hours according to that link.

elcapitan 9y ago

What would be a good starting point to learn about the AI behind that for a "normal" programmer? There seem to be so many resources now that it's hard to choose. Combination of hands-on plus theory would be good.

thanatropism 9y ago
The keyword is "reinforcement learning".
- elcapitan 9y ago
  I know the names of the general concepts, I was wondering if someone has concrete recommendations on where to start and which books/frameworks are sort of beginner-friendly.
  - AlexCoventry 9y ago
    Try Hands-On Machine Learning with Scikit-Learn and TensorFlow Concepts, Tools, and Techniques to Build Intelligent Systems for the fundamentals. http://shop.oreilly.com/product/0636920052289.do http://shop.oreilly.com/product/0636920052289.do For reinforcement learning, I hear Barto and Sutton is very readable, but I haven't read it myself. You can just pick the concepts up by reading papers. The introduction in the Deep Q-Learning paper is not great, but it's how I first learned the concept. http://ufal.mff.cuni.cz/~straka/courses/npfl114/2016/sutton-bookdraft2016sep.pdf http://ufal.mff.cuni.cz/~straka/courses/npfl114/2016/sutton-... https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf
  - magoghm 9y ago
    You can check this Reinforcement Learning Course by David Silver on YouTube: https://www.youtube.com/watch?v=2pWv7GOvuf0&t=836s https://www.youtube.com/watch?v=2pWv7GOvuf0&t=836s By the way, I believe David Silver was the lead programmer for AlphaZero.
tom_wilde 9y ago
Coursera - Andrew NG's course => Classic starting point, very thorough and digestable introduction to Neural Networks. I found he covered the 'how the heck do I use this?' rather well.. :) From there, Coursera has a paid(?) DL course by Andrew NG or there's Fast.ai which looks good. Good luck!

narrator 9y ago

So when are they going to apply this to Atari Games or well anything? The next step is they have one AI figure out the rules by making a GAN that imitates player behavior and the other AI be Alpha Go which tweaks the GAN inputs to generate different moves to win. Voila...Almost General Purpose AI that can learn to play any game.

eref 9y ago
The main problem is that we still lack good generative models and good ways of interrogating them. GANs are unstable and difficult to apply to time series, VAEs suffer from posterior collapse, WaveNet/PixelRNN grow with the input size and overemphasize the details, RNNs are hard to train because we lack good training algorithms. Generally, small errors tend to compound in step-wise predictions because NNs do not generalize very well and gradients tend to vanish and shatter. If you just regard computation time to roll out the future, modeling domains in which the rules are simple enough to be hand-coded and evaluated quickly (such as Go and Chess) probably makes MCTS a million times more suitable compared to domains in which you need a complex model.
- gwern 9y ago
  To expand on eref's comment a little: you absolutely could apply this or MCTS to ALE (and Guo et al 2014 did it very nicely). After all, the ALE is deterministic and simulatable by definition, so of course you can explore the game tree and reset the simulation as necessary. But people aren't much interested in this approach because using the ALE as a 'simulator' is cheating as far as testing full-strength AI techniques (we don't have simulators of the real world, after all), and the ALE games themselves (unlike Go) are of little intrinsic interest so there's no real benefit to engaging in cheating.
namarie 9y ago
Didn't they start it all by playing atari games?

110011 9y ago

What an amazing result! Evaluating fewer (by a factor of 1000) positions AlphaZero still beats Stockfish. In the figure on its preferred openings I find it very interesting that it doesn't like the Ruy Lopez very much over training time (there is a small bump but that is transient). I am hardly a chess expert but I know that it was very favored at the world championships so maybe the chess world will be turned upside down by this result now? Positing that the chess world is bigger than the Go world (in terms of interest and finances) there is probably going to be a race to replicate these results "at home" and train yourself before your competitors :)

soveran 9y ago

The ten sample games: Sample game 1 https://lichess.org/VMe0gfa2 https://lichess.org/VMe0gfa2 Sample game 2 https://lichess.org/Zqwn4Gzk https://lichess.org/Zqwn4Gzk Sample game 3 https://lichess.org/G2fPHci8 https://lichess.org/G2fPHci8 Sample game 4 https://lichess.org/LLt8wyYp https://lichess.org/LLt8wyYp Sample game 5 https://lichess.org/3r6CXx3H https://lichess.org/3r6CXx3H Sample game 6 https://lichess.org/sbdyUYS4 https://lichess.org/sbdyUYS4 Sample game 7 https://lichess.org/88vsAftE https://lichess.org/88vsAftE Sample game 8 https://lichess.org/1uvCwaeB https://lichess.org/1uvCwaeB Sample game 9 https://lichess.org/743quCXj https://lichess.org/743quCXj Sample game 10 https://lichess.org/SkCjxXkb https://lichess.org/SkCjxXkb

EvgeniyZh 9y ago
So, 1. d4 for White, Berlin for Black. I got it
sireat 9y ago
Time and again Alpha shows it is much better at eval than Stockfish. Alpha play feels "human" at least to this FM. This is fantastic news! It is what I would imagine a good correspondence GM would play like with engine assistance. I already commented on Game 1 where Stockfish played extremely aggressively with 13. Ncxe5 ??! and 31. Qxc7 ?! Game 3 is a positional masterpiece. Alpha is willing to play pawns + exchange down when it correctly evaluates that Black queen and rooks will be tied down. This kind of long term thinking is beyond what regular engines perform. Game 10 is also an impressive showing by Alpha. Alpha is willing to play down a piece and a pawn for 15 (30 ply) moves in a middle game beyond the reach of Stockfish's raw calculations. If one could only get access to Alpha evals :) When do mere mortals get access to TPUs on Google Compute Engine?
- forgot-my-pw 9y ago
  Thanks for the analysis. There's a project currently that emulates AlphaGo Zero using distributed computing / crowdsourcing: https://github.com/gcp/leela-zero https://github.com/gcp/leela-zero . You can run it on the browser too and it will submit the games after: https://ntt123.github.io/leela-zero/ https://ntt123.github.io/leela-zero/ Hope such a project will be available soon for the chess variant. Or maybe Deepmind will release this as a SaaS product?

bfirsh 9y ago

Here's an HTML version of the paper: https://www.arxiv-vanity.com/papers/1712.01815/ https://www.arxiv-vanity.com/papers/1712.01815/ Table 2 is broken, but the rest is much more readable if you're on a phone.

Scarblac 9y ago

As a chess player I find the win rate astonishing. Given the drawish tendency at top level, among human players, in correspondence chess and also in the TCEC final, I thought that even absolutely perfect play wouldn't score so well against a decent Stockfish setup (which 64 cores and 1 minute per move should be).

thomasahle 9y ago

Discussion at the Computer Chess Club (CCC) forum: http://www.talkchess.com/forum/viewtopic.php?topic_view=threads&p=741214&t=65910 http://www.talkchess.com/forum/viewtopic.php?topic_view=thre... and http://www.talkchess.com/forum/viewtopic.php?topic_view=threads&p=741211&t=65909 http://www.talkchess.com/forum/viewtopic.php?topic_view=thre...

hmate9 9y ago

So AlphaGo Zero used 4 TPUs while AlphaZero used 1500. It’s not immediately obvious to me why there is this massive difference. Can anyone elaborate?

sanxiyn 9y ago
Both used 4 TPUs at playing time. At training time, AlphaGo Zero used unspecified amount of computing resource, AlphaZero used 5000 TPUs for self-play.
- hmate9 9y ago
  Ah, thanks for clearing that up! Makes sense.

imrehg 9y ago

As a Shogi enthusiast (but complete beginner), I'd like to have seen more Shogi details in the article. Nevertheless there's plenty of other things to geek out on...

naveen99 9y ago

Is there an sdk or compiler for using the google tpu's beyond just using tensorflow ? Is the tpu backend of tensorflow based on cuda, opencl, plain c or something else ?

deleted 9y ago

[deleted]

k2xl 9y ago

Chess.com forum thread https://www.chess.com/forum/view/general/stockfish-dethroned https://www.chess.com/forum/view/general/stockfish-dethroned

auggierose 9y ago

Great result, but without access to source code this is not a scientific paper.

plg 9y ago

source code?

cdelsolar 9y ago

I wanted to contact the authors directly but can't seem to find contact info at the moment, with a question. I hope some of you might know enough to answer it. I'm interested in applying this method, or a similar neural-network / tabula rasa based method to the game of Scrabble. I read the original AlphaGo Zero paper and they mentioned that this method works best for games of perfect information. The standard Scrabble AI right now is quite good and can definitely beat top experts close to 50% of the time, but it uses simple Monte Carlo simulations to evaluate positions and just picks the ones that perform better. It doesn't quite account for defensive considerations or other subtleties of the game. I was wondering if anyone who had more insight into MCTS and NN would be able to talk me through how to apply this to Scrabble, or if it even makes sense. One of the issues I can see currently would be very slow convergence; as it has a luck factor, the algorithm could make occasional terrible moves and still win games, and thus be "wrongly trained".

bo1024 9y ago
Step 1: millions (?) of dollars of hardware.

SubiculumCode 9y ago

There is only one way for a human to win at chess against these computers; and it involves violence against the chess board.

wskish 9y ago

The more interesting metric going forward is performance at a given power budget (not unlike with motorsports). The TPUs are consuming sooo much power here! Most interesting real-world problems are power-limited, including in nature (e.g. metabolic limits).

orasis 9y ago
When a lot of money is on the line you can use a lot of resources.

skc 9y ago

I'm only a fairly pedestrian chess player, but I looked at one of these games between AGZ and SF and aside from the endgame, AGZ played in a manner that almost seemed alien. It seemed to completely ignore various little rules of thumb which is to be expected in hindsight but fairly mind-blowing when you actually watch a game.

TwoBit 9y ago

Wasn't Stockfish gimped for this competition? No openings, no endgame tables, low RAM, etc? If that's so then this AI did not in fact beat the computer chess champ.