8 ms·

Considering AlphaZero was orders of magnitude stronger than stock fish, it would be interesting to see AlphaZero's analysis if it was available. Edit: Alphazer

Considering AlphaZero was orders of magnitude stronger than stock fish, it would be interesting to see AlphaZero's analysis if it was available. Edit: Alphazero's record against Stock Fish is 28 wins, 72 ties and 0 losses. In a 100 game matchup, AlphaZero had 28 wins 72 draws and 0 losses. I'd classify that as orders of magnitude better. https://www.chess.com/news/view/google-s-alphazero-destroys-stockfish-in-100-game-match https://www.chess.com/news/view/google-s-alphazero-destroys-...

nullbyte 8y ago
It's not orders of magnitude stronger. But I agree, it's be interesting to see Alpha's analysis.
- liftbigweights 8y ago
  In a 100 game matchup, AlphaZero had 28 wins 72 draws and 0 losses. I'd classify that as orders of magnitude better. https://www.chess.com/news/view/google-s-alphazero-destroys-stockfish-in-100-game-match https://www.chess.com/news/view/google-s-alphazero-destroys-... But yeah, it would be interesting to see a side by side comparison of alphazero and stockfish's live analysis of these games.
  - TwoBit 8y ago
    Wasn't Stockfish crippled in that set of games? Such as no opening book, which would give a computer like AlphaZero an initial advantage? And a relatively weak CPU setup, compared to the massive computer AlphaZero used?
    - TomatoTomato 8y ago
      So, 70 million nps or about ~3x 2950x threadrippers [2] is relatively weak cpu compared to 80 thousand nps or approximately 2x 2080tis [3] for Alphazero? > AlphaZero compensates for the lower number of evaluations by using its deep neural network to focus much more selectively on the most promising variation [1] Even if you compare CPU to GPU by price, and not speed, it seems pretty even. It clearly has to sacrifice speed by making more intelligent pruning decisions than stockfish. 1: https://en.wikipedia.org/wiki/AlphaZero#AlphaZero_vs._Stockfish_and_elmo https://en.wikipedia.org/wiki/AlphaZero#AlphaZero_vs._Stockf... 2: https://sites.google.com/site/computerschess/stockfish9-benchmarks https://sites.google.com/site/computerschess/stockfish9-benc... 3: https://www.reddit.com/r/hardware/comments/9jyts8/rtx_2080_ti_deep_learning_benchmarks_lambda_labs/e6wsly1/ https://www.reddit.com/r/hardware/comments/9jyts8/rtx_2080_t...
      - dragontamer 8y ago
        AlphaGo engineers didn't really know how to configure Stockfish correctly. The computer itself was strong. But the weird timing setup, the disabled databases (ie: both opening book AND endgame book was disabled. Stockfish normally plays PERFECTLY when the board is reduced to 6 pieces or less, as well as perfectly knows the winner / loser in every 6-piece setup. But that was disabled for the AlphaGo games) Without the ability to run AlphaGo on our own and recreate the test, we have no way in knowing how AlphaGo would work under "fair" conditions. ------- And btw: 1GB of RAM is a lulzy setup. You put all the CPU time you want, but gimping the RAM down to 1GB is... weird. Its a VERY suspect "test" that none of us can replicate.
        ganeshkrishnan 8y ago
        It's not that they didn't know how to configure it correctly; some of it was purposefully (and rightfully) disabled to see real difference between stockfish and AZ. Remember that if stockfish uses tablebases and opening books it's really "cheating" by using human knowledge. Although I am surprised that they reduced the hashtable so much and also used stockfish 8 when stockfish 9 was available. It was not RAM but just the hashtable size also their "test" has been replicated by plenty of people. We have our own "replica" of the NN playing chess in lichess.
        TomatoTomato 8y ago
        Opening book I agree, but tablebases are definitely not human knowledge. They are a bruteforced database of every possible position, or in essence a precomputed position. Put simply, giving a massive time advantage to the user during the late middle and endgame. So if A0 wasn't built to access them, but stockfish did in the match, then that would be an unfair advantage.
        dragontamer 8y ago
        > So if A0 wasn't built to access them, but stockfish did in the match, then that would be an unfair advantage. Why is that an unfair advantage? AlphaZero developers wanted to prove that their AI was better than Stockfish. They failed to do that. If AlphaZero wasn't built to access Tablebases, then they should have built it to access tablebases. Don't unfairly gimp Stockfish because you're lazy at programming.
  - deepnotderp 8y ago
    Stockfish was severely crippled in that comparison, Stockfish had no opening book and only like a gig of RAM, as well as a very weak CPU and also it had a fixed amount of time per move, even though variable "thinking" compute time is a major advantage for stockfish
  - bonzini 8y ago
    White was winning twice as many points as black (64 points for white, 36 for black; draws give 0.5 points to both players) and points difference is all that matters for Elo. As a matter of fact, 64-36 is pretty much exactly what you'd expect when two players have 100 Elo difference. Your incorrect perception comes from the fact that, at very high levels (as shown by this match as well), draws become way more common than for lower levels. A 3400 vs 3300 Elo match might be 28-72-0, a 1400 vs 1300 match might be 60-8-32. At the lower level, you see one player winning twice as many games as the other, at the higher level you see one player losing all the time; that looks different to you, but as far as Elo difference is concerned the two results are exactly the same.
nurettin 8y ago
According to the ChessNetwork yt channel, Alpha zero does not consistently beat stockfish. But when it does, it does so in a very interesting manner.
- 6nf 8y ago
  AlphaZero draws a lot against Stockfish but Stockfish won ZERO games in their 100 game match up.
  - dragontamer 8y ago
    Stockfish wasn't really configured correctly: they disabled both the opening database and the endgame database in the tests. I dunno why AlphaGo devs would do that, aside from artificially gimping Stockfish on purpose.
    - nurettin 8y ago
      I can imagine them thinking "well, alphazero doesn't employ opening and endgame databases" to which I would counter-argue with "stockfish doesn't use machine learning, either"
thomasahle 8y ago
AlphaZero was playing Stockfish 8. This is done using Stockfish dev which is more than 110 ELO higher than Stockfish 8 (see https://github.com/glinscott/fishtest/wiki/Regression-Tests https://github.com/glinscott/fishtest/wiki/Regression-Tests) and higher than what was reported for AlphaZero. Check out the Leela project, which is trying to reproduce and improve the AlphaZero project to also beat Stockfish dev: http://lczero.org/ http://lczero.org/ (at this point they are about equal with Stockfish 9)
- antaviana 8y ago
  A 100 ELO difference is about a player twice as stronger. With a 100 ELO difference, Stockfish would have won some games. 28Win 72Draw and O Loss in my opinion reflects at least 300 ELO (8 times stronger).
  - ammon 8y ago
    Elo is not a matter of opinion. There's a formula for it, and 28 wins, 72 draws and 0 losses represents almost exactly a 100 point Elo difference. https://www.3dkingdoms.com/chess/elo.htm https://www.3dkingdoms.com/chess/elo.htm
    - antaviana 8y ago
      You are right, thank you for the pointer. I guess that the spectacular 28-0 was somehow confusing my ELO opinion engine by concealing the fact that 72 draws are a lot more difficult to achieve than 12 draws (28-12-0 would have been a 300 ELO difference).
- TomatoTomato 8y ago
  The A0 vs Stockfish 8 match also didn't use endgame tablebases.
thanatropism 8y ago
https://en.m.wikipedia.org/wiki/Order_of_magnitude https://en.m.wikipedia.org/wiki/Order_of_magnitude

nullbyte 8y ago

It's not orders of magnitude stronger. But I agree, it's be interesting to see Alpha's analysis.

liftbigweights 8y ago
In a 100 game matchup, AlphaZero had 28 wins 72 draws and 0 losses. I'd classify that as orders of magnitude better. https://www.chess.com/news/view/google-s-alphazero-destroys-stockfish-in-100-game-match https://www.chess.com/news/view/google-s-alphazero-destroys-... But yeah, it would be interesting to see a side by side comparison of alphazero and stockfish's live analysis of these games.
- TwoBit 8y ago
  Wasn't Stockfish crippled in that set of games? Such as no opening book, which would give a computer like AlphaZero an initial advantage? And a relatively weak CPU setup, compared to the massive computer AlphaZero used?
  - TomatoTomato 8y ago
    So, 70 million nps or about ~3x 2950x threadrippers [2] is relatively weak cpu compared to 80 thousand nps or approximately 2x 2080tis [3] for Alphazero? > AlphaZero compensates for the lower number of evaluations by using its deep neural network to focus much more selectively on the most promising variation [1] Even if you compare CPU to GPU by price, and not speed, it seems pretty even. It clearly has to sacrifice speed by making more intelligent pruning decisions than stockfish. 1: https://en.wikipedia.org/wiki/AlphaZero#AlphaZero_vs._Stockfish_and_elmo https://en.wikipedia.org/wiki/AlphaZero#AlphaZero_vs._Stockf... 2: https://sites.google.com/site/computerschess/stockfish9-benchmarks https://sites.google.com/site/computerschess/stockfish9-benc... 3: https://www.reddit.com/r/hardware/comments/9jyts8/rtx_2080_ti_deep_learning_benchmarks_lambda_labs/e6wsly1/ https://www.reddit.com/r/hardware/comments/9jyts8/rtx_2080_t...
    - dragontamer 8y ago
      AlphaGo engineers didn't really know how to configure Stockfish correctly. The computer itself was strong. But the weird timing setup, the disabled databases (ie: both opening book AND endgame book was disabled. Stockfish normally plays PERFECTLY when the board is reduced to 6 pieces or less, as well as perfectly knows the winner / loser in every 6-piece setup. But that was disabled for the AlphaGo games) Without the ability to run AlphaGo on our own and recreate the test, we have no way in knowing how AlphaGo would work under "fair" conditions. ------- And btw: 1GB of RAM is a lulzy setup. You put all the CPU time you want, but gimping the RAM down to 1GB is... weird. Its a VERY suspect "test" that none of us can replicate.
      - ganeshkrishnan 8y ago
        It's not that they didn't know how to configure it correctly; some of it was purposefully (and rightfully) disabled to see real difference between stockfish and AZ. Remember that if stockfish uses tablebases and opening books it's really "cheating" by using human knowledge. Although I am surprised that they reduced the hashtable so much and also used stockfish 8 when stockfish 9 was available. It was not RAM but just the hashtable size also their "test" has been replicated by plenty of people. We have our own "replica" of the NN playing chess in lichess.
        TomatoTomato 8y ago
        Opening book I agree, but tablebases are definitely not human knowledge. They are a bruteforced database of every possible position, or in essence a precomputed position. Put simply, giving a massive time advantage to the user during the late middle and endgame. So if A0 wasn't built to access them, but stockfish did in the match, then that would be an unfair advantage.
        dragontamer 8y ago
        > So if A0 wasn't built to access them, but stockfish did in the match, then that would be an unfair advantage. Why is that an unfair advantage? AlphaZero developers wanted to prove that their AI was better than Stockfish. They failed to do that. If AlphaZero wasn't built to access Tablebases, then they should have built it to access tablebases. Don't unfairly gimp Stockfish because you're lazy at programming.
- deepnotderp 8y ago
  Stockfish was severely crippled in that comparison, Stockfish had no opening book and only like a gig of RAM, as well as a very weak CPU and also it had a fixed amount of time per move, even though variable "thinking" compute time is a major advantage for stockfish
- bonzini 8y ago
  White was winning twice as many points as black (64 points for white, 36 for black; draws give 0.5 points to both players) and points difference is all that matters for Elo. As a matter of fact, 64-36 is pretty much exactly what you'd expect when two players have 100 Elo difference. Your incorrect perception comes from the fact that, at very high levels (as shown by this match as well), draws become way more common than for lower levels. A 3400 vs 3300 Elo match might be 28-72-0, a 1400 vs 1300 match might be 60-8-32. At the lower level, you see one player winning twice as many games as the other, at the higher level you see one player losing all the time; that looks different to you, but as far as Elo difference is concerned the two results are exactly the same.

nurettin 8y ago

According to the ChessNetwork yt channel, Alpha zero does not consistently beat stockfish. But when it does, it does so in a very interesting manner.

6nf 8y ago
AlphaZero draws a lot against Stockfish but Stockfish won ZERO games in their 100 game match up.
- dragontamer 8y ago
  Stockfish wasn't really configured correctly: they disabled both the opening database and the endgame database in the tests. I dunno why AlphaGo devs would do that, aside from artificially gimping Stockfish on purpose.
  - nurettin 8y ago
    I can imagine them thinking "well, alphazero doesn't employ opening and endgame databases" to which I would counter-argue with "stockfish doesn't use machine learning, either"

thomasahle 8y ago

AlphaZero was playing Stockfish 8. This is done using Stockfish dev which is more than 110 ELO higher than Stockfish 8 (see https://github.com/glinscott/fishtest/wiki/Regression-Tests https://github.com/glinscott/fishtest/wiki/Regression-Tests) and higher than what was reported for AlphaZero. Check out the Leela project, which is trying to reproduce and improve the AlphaZero project to also beat Stockfish dev: http://lczero.org/ http://lczero.org/ (at this point they are about equal with Stockfish 9)

antaviana 8y ago
A 100 ELO difference is about a player twice as stronger. With a 100 ELO difference, Stockfish would have won some games. 28Win 72Draw and O Loss in my opinion reflects at least 300 ELO (8 times stronger).
- ammon 8y ago
  Elo is not a matter of opinion. There's a formula for it, and 28 wins, 72 draws and 0 losses represents almost exactly a 100 point Elo difference. https://www.3dkingdoms.com/chess/elo.htm https://www.3dkingdoms.com/chess/elo.htm
  - antaviana 8y ago
    You are right, thank you for the pointer. I guess that the spectacular 28-0 was somehow confusing my ELO opinion engine by concealing the fact that 72 draws are a lot more difficult to achieve than 12 draws (28-12-0 would have been a 300 ELO difference).
TomatoTomato 8y ago
The A0 vs Stockfish 8 match also didn't use endgame tablebases.

thanatropism 8y ago

https://en.m.wikipedia.org/wiki/Order_of_magnitude https://en.m.wikipedia.org/wiki/Order_of_magnitude