3 ms·

Wasn't Stockfish crippled in that set of games? Such as no opening book, which would give a computer like AlphaZero an initial advantage? And a relatively weak

by TwoBit 8y ago

Wasn't Stockfish crippled in that set of games? Such as no opening book, which would give a computer like AlphaZero an initial advantage? And a relatively weak CPU setup, compared to the massive computer AlphaZero used?

TomatoTomato 8y ago
So, 70 million nps or about ~3x 2950x threadrippers [2] is relatively weak cpu compared to 80 thousand nps or approximately 2x 2080tis [3] for Alphazero? > AlphaZero compensates for the lower number of evaluations by using its deep neural network to focus much more selectively on the most promising variation [1] Even if you compare CPU to GPU by price, and not speed, it seems pretty even. It clearly has to sacrifice speed by making more intelligent pruning decisions than stockfish. 1: https://en.wikipedia.org/wiki/AlphaZero#AlphaZero_vs._Stockfish_and_elmo https://en.wikipedia.org/wiki/AlphaZero#AlphaZero_vs._Stockf... 2: https://sites.google.com/site/computerschess/stockfish9-benchmarks https://sites.google.com/site/computerschess/stockfish9-benc... 3: https://www.reddit.com/r/hardware/comments/9jyts8/rtx_2080_ti_deep_learning_benchmarks_lambda_labs/e6wsly1/ https://www.reddit.com/r/hardware/comments/9jyts8/rtx_2080_t...
- dragontamer 8y ago
  AlphaGo engineers didn't really know how to configure Stockfish correctly. The computer itself was strong. But the weird timing setup, the disabled databases (ie: both opening book AND endgame book was disabled. Stockfish normally plays PERFECTLY when the board is reduced to 6 pieces or less, as well as perfectly knows the winner / loser in every 6-piece setup. But that was disabled for the AlphaGo games) Without the ability to run AlphaGo on our own and recreate the test, we have no way in knowing how AlphaGo would work under "fair" conditions. ------- And btw: 1GB of RAM is a lulzy setup. You put all the CPU time you want, but gimping the RAM down to 1GB is... weird. Its a VERY suspect "test" that none of us can replicate.
  - ganeshkrishnan 8y ago
    It's not that they didn't know how to configure it correctly; some of it was purposefully (and rightfully) disabled to see real difference between stockfish and AZ. Remember that if stockfish uses tablebases and opening books it's really "cheating" by using human knowledge. Although I am surprised that they reduced the hashtable so much and also used stockfish 8 when stockfish 9 was available. It was not RAM but just the hashtable size also their "test" has been replicated by plenty of people. We have our own "replica" of the NN playing chess in lichess.
    - TomatoTomato 8y ago
      Opening book I agree, but tablebases are definitely not human knowledge. They are a bruteforced database of every possible position, or in essence a precomputed position. Put simply, giving a massive time advantage to the user during the late middle and endgame. So if A0 wasn't built to access them, but stockfish did in the match, then that would be an unfair advantage.
      - dragontamer 8y ago
        > So if A0 wasn't built to access them, but stockfish did in the match, then that would be an unfair advantage. Why is that an unfair advantage? AlphaZero developers wanted to prove that their AI was better than Stockfish. They failed to do that. If AlphaZero wasn't built to access Tablebases, then they should have built it to access tablebases. Don't unfairly gimp Stockfish because you're lazy at programming.
        ganeshkrishnan 8y ago
        Alpha zero can access the tablebase. This proof of concept was to let both the engines play with minimal interference of outside pre computed knowledge.
    - zzo38computer 8y ago
      The programs are different programs. You can try it both ways, with and without the opening books. Because there are different configurations, in order to see who win must be specified by the configuration in use. One way is to do according to the most capable configurations of each program, so that means Stockfish with all of them. Other way is to specify the same limits of time, memory, etc of each one, although that is still not the excuse to avoid the opening books and tablebases and so on unless they use up more memory than AlphaZero does.

by TwoBit 8y ago

TomatoTomato 8y ago

So, 70 million nps or about ~3x 2950x threadrippers [2] is relatively weak cpu compared to 80 thousand nps or approximately 2x 2080tis [3] for Alphazero? > AlphaZero compensates for the lower number of evaluations by using its deep neural network to focus much more selectively on the most promising variation [1] Even if you compare CPU to GPU by price, and not speed, it seems pretty even. It clearly has to sacrifice speed by making more intelligent pruning decisions than stockfish. 1: https://en.wikipedia.org/wiki/AlphaZero#AlphaZero_vs._Stockfish_and_elmo https://en.wikipedia.org/wiki/AlphaZero#AlphaZero_vs._Stockf... 2: https://sites.google.com/site/computerschess/stockfish9-benchmarks https://sites.google.com/site/computerschess/stockfish9-benc... 3: https://www.reddit.com/r/hardware/comments/9jyts8/rtx_2080_ti_deep_learning_benchmarks_lambda_labs/e6wsly1/ https://www.reddit.com/r/hardware/comments/9jyts8/rtx_2080_t...

dragontamer 8y ago
AlphaGo engineers didn't really know how to configure Stockfish correctly. The computer itself was strong. But the weird timing setup, the disabled databases (ie: both opening book AND endgame book was disabled. Stockfish normally plays PERFECTLY when the board is reduced to 6 pieces or less, as well as perfectly knows the winner / loser in every 6-piece setup. But that was disabled for the AlphaGo games) Without the ability to run AlphaGo on our own and recreate the test, we have no way in knowing how AlphaGo would work under "fair" conditions. ------- And btw: 1GB of RAM is a lulzy setup. You put all the CPU time you want, but gimping the RAM down to 1GB is... weird. Its a VERY suspect "test" that none of us can replicate.
- ganeshkrishnan 8y ago
  It's not that they didn't know how to configure it correctly; some of it was purposefully (and rightfully) disabled to see real difference between stockfish and AZ. Remember that if stockfish uses tablebases and opening books it's really "cheating" by using human knowledge. Although I am surprised that they reduced the hashtable so much and also used stockfish 8 when stockfish 9 was available. It was not RAM but just the hashtable size also their "test" has been replicated by plenty of people. We have our own "replica" of the NN playing chess in lichess.
  - TomatoTomato 8y ago
    Opening book I agree, but tablebases are definitely not human knowledge. They are a bruteforced database of every possible position, or in essence a precomputed position. Put simply, giving a massive time advantage to the user during the late middle and endgame. So if A0 wasn't built to access them, but stockfish did in the match, then that would be an unfair advantage.
    - dragontamer 8y ago
      > So if A0 wasn't built to access them, but stockfish did in the match, then that would be an unfair advantage. Why is that an unfair advantage? AlphaZero developers wanted to prove that their AI was better than Stockfish. They failed to do that. If AlphaZero wasn't built to access Tablebases, then they should have built it to access tablebases. Don't unfairly gimp Stockfish because you're lazy at programming.
      - ganeshkrishnan 8y ago
        Alpha zero can access the tablebase. This proof of concept was to let both the engines play with minimal interference of outside pre computed knowledge.
  - zzo38computer 8y ago
    The programs are different programs. You can try it both ways, with and without the opening books. Because there are different configurations, in order to see who win must be specified by the configuration in use. One way is to do according to the most capable configurations of each program, so that means Stockfish with all of them. Other way is to specify the same limits of time, memory, etc of each one, although that is still not the excuse to avoid the opening books and tablebases and so on unless they use up more memory than AlphaZero does.