4 ms·
It's a shame they still played against 2016 Stockfish (Stockfish 8), when Stockfish 9 or Stockfish Dev were available (Stockfish 10 is out now, but only very re
by gamegoblin 8y ago
It's a shame they still played against 2016 Stockfish (Stockfish 8), when Stockfish 9 or Stockfish Dev were available (Stockfish 10 is out now, but only very recently, so I can understand why they didn't use it).
Their results show that they are only just barely stronger than Stockfish 8, but Stockfish 9 and 10 are stronger than 8 as well.
EDIT: Also meant to include a shout-out to http://www.lczero.org/ http://www.lczero.org/ which is an open source implementation of AlphaZero chess. Here is their forum post for this paper: https://groups.google.com/forum/#!topic/lczero/TfmaNHI99gk https://groups.google.com/forum/#!topic/lczero/TfmaNHI99gk
SECOND EDIT: I was wrong! They did play against a newer SF than 8, specifically, SF at this commit: https://github.com/official-stockfish/Stockfish/commit/b508f9561cc2302c129efe8d60f201ff03ee72c8 https://github.com/official-stockfish/Stockfish/commit/b508f... , which was about 2 weeks before SF 9 was released, so maybe it is close in strength to SF 9.
- DannyBee 8y agoThe paper was in peer review for a year, so it couldn't have compared against those. AFAICT based on dates, they chose nearly the latest version they could have.
- thom 8y agoI'll never forget the last round of games with everyone gushing over AlphaZero's wonder human-like play as it reached a drawn position only for Stockfish to blunder a whole queen with nobody remarking on it. I played along while watching the Danny King video: https://twitter.com/DanielKingChess/status/1070755986636488704 https://twitter.com/DanielKingChess/status/10707559866364887... Recent Stockfishes recommend many of the quiet, strategic moves that he seems particularly enamoured with.
- plopz 8y agoIt said it was winning games with 1:10-100 time control. Why would you say thats only just barely stronger?
- gamegoblin 8y agoThe Elo graph in Figure 1 of the paper suggests it is maybe 30 Elo higher (eyeballing the graph, as they don't provide exact figures).
- jeremysalwen 8y agoTime handicaps are supposed to correspond to handicaps in computational power. But here according to the paper AlphaZero is running on ~100 teraFLOPS, but the stockfish machine has about 1 teraFLOP (my best estimate based on https://www.microway.com/knowledge-center-articles/detailed-specifications-of-the-intel-xeon-e5-2600v4-broadwell-ep-processors/ https://www.microway.com/knowledge-center-articles/detailed-...). So 100:1 time control would be fair from a computational perspective (of course comparing the computational power used is more subtle than the rough calculation I did here).
- FartyMcFarter 8y agoAs far as I know, FLOPS are mostly meaningless for traditional chess engines. They don't really use floating point operations much or at all.
- jeremysalwen 8y agoI know, but it still gives you a rough measure of the amount of silicon used.
- FartyMcFarter 8y agoI don't think so. For example, GPUs have way more TFLOPS per area than CPUs.
- aurelwu 8y agoAlphaZero used a few ( I think 4 TPUs) on top of the same or roughly same CPU Count, which is not that outrageous. I think the most fair comparison would be to pick a configuration for AZ and Stockfish which uses roughly the same amount of power and is best suited for each one respectively.
- RivieraKid 8y agoPerformance against SF9 was only marginally worse than vs SF8.
- conistonwater 8y agoThe most likely reason why SF9 or SF10 wouldn't do much better is that the Elo model isn't applicable: imagine if it's not about two engines doing similar things at different levels of competence, it could also be that AlphaZero's evaluation function is sufficiently different that it can exploit mistakes in SF's evaluation function. Then Elo(SF10) = Elo(SF8) + 100 wouldn't matter so much, it would give you SF's "strength", but not probability of winning against AlphaZero.
- gamegoblin 8y agoI believe you are correct and the not-significantly-changing win rates vs SF8 and SF9 imply that.