7 ms·

Props to DeepMind for an impressive result. And for fixing the unfair conditions from the previous match where they “crushed” a Stockfish that was severely hand

Props to DeepMind for an impressive result. And for fixing the unfair conditions from the previous match where they “crushed” a Stockfish that was severely handicapped due to time control and no opening books. Note that Stockfish score was significantly improved this time around. That said, can someone please explain what the logic is for DeepMind only releasing 20 games out of a 1000 game match? That is a relatively outstanding degree of cherry picking results. As a former chess prodigy and software engineer, I want to believe these results but they would be much more credible if DeepMind open sourced the games. I don’t see how that would expose any valuable proprietary IP, but I’m a total machine learning noob so maybe I’m missing something.

CJefferson 8y ago
I agree. I'm annoyed how companies like Google get to release selected parts of their work, keep the rest secret, yet get accepted by top Journals. I am positive (and have seen reviews saying such) that academics would never get away with this kind of filtering of results.
- deleted 8y ago
  [deleted]
- ACow_Adonis 8y ago
  I hate to break it to you, but academics certainly do get away with this practice. You simply don't have to report the 1000 results or hypothese that didn't generate positive findings: also see publication bias. Indeed not only is it present, but due to the various ways academia ranks citations and papers, it's fairer to say it's actually incentivised by academic culture.
- skybrian 8y ago
  I agree they should do it, but it looks like other researchers do sometimes get away without publishing their data? "Data sharing is a common requirement of funding or publication, though this obligation may come as a surprise to some authors—and to their colleagues who have had trouble acquiring data from other laboratories." https://www.nature.com/articles/nn0807-931 https://www.nature.com/articles/nn0807-931
- newen 8y ago
  Academics regularly get away with just posting results as graphs, without even the actual numbers in the graphs given out. And certainly not data, code, etc. I've done that for multiple papers in top journals in my field, and lots of others do the same. I give out the data and sometimes the code when people email me, but it's never required during the peer review process. Reviewers are just expected to believe the results in the submitted paper. I always find it funny when people hold so much faith in the peer review process, like the reviewers have some special knowledge or insight that let's them legitimize a paper. Majority of the time, the peer reviewers read the exact same thing everyone else reads, then provide suggestions for change, etc and eventually accept the paper for publication. But in terms of actual verification that the authors of the paper are not fabricating results, there is nothing remotely close to that.
dmurray 8y ago
There are at least 230 games released according to https://deepmind.com/research/alphago/alphazero-resources/ https://deepmind.com/research/alphago/alphazero-resources/ Agreed that it would be nice to get all 1000 (or maybe there are more? Since these seem to have come from two matches played under different conditions, and they mention pitting AZ against Stockfish in many other sets of conditions) and it really shouldn't compromise DeepMind's IP.
WestCoastJustin 8y ago
Maybe you could partially reverse the model if you had this data + timeline? I'm not sure though. Probably just didn't think anyone would want to dig through a 1000+ games.
- mindgam3 8y ago
  "Maybe you could partially reverse the model" Even with my rudimentary understanding of machine learning, I'm fairly certain that reverse engineering a model presumably based on millions of training games would be utterly impossible just by looking at a few thousand games. "Probably just didn't think anyone would want to dig through a 1000+ games." Actually, I think a lot of people would be interested: - chess players like myself - skeptics, academic researchers, and other "data scientists" looking for actual data to back up these impressive claims
usgroup 8y ago
Still think there’s a total hardware mismatch here. AlphaGo got 2 TPUs and 44 cores vs Stockfish on just 44 cores. AlphaGo is undoubtedly impressive but would love to see an FPGA version of Stockfish vs AlphaGo.
aerovistae 8y ago
Anyone else see the first sentence of this, think about passing props to Deepmind, and reflect that they've spent too much time writing JS lately?
Twirrim 8y ago
Not sure if this is in your wheel-house, but there's an open-source community driven re-implementation of AlphaZero, targeting Chess: https://github.com/LeelaChessZero/ https://github.com/LeelaChessZero/, http://lczero.org/ http://lczero.org/ It's reaching quite a competitive state, having trained to an ELO of ~3500 or so (based on evaluations against stockfish et al)
- mindgam3 8y ago
  Thanks, I’ve heard about LeelaChess a bit but haven’t gotten a chance to check it out in depth. It sounds like a really interesting project. I’m glad somebody is doing some open source work in this area, just wish Google would join the party a bit more in this area.

CJefferson 8y ago

I agree. I'm annoyed how companies like Google get to release selected parts of their work, keep the rest secret, yet get accepted by top Journals. I am positive (and have seen reviews saying such) that academics would never get away with this kind of filtering of results.

deleted 8y ago
[deleted]
ACow_Adonis 8y ago
I hate to break it to you, but academics certainly do get away with this practice. You simply don't have to report the 1000 results or hypothese that didn't generate positive findings: also see publication bias. Indeed not only is it present, but due to the various ways academia ranks citations and papers, it's fairer to say it's actually incentivised by academic culture.
skybrian 8y ago
I agree they should do it, but it looks like other researchers do sometimes get away without publishing their data? "Data sharing is a common requirement of funding or publication, though this obligation may come as a surprise to some authors—and to their colleagues who have had trouble acquiring data from other laboratories." https://www.nature.com/articles/nn0807-931 https://www.nature.com/articles/nn0807-931
newen 8y ago
Academics regularly get away with just posting results as graphs, without even the actual numbers in the graphs given out. And certainly not data, code, etc. I've done that for multiple papers in top journals in my field, and lots of others do the same. I give out the data and sometimes the code when people email me, but it's never required during the peer review process. Reviewers are just expected to believe the results in the submitted paper. I always find it funny when people hold so much faith in the peer review process, like the reviewers have some special knowledge or insight that let's them legitimize a paper. Majority of the time, the peer reviewers read the exact same thing everyone else reads, then provide suggestions for change, etc and eventually accept the paper for publication. But in terms of actual verification that the authors of the paper are not fabricating results, there is nothing remotely close to that.

dmurray 8y ago

There are at least 230 games released according to https://deepmind.com/research/alphago/alphazero-resources/ https://deepmind.com/research/alphago/alphazero-resources/ Agreed that it would be nice to get all 1000 (or maybe there are more? Since these seem to have come from two matches played under different conditions, and they mention pitting AZ against Stockfish in many other sets of conditions) and it really shouldn't compromise DeepMind's IP.

WestCoastJustin 8y ago

Maybe you could partially reverse the model if you had this data + timeline? I'm not sure though. Probably just didn't think anyone would want to dig through a 1000+ games.

mindgam3 8y ago
"Maybe you could partially reverse the model" Even with my rudimentary understanding of machine learning, I'm fairly certain that reverse engineering a model presumably based on millions of training games would be utterly impossible just by looking at a few thousand games. "Probably just didn't think anyone would want to dig through a 1000+ games." Actually, I think a lot of people would be interested: - chess players like myself - skeptics, academic researchers, and other "data scientists" looking for actual data to back up these impressive claims

usgroup 8y ago

Still think there’s a total hardware mismatch here. AlphaGo got 2 TPUs and 44 cores vs Stockfish on just 44 cores. AlphaGo is undoubtedly impressive but would love to see an FPGA version of Stockfish vs AlphaGo.

aerovistae 8y ago

Anyone else see the first sentence of this, think about passing props to Deepmind, and reflect that they've spent too much time writing JS lately?

Twirrim 8y ago

Not sure if this is in your wheel-house, but there's an open-source community driven re-implementation of AlphaZero, targeting Chess: https://github.com/LeelaChessZero/ https://github.com/LeelaChessZero/, http://lczero.org/ http://lczero.org/ It's reaching quite a competitive state, having trained to an ELO of ~3500 or so (based on evaluations against stockfish et al)

mindgam3 8y ago
Thanks, I’ve heard about LeelaChess a bit but haven’t gotten a chance to check it out in depth. It sounds like a really interesting project. I’m glad somebody is doing some open source work in this area, just wish Google would join the party a bit more in this area.