5 ms·

Serious question: how does one evaluate the results reproducibility of this paper ? Maybe I'm missing some things but: - Are 1st gen TPUs even accessible ? Yo

by Aissen 9y ago

Serious question: how does one evaluate the results reproducibility of this paper ? Maybe I'm missing some things but: - Are 1st gen TPUs even accessible ? You have to fill out a form to learn more about those second generation TPUs: https://cloud.google.com/tpu/ https://cloud.google.com/tpu/ - I can't find the source code This does not look like a scientific paper, but a (very impressive) tech demo.

gcp 9y ago
There are too many details missing for the results to be reproducible. Does it even qualify as a tech demo if the result only exists in DeepMind's lab?
- bawana 9y ago
  This raises an interesting concept. If you cannot reproduce an experiment because of lack of resources, can you believe it? Or is this the equivalent of 'photoshopping your results'? A similar problem exists in cosmology. Can you verify the multiverse model if you only have one universe to experiment in? As data storage requirements in RAM and TPU power requirements increase to run certain models/algorithms, machine learning is becoming more obscure. Not only can we not understand how an AI is reaching its conclusions (inscrutability), we cannot even probe it (by tweaking parameters, etc) to find weak points (inaccesibility). This is actually a good thing. Where humans cannot tread, there can be no evil?
  - gcp 9y ago
    At least in computer chess such experiments were typically demonstrated by winning the World Championship. (And sometimes they failed...cough Deep Blue cough)
317070 9y ago
How do you replicate CERN experiments? The LHC? Hubble? LIGO? LISA? At least this paper is reproducible by people who have the compute, and many universities have super computers. Even at home, you can verify the results by replaying the games against stockfish. You might not be able to replicate the setup at home, but that does not mean it is not science.
- Aissen 9y ago
  Comparing projects done in the open with multiple different universities, on public funds, with something done behind closed doors with only personnel from a commercial entity is pretty far-fetched.
  - 317070 9y ago
    Why? How are any of the factors you mention related to verifiability? How does being supported by public funds with academic personnel from multiple universities make LIGO any more verifiable for me at home? At least I can run these games against my stockfish, thus verifying the result. The method I cannot verify, but being able to verify the results is already more than most of science.
    - erk__ 9y ago
      You can maybe not do the experiments from Cern, but you can do the calculations most of the software they use for that is open source iirc.
dandermotj 9y ago
This is definitely a scientific paper. Pretty much no scientific paper comes with source code and the majority of scientific papers are not reproducible without an entire university department of resources anyway.
- Aissen 9y ago
  > Pretty much no scientific paper comes with source code Are we blindly accepting this as science now ?
  - voxl 9y ago
    Yes? Sorry you've been out of the loop so long but science doesn't cater to your idealistic ideas of what it ought to be.
- gcp 9y ago
  ...and this attitude causes quite a bit of them to not be reproducible even when people try with the same resources.
- tdb7893 9y ago
  My main thing about source code and scientific papers is that it would just be so easy to release the source code along with the paper. Even if people don't reproduce work source code would often help to understand it as often I'm a little unclear on implementation details, which source code would be able to greatly clarify.
hyperbovine 9y ago
Definitely a scientific paper. It’s obvious from they way they formatted it that they’re going to submit this to Nature or Science.
- sndean 9y ago
  This then may suggest that there’ll be this detail-light manuscript in the journal and a 50-100 page supplemental document available to download, with all the details to reproduce (hopefully)
forgot-my-pw 9y ago
There's a project currently that emulates AlphaGo Zero using crowdsourcing: https://github.com/gcp/leela-zero https://github.com/gcp/leela-zero . You can run it on the browser too and it will submit the games after: https://ntt123.github.io/leela-zero/ https://ntt123.github.io/leela-zero/ Hope such a chess project like this will be available in the future.

by Aissen 9y ago

gcp 9y ago

There are too many details missing for the results to be reproducible. Does it even qualify as a tech demo if the result only exists in DeepMind's lab?

bawana 9y ago
This raises an interesting concept. If you cannot reproduce an experiment because of lack of resources, can you believe it? Or is this the equivalent of 'photoshopping your results'? A similar problem exists in cosmology. Can you verify the multiverse model if you only have one universe to experiment in? As data storage requirements in RAM and TPU power requirements increase to run certain models/algorithms, machine learning is becoming more obscure. Not only can we not understand how an AI is reaching its conclusions (inscrutability), we cannot even probe it (by tweaking parameters, etc) to find weak points (inaccesibility). This is actually a good thing. Where humans cannot tread, there can be no evil?
- gcp 9y ago
  At least in computer chess such experiments were typically demonstrated by winning the World Championship. (And sometimes they failed...cough Deep Blue cough)

317070 9y ago

How do you replicate CERN experiments? The LHC? Hubble? LIGO? LISA? At least this paper is reproducible by people who have the compute, and many universities have super computers. Even at home, you can verify the results by replaying the games against stockfish. You might not be able to replicate the setup at home, but that does not mean it is not science.

Aissen 9y ago
Comparing projects done in the open with multiple different universities, on public funds, with something done behind closed doors with only personnel from a commercial entity is pretty far-fetched.
- 317070 9y ago
  Why? How are any of the factors you mention related to verifiability? How does being supported by public funds with academic personnel from multiple universities make LIGO any more verifiable for me at home? At least I can run these games against my stockfish, thus verifying the result. The method I cannot verify, but being able to verify the results is already more than most of science.
  - erk__ 9y ago
    You can maybe not do the experiments from Cern, but you can do the calculations most of the software they use for that is open source iirc.

dandermotj 9y ago

This is definitely a scientific paper. Pretty much no scientific paper comes with source code and the majority of scientific papers are not reproducible without an entire university department of resources anyway.

Aissen 9y ago
> Pretty much no scientific paper comes with source code Are we blindly accepting this as science now ?
- voxl 9y ago
  Yes? Sorry you've been out of the loop so long but science doesn't cater to your idealistic ideas of what it ought to be.
gcp 9y ago
...and this attitude causes quite a bit of them to not be reproducible even when people try with the same resources.
tdb7893 9y ago
My main thing about source code and scientific papers is that it would just be so easy to release the source code along with the paper. Even if people don't reproduce work source code would often help to understand it as often I'm a little unclear on implementation details, which source code would be able to greatly clarify.

hyperbovine 9y ago

Definitely a scientific paper. It’s obvious from they way they formatted it that they’re going to submit this to Nature or Science.

sndean 9y ago
This then may suggest that there’ll be this detail-light manuscript in the journal and a 50-100 page supplemental document available to download, with all the details to reproduce (hopefully)

forgot-my-pw 9y ago

There's a project currently that emulates AlphaGo Zero using crowdsourcing: https://github.com/gcp/leela-zero https://github.com/gcp/leela-zero . You can run it on the browser too and it will submit the games after: https://ntt123.github.io/leela-zero/ https://ntt123.github.io/leela-zero/ Hope such a chess project like this will be available in the future.