4 ms·

I applaud the work that went into this! But at the same time I feel sad for the state of science where working code is still not considered part of the paper,

by vog 9y ago

I applaud the work that went into this! But at the same time I feel sad for the state of science where working code is still not considered part of the paper, so the code is almost never published, let alone published as Free Software for other scienists (and hobbyists) to tinker with it, to fix it, to improve it, and to adapt it. For example, why isn't it possible for everyone interested to let Leela Zero play against AlphaGo Zero, on their own hardware (or their university's hardware)?

pmontra 9y ago
Agreed, but the main goal of the average researcher is getting the paper published. The code is a byproduct and probably written not too well (they are pro of another profession) and maybe they don't know how to properly distribute it. Furthermore their publishing channel is made for paper / PDF, not source code. In this case the author is DeepMind/Google, so they know how to code and how to distribute it. However we need not only the code but also the network. Leela Zero is code but more importantly an effort to spread the effort of training the network among many people. Each one volunteers some hw (CPU/GPU) and contributes a part of the training runs. The README states that the estimated training duration on commodity hardware would be 1,700 years. We're not that patient :-)
- ForHackernews 9y ago
  Could you design a cryptocurrency that used neural network training as part of a proof-of-work function? Then you'd actually get something useful at the end instead of brute-force hash collisions.
  - zdkl 9y ago
    You'd need to find some task that is expensive to compute, but trivial to verify to "port" current PoW methods, but the hard part is finding a task that fills this condition and is useful to actually compute. In any case, you'd still be running a competition between nodes for fastest/best solution, with lots of duplicated effort and consumed resources. That is the key part of crypto consensus afaiu, and regardless of "backend task", there's always going to be waste in this architecture.
    - bufferoverflow 9y ago
      Which should be possible. For instance, your PoW could be "design a Go game bot neural network that's better than the system currently challenges you with". Hard to make, easy to verify.
      - vog 9y ago
        > easy to verify This may be a lot harder than you think. How do you decide that one AI plays better than another? By playing lots of games. These games must be random, i.e. not known beforehand, otherwise the new AI could cheat. But then, every cryptocurrency node would use a different set of game settings for validation. Which means that some (unlucky) nodes may reject the new AI while others accept it. The result are regular net splits, making it very hard to establish an keep the global consensus that is absolutely needed for a stable cryptocurrency system. The core issue here is that validation must be 100% deterministic. Each node must run exactly the same calculations for validation, otherwise the consensus network is doomed.
        setr 9y ago
        N seeds for the prng algorithm shipped with the program to be verified. Each seed provides for one game. N is sufficiently large to effectively remove "luck" for the skill comparison, and sufficiently large to make finding enough seeds allowing wins for a cheating weaker player more difficult than generating a better go player Im assuming all games played against the incumbent; seeds are basically proof of wins. I guess you could still cheat though by copying the incumbent, get a 50-50 ratio of games, and then just find enough winners to make it 49-51.
    - yorwba 9y ago
      The expensive task in machine learning is the training, it can be verified by checking for performance improvement on a validation set. You can make use of the duplicated effort if everyone trains on different examples, then their learned models can be merged/ensembled together for better performance. That's the ideal situation when everyone plays by the rules. To make that scheme work as a "crypto"currency, you'll also have to make sure that breaking the rules doesn't bring a benefit. E.g. the validation examples can't be known beforehand, otherwise someone could just overfit to them to get a higher score with less work. But I think it might be possible to make it work.
  - deleted 9y ago
    [deleted]
  - romaniv 9y ago
    I think this is a very interesting idea. Except I would probably use a random forest instead of ANN for something like this. It is trivially parallelizable, you can train on subsets of features, you can slit and merge your ensembles/forests and so on.
alexott 9y ago
it becoming better in the recent times with projects like http://www.gitxiv.com/ http://www.gitxiv.com/
dkersten 9y ago
> so the code is almost never published Even worse is that a friend of mine who is a researcher in neuroscience/BCI has found that many of the papers couldn't be reproduced at all with flawed or incorrect conclusions. Since they typically don't publish their code, its very difficult for others to check the correctness.
- vog 9y ago
  Moreover, in this context "code" is not just Python/C++/Java, but also spreadsheets! Publishing spreadsheets is perhaps an even more urgent issue than publishing other types of code. And of course the underlying data (anonymized) in case of statistical elicitation, but that's another topic. (Open Access, Open Data and Free Software are very closely related, though).
- raverbashing 9y ago
  Really, the academic world is sometimes obsessed with pseudocode. It might make some didactic sense, but it makes fewer and fewer sense as times go by when you can ship the actual code and shows actual results
- fjfaase 9y ago
  And even when it is made available, it is of very poor quality. I heard about someome who studied the programs used for analyzing DNA sequences. While tinkering with it, he found a bug, fixed it and submitted a pull request. Two years later, the pull request was still not merged by the maintainer of this 'standard' package. He also told that the software was of a poor quality. For example, in theory, four base pairs could be stored in one byte, instead they are simply storing the letters A,C,T, or G in a single byte for each pair.
  - dkersten 9y ago
    Yeah, that’s what my friend was saying too. He mentioned one often-cited paper which, when he fixed a bug he found in the code, no longer supported the papers conclusion at all. He also complained about the poor quality in a lot of the code. :-(
  - 9y ago

by vog 9y ago

pmontra 9y ago

Agreed, but the main goal of the average researcher is getting the paper published. The code is a byproduct and probably written not too well (they are pro of another profession) and maybe they don't know how to properly distribute it. Furthermore their publishing channel is made for paper / PDF, not source code. In this case the author is DeepMind/Google, so they know how to code and how to distribute it. However we need not only the code but also the network. Leela Zero is code but more importantly an effort to spread the effort of training the network among many people. Each one volunteers some hw (CPU/GPU) and contributes a part of the training runs. The README states that the estimated training duration on commodity hardware would be 1,700 years. We're not that patient :-)

ForHackernews 9y ago
Could you design a cryptocurrency that used neural network training as part of a proof-of-work function? Then you'd actually get something useful at the end instead of brute-force hash collisions.
- zdkl 9y ago
  You'd need to find some task that is expensive to compute, but trivial to verify to "port" current PoW methods, but the hard part is finding a task that fills this condition and is useful to actually compute. In any case, you'd still be running a competition between nodes for fastest/best solution, with lots of duplicated effort and consumed resources. That is the key part of crypto consensus afaiu, and regardless of "backend task", there's always going to be waste in this architecture.
  - bufferoverflow 9y ago
    Which should be possible. For instance, your PoW could be "design a Go game bot neural network that's better than the system currently challenges you with". Hard to make, easy to verify.
    - vog 9y ago
      > easy to verify This may be a lot harder than you think. How do you decide that one AI plays better than another? By playing lots of games. These games must be random, i.e. not known beforehand, otherwise the new AI could cheat. But then, every cryptocurrency node would use a different set of game settings for validation. Which means that some (unlucky) nodes may reject the new AI while others accept it. The result are regular net splits, making it very hard to establish an keep the global consensus that is absolutely needed for a stable cryptocurrency system. The core issue here is that validation must be 100% deterministic. Each node must run exactly the same calculations for validation, otherwise the consensus network is doomed.
      - setr 9y ago
        N seeds for the prng algorithm shipped with the program to be verified. Each seed provides for one game. N is sufficiently large to effectively remove "luck" for the skill comparison, and sufficiently large to make finding enough seeds allowing wins for a cheating weaker player more difficult than generating a better go player Im assuming all games played against the incumbent; seeds are basically proof of wins. I guess you could still cheat though by copying the incumbent, get a 50-50 ratio of games, and then just find enough winners to make it 49-51.
  - yorwba 9y ago
    The expensive task in machine learning is the training, it can be verified by checking for performance improvement on a validation set. You can make use of the duplicated effort if everyone trains on different examples, then their learned models can be merged/ensembled together for better performance. That's the ideal situation when everyone plays by the rules. To make that scheme work as a "crypto"currency, you'll also have to make sure that breaking the rules doesn't bring a benefit. E.g. the validation examples can't be known beforehand, otherwise someone could just overfit to them to get a higher score with less work. But I think it might be possible to make it work.
- deleted 9y ago
  [deleted]
- romaniv 9y ago
  I think this is a very interesting idea. Except I would probably use a random forest instead of ANN for something like this. It is trivially parallelizable, you can train on subsets of features, you can slit and merge your ensembles/forests and so on.

alexott 9y ago

it becoming better in the recent times with projects like http://www.gitxiv.com/ http://www.gitxiv.com/

dkersten 9y ago

> so the code is almost never published Even worse is that a friend of mine who is a researcher in neuroscience/BCI has found that many of the papers couldn't be reproduced at all with flawed or incorrect conclusions. Since they typically don't publish their code, its very difficult for others to check the correctness.

vog 9y ago
Moreover, in this context "code" is not just Python/C++/Java, but also spreadsheets! Publishing spreadsheets is perhaps an even more urgent issue than publishing other types of code. And of course the underlying data (anonymized) in case of statistical elicitation, but that's another topic. (Open Access, Open Data and Free Software are very closely related, though).
raverbashing 9y ago
Really, the academic world is sometimes obsessed with pseudocode. It might make some didactic sense, but it makes fewer and fewer sense as times go by when you can ship the actual code and shows actual results
fjfaase 9y ago
And even when it is made available, it is of very poor quality. I heard about someome who studied the programs used for analyzing DNA sequences. While tinkering with it, he found a bug, fixed it and submitted a pull request. Two years later, the pull request was still not merged by the maintainer of this 'standard' package. He also told that the software was of a poor quality. For example, in theory, four base pairs could be stored in one byte, instead they are simply storing the letters A,C,T, or G in a single byte for each pair.
- dkersten 9y ago
  Yeah, that’s what my friend was saying too. He mentioned one often-cited paper which, when he fixed a bug he found in the code, no longer supported the papers conclusion at all. He also complained about the poor quality in a lot of the code. :-(
- 9y ago