7 ms·

Why isn't more research and research materials open source in the AI world? I don't really understand. If one doesn't have your training dataset or your code,

Why isn't more research and research materials open source in the AI world? I don't really understand. If one doesn't have your training dataset or your code, how could they possibly replicate your results?

__s 8y ago
So the argument is that running buggy code makes your replication tainted. So ideally experiments replicate everything, meaning part of replication is implementing the code & collecting a dataset. The exact code & dataset aren't suppose to be required for the results: you should be able to replicate the results by substituting your own code & data. When you have a rat maze experiment, it isn't expected that they include a 3d printer blueprint of the maze & genomes of the rats involved Or, more succinctly, code & data are left as an exercise for the reader
- porphyrogene 8y ago
  It is common to publish the layouts and dimensions of mazes in such studies. The mazes[1] themselves often reflect the nature of the study (e.g. a simple fork to test desicion-making or a complex maze with dead ends to test navigation and/or memory.) 1. http://www.ratbehavior.org/RatsAndMazes.htm http://www.ratbehavior.org/RatsAndMazes.htm
- djsumdog 8y ago
  But if more journals archive artifices (code + data) and there was a bigger push to keep these around, when replication fails, they can at least go back to the exact code and say, "huh .. they implemented this differently. I think this is the problem with their algo" or "oh, we forgot to account for this." .. ideally after you've written your own, looking only at the methodology and without looking at the code for the original.
  - secabeen 8y ago
    Having the journals archive artifices seems like a good idea, but the librarians hate it. The last thing they want to do is to give the for-profit journals another thing to monetize against the academy. What makes more sense is to have the universities and the library systems archive that stuff.
    - lumost 8y ago
      why not use a public artifact repository? e.g. github, bitbucket, or a new academic collaboration.
- androidgirl 8y ago
  I see. That does make sense, like making a MIT version of GPL reference code. Having more researchers avaliable to audit code seems like it would help prevent flaws from slipping through, too, to prevent false conclusions. Thank you for explaining a bit more.
porphyrogene 8y ago
Research is often funded by firms that want the study to serve as a “scientific” basis for the efficacy of a product or service. Experiments that cannot be replicated are meaningless. That standard should not be compromised.
taeric 8y ago
Amusingly, this is often taking to the other extreme. Just being able to run the exact same code on the exact same data doesn't really tell me much about if it replicates anywhere else. That is, I agree with you that the code and the data should, ideally, be available. I lose confidence when people just rerun the same code on the same data. The slides a while back about why someone didn't like notebooks resonates well with me. Something like "Shift enter through the lesson. Is this learning?"
jackcosgrove 8y ago
Are the 30% of papers including training data actually hosting the data set somewhere for download, or are they referencing a public data set elsewhere? Both approaches are acceptable in my book. If a study uses a private data set, or one that the researcher controls and only gives out to approved partners, that study should be discounted. I understand corporate labs cannot give away their data in many cases, but corporate research carries less authority than academic research anyways. Academic research should always make the data set publicly available.
- 8note 8y ago
  I think if you're going to do a replication study, you should collect a different data set about the same subject, and use the same method, then see if you get the same results.
  - mamon 8y ago
    Yes, simply compiling the provided code and running it against provided data set does not really count as replication. Writing your own code and creating your own dataset is the simplest way to rule out the situations where there is something fishy in either the original code or data. And the paper itself should contain enough details to make it possible to recreate the experiment this way.
    - spc476 8y ago
      I think compiling the provided code and running it against the provided data set does do something---you know the code reports on the data. If you get a different result with the provided code and data, then there's something different with your environment vs. the original researcher, perhaps the rounding mode, or some assumption [1]. Once that's straightened out, then you can use new data and see if you can replicate the result with the provided code and new data. Think of the provided data as a sanity test. [1] I recently fixed a bug wherein I was inadvertently relying upon Linux-specific behavior that failed when tested under Solaris.
MAXPOOL 8y ago
If you have a good paper with important result, providing code and data is not necessary. Providing the code to replicate is good form. It shows good faith and confidence. Exact replication (exactly replicating the study) is just the starting point to check that the code works and no obvious mistakes were made. replication / reproducibility / hyperparameter sensitivity If the research yields something really important and the method is well documented, usually it can be easily checked without having the data and the code. Things like dropout, batch normalization, residual learning, .. work over multiple different datasets and hyperparameters. You can reproduce the results without faithfully replicating the experiment. If the claimed result vanishes unless you have the exact data, code or the hyperparameters, the research can't be said to be meaningfully reproducible in the scientific sense. Hyperparameter sensitivity is ML equivalent to P-Hacking.
- lumost 8y ago
  how many papers important results are simply bugs? Numerical code is already bug prone due to subtle and hard to test errors, research code that's not code reviewed or necessarily even tested for correctness can easily generate important results erroneously. It's also counter-productive not to publish the underlying source code for these papers, as it adds a barrier to other researchers applying the algorithm in new situations. I'd be interested in seeing if those 6% of papers which include the code get more citations than the population of papers which do not include code.
  - MAXPOOL 8y ago
    > important results are simply bugs Probably none. If the paper is important and collects citations, the algorithm is in use. Computer science != working code. Code is required when you produce something where the scientific importance is less clear. There is need to provide more evidence. Many papers are just "Hey I made some some tweaks and it works in this particular case." Those papers should have working code.
    - lumost 8y ago
      Most papers leverage a results table which compares the newly proposed approach with existing approaches. This section is baselines the results of the new approach with prior work and helps determine whether a new result is actually important or yet another way to achieve the same results as previous work. e.g. the tables on page 7 of this paper https://www.semanticscholar.org/paper/Automatic-Acquisition-of-Lexical-Formality-Brooke-Wang/823a397b29bf596e2734d3dff7ab5abec2f60ac9 https://www.semanticscholar.org/paper/Automatic-Acquisition-... These tables are generated using real implementations that may or may not be correct, and should be subject to review when the paper is published.
marcosdumay 8y ago
There is some famous citation from a physicist about people that used to replicate each other experiments before, but now they share their fortran models, so they can agree on all the bugs.

__s 8y ago

So the argument is that running buggy code makes your replication tainted. So ideally experiments replicate everything, meaning part of replication is implementing the code & collecting a dataset. The exact code & dataset aren't suppose to be required for the results: you should be able to replicate the results by substituting your own code & data. When you have a rat maze experiment, it isn't expected that they include a 3d printer blueprint of the maze & genomes of the rats involved Or, more succinctly, code & data are left as an exercise for the reader

porphyrogene 8y ago
It is common to publish the layouts and dimensions of mazes in such studies. The mazes[1] themselves often reflect the nature of the study (e.g. a simple fork to test desicion-making or a complex maze with dead ends to test navigation and/or memory.) 1. http://www.ratbehavior.org/RatsAndMazes.htm http://www.ratbehavior.org/RatsAndMazes.htm
djsumdog 8y ago
But if more journals archive artifices (code + data) and there was a bigger push to keep these around, when replication fails, they can at least go back to the exact code and say, "huh .. they implemented this differently. I think this is the problem with their algo" or "oh, we forgot to account for this." .. ideally after you've written your own, looking only at the methodology and without looking at the code for the original.
- secabeen 8y ago
  Having the journals archive artifices seems like a good idea, but the librarians hate it. The last thing they want to do is to give the for-profit journals another thing to monetize against the academy. What makes more sense is to have the universities and the library systems archive that stuff.
  - lumost 8y ago
    why not use a public artifact repository? e.g. github, bitbucket, or a new academic collaboration.
androidgirl 8y ago
I see. That does make sense, like making a MIT version of GPL reference code. Having more researchers avaliable to audit code seems like it would help prevent flaws from slipping through, too, to prevent false conclusions. Thank you for explaining a bit more.

porphyrogene 8y ago

Research is often funded by firms that want the study to serve as a “scientific” basis for the efficacy of a product or service. Experiments that cannot be replicated are meaningless. That standard should not be compromised.

taeric 8y ago

Amusingly, this is often taking to the other extreme. Just being able to run the exact same code on the exact same data doesn't really tell me much about if it replicates anywhere else. That is, I agree with you that the code and the data should, ideally, be available. I lose confidence when people just rerun the same code on the same data. The slides a while back about why someone didn't like notebooks resonates well with me. Something like "Shift enter through the lesson. Is this learning?"

jackcosgrove 8y ago

Are the 30% of papers including training data actually hosting the data set somewhere for download, or are they referencing a public data set elsewhere? Both approaches are acceptable in my book. If a study uses a private data set, or one that the researcher controls and only gives out to approved partners, that study should be discounted. I understand corporate labs cannot give away their data in many cases, but corporate research carries less authority than academic research anyways. Academic research should always make the data set publicly available.

8note 8y ago
I think if you're going to do a replication study, you should collect a different data set about the same subject, and use the same method, then see if you get the same results.
- mamon 8y ago
  Yes, simply compiling the provided code and running it against provided data set does not really count as replication. Writing your own code and creating your own dataset is the simplest way to rule out the situations where there is something fishy in either the original code or data. And the paper itself should contain enough details to make it possible to recreate the experiment this way.
  - spc476 8y ago
    I think compiling the provided code and running it against the provided data set does do something---you know the code reports on the data. If you get a different result with the provided code and data, then there's something different with your environment vs. the original researcher, perhaps the rounding mode, or some assumption [1]. Once that's straightened out, then you can use new data and see if you can replicate the result with the provided code and new data. Think of the provided data as a sanity test. [1] I recently fixed a bug wherein I was inadvertently relying upon Linux-specific behavior that failed when tested under Solaris.

MAXPOOL 8y ago

If you have a good paper with important result, providing code and data is not necessary. Providing the code to replicate is good form. It shows good faith and confidence. Exact replication (exactly replicating the study) is just the starting point to check that the code works and no obvious mistakes were made. replication / reproducibility / hyperparameter sensitivity If the research yields something really important and the method is well documented, usually it can be easily checked without having the data and the code. Things like dropout, batch normalization, residual learning, .. work over multiple different datasets and hyperparameters. You can reproduce the results without faithfully replicating the experiment. If the claimed result vanishes unless you have the exact data, code or the hyperparameters, the research can't be said to be meaningfully reproducible in the scientific sense. Hyperparameter sensitivity is ML equivalent to P-Hacking.

lumost 8y ago
how many papers important results are simply bugs? Numerical code is already bug prone due to subtle and hard to test errors, research code that's not code reviewed or necessarily even tested for correctness can easily generate important results erroneously. It's also counter-productive not to publish the underlying source code for these papers, as it adds a barrier to other researchers applying the algorithm in new situations. I'd be interested in seeing if those 6% of papers which include the code get more citations than the population of papers which do not include code.
- MAXPOOL 8y ago
  > important results are simply bugs Probably none. If the paper is important and collects citations, the algorithm is in use. Computer science != working code. Code is required when you produce something where the scientific importance is less clear. There is need to provide more evidence. Many papers are just "Hey I made some some tweaks and it works in this particular case." Those papers should have working code.
  - lumost 8y ago
    Most papers leverage a results table which compares the newly proposed approach with existing approaches. This section is baselines the results of the new approach with prior work and helps determine whether a new result is actually important or yet another way to achieve the same results as previous work. e.g. the tables on page 7 of this paper https://www.semanticscholar.org/paper/Automatic-Acquisition-of-Lexical-Formality-Brooke-Wang/823a397b29bf596e2734d3dff7ab5abec2f60ac9 https://www.semanticscholar.org/paper/Automatic-Acquisition-... These tables are generated using real implementations that may or may not be correct, and should be subject to review when the paper is published.

marcosdumay 8y ago

There is some famous citation from a physicist about people that used to replicate each other experiments before, but now they share their fortran models, so they can agree on all the bugs.