3 ms·

So the argument is that running buggy code makes your replication tainted. So ideally experiments replicate everything, meaning part of replication is implement

by __s 8y ago

So the argument is that running buggy code makes your replication tainted. So ideally experiments replicate everything, meaning part of replication is implementing the code & collecting a dataset. The exact code & dataset aren't suppose to be required for the results: you should be able to replicate the results by substituting your own code & data. When you have a rat maze experiment, it isn't expected that they include a 3d printer blueprint of the maze & genomes of the rats involved Or, more succinctly, code & data are left as an exercise for the reader

porphyrogene 8y ago
It is common to publish the layouts and dimensions of mazes in such studies. The mazes[1] themselves often reflect the nature of the study (e.g. a simple fork to test desicion-making or a complex maze with dead ends to test navigation and/or memory.) 1. http://www.ratbehavior.org/RatsAndMazes.htm http://www.ratbehavior.org/RatsAndMazes.htm
djsumdog 8y ago
But if more journals archive artifices (code + data) and there was a bigger push to keep these around, when replication fails, they can at least go back to the exact code and say, "huh .. they implemented this differently. I think this is the problem with their algo" or "oh, we forgot to account for this." .. ideally after you've written your own, looking only at the methodology and without looking at the code for the original.
- secabeen 8y ago
  Having the journals archive artifices seems like a good idea, but the librarians hate it. The last thing they want to do is to give the for-profit journals another thing to monetize against the academy. What makes more sense is to have the universities and the library systems archive that stuff.
  - lumost 8y ago
    why not use a public artifact repository? e.g. github, bitbucket, or a new academic collaboration.
androidgirl 8y ago
I see. That does make sense, like making a MIT version of GPL reference code. Having more researchers avaliable to audit code seems like it would help prevent flaws from slipping through, too, to prevent false conclusions. Thank you for explaining a bit more.

by __s 8y ago

porphyrogene 8y ago

It is common to publish the layouts and dimensions of mazes in such studies. The mazes[1] themselves often reflect the nature of the study (e.g. a simple fork to test desicion-making or a complex maze with dead ends to test navigation and/or memory.) 1. http://www.ratbehavior.org/RatsAndMazes.htm http://www.ratbehavior.org/RatsAndMazes.htm

djsumdog 8y ago

But if more journals archive artifices (code + data) and there was a bigger push to keep these around, when replication fails, they can at least go back to the exact code and say, "huh .. they implemented this differently. I think this is the problem with their algo" or "oh, we forgot to account for this." .. ideally after you've written your own, looking only at the methodology and without looking at the code for the original.

secabeen 8y ago
Having the journals archive artifices seems like a good idea, but the librarians hate it. The last thing they want to do is to give the for-profit journals another thing to monetize against the academy. What makes more sense is to have the universities and the library systems archive that stuff.
- lumost 8y ago
  why not use a public artifact repository? e.g. github, bitbucket, or a new academic collaboration.

androidgirl 8y ago

I see. That does make sense, like making a MIT version of GPL reference code. Having more researchers avaliable to audit code seems like it would help prevent flaws from slipping through, too, to prevent false conclusions. Thank you for explaining a bit more.