72 ms·

(I'm not a scientist) There are two issues I see with closed source research software. It seems to me like someone outside the research group should review al

(I'm not a scientist) There are two issues I see with closed source research software. It seems to me like someone outside the research group should review all code run for a paper as part of peer review. If there are glaring off-by-one errors, race conditions, etc., then why should we trust the results produced by the software? There are a lot of things that should get caught in peer review that would only be red flags if someone experienced was able to look at the program source. Also, I've always been under the impression that science should be reproducible. If no one can replicate your results, then how can we trust your findings? While this is technically possible without access to source code, closing source code off from the community really makes reproducing results harder. I'm not sure that it would be fair for journals to require open source availability for publishing, but if they don't then we need a creative solution because these are real problems.

dalke 11y ago
The situations you pose have little to do with being open source. It's easily possible with non-open source software. Consider the X-PLOR program for crystallography refinement, where an academic/ research license was a few hundred dollars. That came with source code, plus the right to distribute patches and other modifications to anyone else who had an X-PLOR license, but not to everyone else. Consider the NAUTY program for graph isomorphism, from http://users.cecs.anu.edu.au/~bdm/nauty/ http://users.cecs.anu.edu.au/~bdm/nauty/ . The license says "Permission is hereby given for use and/or distribution with the exception of sale for profit or application with nontrivial military significance." Consider the many programs available for free and unrestricted download from university web sites which are "for academic use only", some of which require users to cite a given paper. (Eg, http://www.maths.lth.se/matematiklth/personal/sminchis/code/cpmc/index.html http://www.maths.lth.se/matematiklth/personal/sminchis/code/... is one I easily found with a web search which is available in source code and is "free of charge for non-commercial research and education purposes".) These are neither open source nor free, so are they "closed source research software"? If "closed source" means "not open source" - which is the usual view - then yes, the above programs are all closed source. Yet all of them are available for peer review, reproducibility studies, etc. that you want. While on the other hand, to get what you want requires principles different than what the Free Software Foundation considers to be one of the essential freedoms in programming - the freedom to sell software. So at the very least, "peer-verifiable" software, for lack of a better term, is not compatible with "free software." I personally think it's unwise to even use the terms "open source" and "closed source" in this discussion because of the confusion it adds. However, as most researchers come from academic or government labs with non-commercial funding sources, I can see why self-funded, for-profit scientific software may be overlooked.
- dikaiosune 11y ago
  You'll note that I'm not advocating for FOSS (and the licensing nightmares that can entail), just available source code for peer and community review. Perhaps I'm being too loose in my terminology, but in this context I used "closed source" to mean "source code unavailable to the community." In my view the attached license(s) matter a lot less than making it so that the science can be reviewed, reproduced, and improved upon. If the source code can be reviewed for free but not redistributed, reused or commercialized for free, then I don't see why that would hinder endeavors to review and reproduce the research. However having to go through byzantine and expensive processes to procure source code would be an impediment to those in academia who don't have funding to pay for licenses to source code just to review a paper, for example. Maybe I'm not reading closely enough, but that sounds a lot like what you're advocating in your original comment.
  - dalke 11y ago
    I do understand the desire for "available source code for peer and community review". What I don't understand is how to set up practical guidelines. For example, consider "reviewed for free but not redistributed". If I review the software, and find an error, what do I do? Should I publish a paper which demonstrates the difference between the original and corrected versions? If so, I need to include the fixed code, and perhaps also the original. But that's a redistribution. "would be an impediment to those in academia who don't have funding to pay for licenses to source code" As a minor point which is big in my mind - most academic groups have more funding to pay for licenses than I, a self-funded, for-profit researcher, have. "but that sounds a lot like what you're advocating in your original comment" I mentioned that, to explain the view of most people who want access the source code. I was not advocating it. My question was, is this requirement important enough that all of the source code must be made available at no cost? If so, it's in opposition to the FSF's four freedoms, which encourages people to sell free software, so there must be some other philosophical underpinning to justify the no-cost argument. What is that philosophy? It can't simply be "to verify" because there are some problems, like factoring RSA-360: 2186820202343172631466406372285792654649158564828384065217121866374227745448 7764963889680817334211643637752157994969516984539482486678141304751672197524 0052350576247238785129338002757406892629970748212734663781952170745916609168 9358372359962787832802257421757011302526265184263565623426823456522539874717 61591019113926725623095606566457918240614767013806590649 where it's trivial to verify the solution is correct without reproducing the calculations. And what counts as a "byzantine and expensive processes"? The process of reproducing one of the CERN papers, especially if I need to make my own accelerator, is non-trivial and expensive. Some molecular dynamics simulation software only runs on custom-made ASIC hardware, or uses $100K+ of CPU time. A software cost of $20K is only a small part of the overall cost in that case.
x0x0 11y ago
Peer review is nearly impossible for the interesting bits of code. You have the same challenges as peer review of the paper itself: there aren't many reviewers with the prerequisite technical knowledge. The majority of the time, the qualified scientific reviewers and software reviewers will not be the same people. Viz understanding CFD analytically does not mean you understand numerical algorithm implementation, or are an expert in numerical error analysis. Hell, look at the frequent evidence all over the net that the vast majority of developers don't understand ieee 754. Who pays the developers? If you aren't paying the developers much, what exactly are they going to catch in the 2-4 hours they may have to look at it? I wish I had a good solution; I think about the best we can do is community-pooled development efforts ala openfoam, bioconductor, numpy, sklearn.
coliveira 11y ago
Reproducibility in science doesn't mean "easy" reproducibility of experiments. Some experiments in science are famously known to be a one-off. Example: the first test of general relativity was during a solar eclipse, with pictures that were subject to poor lightning conditions. Of course, the reason why we know it was right is that many other tests have been made since then, even though the original one can never be replicated. Same reasoning goes for other areas. A scientific experiment with software doesn't need to be replicated with the same software to be relevant (although it would be nice to do be able to do that). As long as other people can come up with their own implementation and arrive to similar results, the researcher achieved his/her goal.
- netheril96 11y ago
  > Reproducibility in science doesn't mean "easy" reproducibility of experiments. Where it is easy to make it easy to replicate, it should be made easy. Releasing source code, even in unsanitized form, allows for much easier inspection and replication. The research world is also much different today than a century ago. Today the field is crowded with researchers who must publish or perish. The speed of papers, many of which are in fact nonsense, is churning out is unprecedented. As a result, a paper that can be verified but is hard to verified is often never properly verified at all.
- dagw 11y ago
  Reproducibility in science doesn't mean "easy" reproducibility of experiments. No, but making it easy as reasonably possible or at least not going out of your way to make it hard would be nice.

dalke 11y ago

The situations you pose have little to do with being open source. It's easily possible with non-open source software. Consider the X-PLOR program for crystallography refinement, where an academic/ research license was a few hundred dollars. That came with source code, plus the right to distribute patches and other modifications to anyone else who had an X-PLOR license, but not to everyone else. Consider the NAUTY program for graph isomorphism, from http://users.cecs.anu.edu.au/~bdm/nauty/ http://users.cecs.anu.edu.au/~bdm/nauty/ . The license says "Permission is hereby given for use and/or distribution with the exception of sale for profit or application with nontrivial military significance." Consider the many programs available for free and unrestricted download from university web sites which are "for academic use only", some of which require users to cite a given paper. (Eg, http://www.maths.lth.se/matematiklth/personal/sminchis/code/cpmc/index.html http://www.maths.lth.se/matematiklth/personal/sminchis/code/... is one I easily found with a web search which is available in source code and is "free of charge for non-commercial research and education purposes".) These are neither open source nor free, so are they "closed source research software"? If "closed source" means "not open source" - which is the usual view - then yes, the above programs are all closed source. Yet all of them are available for peer review, reproducibility studies, etc. that you want. While on the other hand, to get what you want requires principles different than what the Free Software Foundation considers to be one of the essential freedoms in programming - the freedom to sell software. So at the very least, "peer-verifiable" software, for lack of a better term, is not compatible with "free software." I personally think it's unwise to even use the terms "open source" and "closed source" in this discussion because of the confusion it adds. However, as most researchers come from academic or government labs with non-commercial funding sources, I can see why self-funded, for-profit scientific software may be overlooked.

dikaiosune 11y ago
You'll note that I'm not advocating for FOSS (and the licensing nightmares that can entail), just available source code for peer and community review. Perhaps I'm being too loose in my terminology, but in this context I used "closed source" to mean "source code unavailable to the community." In my view the attached license(s) matter a lot less than making it so that the science can be reviewed, reproduced, and improved upon. If the source code can be reviewed for free but not redistributed, reused or commercialized for free, then I don't see why that would hinder endeavors to review and reproduce the research. However having to go through byzantine and expensive processes to procure source code would be an impediment to those in academia who don't have funding to pay for licenses to source code just to review a paper, for example. Maybe I'm not reading closely enough, but that sounds a lot like what you're advocating in your original comment.
- dalke 11y ago
  I do understand the desire for "available source code for peer and community review". What I don't understand is how to set up practical guidelines. For example, consider "reviewed for free but not redistributed". If I review the software, and find an error, what do I do? Should I publish a paper which demonstrates the difference between the original and corrected versions? If so, I need to include the fixed code, and perhaps also the original. But that's a redistribution. "would be an impediment to those in academia who don't have funding to pay for licenses to source code" As a minor point which is big in my mind - most academic groups have more funding to pay for licenses than I, a self-funded, for-profit researcher, have. "but that sounds a lot like what you're advocating in your original comment" I mentioned that, to explain the view of most people who want access the source code. I was not advocating it. My question was, is this requirement important enough that all of the source code must be made available at no cost? If so, it's in opposition to the FSF's four freedoms, which encourages people to sell free software, so there must be some other philosophical underpinning to justify the no-cost argument. What is that philosophy? It can't simply be "to verify" because there are some problems, like factoring RSA-360: 2186820202343172631466406372285792654649158564828384065217121866374227745448 7764963889680817334211643637752157994969516984539482486678141304751672197524 0052350576247238785129338002757406892629970748212734663781952170745916609168 9358372359962787832802257421757011302526265184263565623426823456522539874717 61591019113926725623095606566457918240614767013806590649 where it's trivial to verify the solution is correct without reproducing the calculations. And what counts as a "byzantine and expensive processes"? The process of reproducing one of the CERN papers, especially if I need to make my own accelerator, is non-trivial and expensive. Some molecular dynamics simulation software only runs on custom-made ASIC hardware, or uses $100K+ of CPU time. A software cost of $20K is only a small part of the overall cost in that case.

x0x0 11y ago

Peer review is nearly impossible for the interesting bits of code. You have the same challenges as peer review of the paper itself: there aren't many reviewers with the prerequisite technical knowledge. The majority of the time, the qualified scientific reviewers and software reviewers will not be the same people. Viz understanding CFD analytically does not mean you understand numerical algorithm implementation, or are an expert in numerical error analysis. Hell, look at the frequent evidence all over the net that the vast majority of developers don't understand ieee 754. Who pays the developers? If you aren't paying the developers much, what exactly are they going to catch in the 2-4 hours they may have to look at it? I wish I had a good solution; I think about the best we can do is community-pooled development efforts ala openfoam, bioconductor, numpy, sklearn.

coliveira 11y ago

Reproducibility in science doesn't mean "easy" reproducibility of experiments. Some experiments in science are famously known to be a one-off. Example: the first test of general relativity was during a solar eclipse, with pictures that were subject to poor lightning conditions. Of course, the reason why we know it was right is that many other tests have been made since then, even though the original one can never be replicated. Same reasoning goes for other areas. A scientific experiment with software doesn't need to be replicated with the same software to be relevant (although it would be nice to do be able to do that). As long as other people can come up with their own implementation and arrive to similar results, the researcher achieved his/her goal.

netheril96 11y ago
> Reproducibility in science doesn't mean "easy" reproducibility of experiments. Where it is easy to make it easy to replicate, it should be made easy. Releasing source code, even in unsanitized form, allows for much easier inspection and replication. The research world is also much different today than a century ago. Today the field is crowded with researchers who must publish or perish. The speed of papers, many of which are in fact nonsense, is churning out is unprecedented. As a result, a paper that can be verified but is hard to verified is often never properly verified at all.
dagw 11y ago
Reproducibility in science doesn't mean "easy" reproducibility of experiments. No, but making it easy as reasonably possible or at least not going out of your way to make it hard would be nice.