8 ms·

Personally I believe that open code, raw data, and cleaned datasets should be requirements for publication in any peer-reviewed journal in which a paper claims

by s_q_b 11y ago

Personally I believe that open code, raw data, and cleaned datasets should be requirements for publication in any peer-reviewed journal in which a paper claims that computational mechanisms were involved. Failure to distribute the data and code not only greatly devalues the contributions of many scientists, it makes replication far more difficult, and opens the door for outright fabrication. If computation is a necessary element of the research, it ought to be a necessary element of publication as well.

gww 11y ago
I could not agree with you more. It is a disconcerting trend to see high impact papers using an "in house program" or "a modified version of a previously described tool", without providing any details about software versions, analysis steps and any custom filtering applied to the data. There could be bugs, unintentionally (or intentionally) introduced biases, or other things such as outlier removal that are not explained in the manuscripts methods. This is also very true for statistical analysis, many papers do not provide R scripts for their analysis and only a brief overview of the analysis.
noname123 11y ago
>Failure to distribute the data and code not only greatly devalues the contributions of many scientists, it makes replication far more difficult, and opens the door for outright fabrication. Reasons why labs don't produce code: (1) Fear of being scooped; suppose you put in tons of man-hours into a custom population genetics association study for malaria in Africa, you don't want your competitor lab to sequence a bunch samples in SE Asia and run your code as-is and publish a paper when you have the sequencers to do the same. (2) Fear of competitors not being able to replicate stochastic ML results; some machine-learning and artificial neural network papers applied to Biology are stochastic. Past authors have been accused of "cherry-picking" the most "optimistic" runs that show positive results, e.g., (https://liorpachter.wordpress.com/2014/02/11/the-network-nonsense-of-manolis-kellis/ https://liorpachter.wordpress.com/2014/02/11/the-network-non...) (3) Focus on science, not tool production; most labs' focus is to produce publications, not open-source software. User base for scientific software is very niche, unlike Web MVC frameworks; making the payoff calculus of packaging and supporting external users not so great. Furthermore, most labs have custom databases making integrating external software, externalizing internal software difficult.
- pvnick 11y ago
  Just to follow up on your last point, I have found that folks are often happy to send code, but code written in research labs is generally horrendous. I don't think that the "publish or die" culture and a culture of releasing all relevant code is compatible. It takes time to refactor and produce clean code - time that can be spent chasing the next publication.
  - kwhitefoot 11y ago
    Then requiring the code to be published and runnable would have a salutary effect on the whole circus.
    - SapphireSun 11y ago
      Not really... I don't see how that would be anything more than adding one more task to a million other tasks. It doesn't change incentives.
  - IanCal 11y ago
    I don't see the need for clean code, I just want the code they used. I've repeatedly tried to find the code used to generate various visualisations (which were the entire point of the paper) and then found 1. "It'll be released soon" on a years old page 2. 404'ing personal pages on university sites 3. Finally some code But then the code didn't, and couldn't, generate the images they have. So now I'm not actually sure if the algorithm works as they say. This was also one of the more successful results I've had in trying to find research code outside of ML. I'm sure other disciplines are also good but ML is mostly what I've had to find. There's a part of the experiment which virtually anyone could replicate for a material cost of near 0 and check in detail if they wanted to and yet that seems to be the most poorly explained and shared part of it. You wouldn't get away with just mentioning the protocol you used in the lab without any references or explanations, would you?
- ppod 11y ago
  (1) You publish the code at the time of publication of the paper, not before. (2) This is a reason to publish the code, not a reason not to publish it. (3) You don't necessarily have to make package it neatly like production software, but some effort is necessary. We just wouldn't accept these kind of excuses for other descriptions of methodology, even though they might apply. There's no reason that code should be an exception.
tdaltonc 11y ago
It would have the side effect of improving and propagating best practices. Most scientists are self taught programers, and it shows. The data models and codebases used by most labs are . . . amateurish.
- dikaiosune 11y ago
  I really like what this group is doing: http://software-carpentry.org/ http://software-carpentry.org/ I took their instructor training last summer but I haven't had a chance to run a workshop yet. I think it's a great idea, especially given my recent exposure to researcher-written codebases.
- Thriptic 11y ago
  Seriously, I consider myself lucky when I get code that's even been commented from labmates...It took me about a year to get 50% of people using github
  - collyw 11y ago
    One ex-colleague of mine wouldn't even indent her Perl code properly. I showed here there was even a button in eclipse to do it for her. (Using eclipse seems problematic when there is a text editor that doesn't involve any cognitive overhead to many bioinformaticians. The step through debugger did seem to appeal to this one though).
dalke 11y ago
Could you explain what you mean by "open code"? What is the philosophical underpinning? That is, it seems that most of the argument for "open code" in the scientific review sense implies that 1) the software should be available at no cost, and 2) there is no need to support commercialization of said software. These principles are different than the four freedoms of "free software", and I have difficulties in reconciling the differences. For a concrete example, I am self-employed. I sell scientific software. All of my customers receive it under the BSD license, after they pay me a good chunk of money. Thus, I sell free software for scientific research. The FSF says "Selling a copy of a free program is legitimate, and we encourage it. (Quoting https://www.gnu.org/philosophy/selling.html https://www.gnu.org/philosophy/selling.html .) But most of the time when I see people say "open code" or something similar, they want the right to view, use, test, and modify the source code at no cost. If I publish a peer-reviewed paper about the software, should I be required to distribute the software to readers for no cost? Or may I set a fee of, say, $25,000 to get access to the source code under a free license? (I'm well aware of the loophole where I could publish that something is "open", but require a payment of $1 billion. My question is, what is a reasonable and fair price to charge?) On the flip side, if I am required to publish my software under a free/open source license and not charge for access to the code, then that means any reader can take my source code and commercialize it, or even simply give it away. Commercialization is part of the four freedoms of free software, but it ends up reducing my market size and knocking other parts of the four freedoms. I won't have as much money to continue my self-funded development and research. The principles are important because they help set guidelines for other questions. Can I require that people register their use of the software before getting access to a no-cost copy? Can I wait 6 months to respond to those requests? How long am I required to host the software, or will the journal manage all of that? May I include non-free license terms, like a requirement to cite X if someone uses the software? Does minimized/obsfucated code suffice? And many more details that have been resolved in the context of F/OSS software but have not, I believe, been resolve in the context of what's needed for peer reviewed publications.
- dikaiosune 11y ago
  (I'm not a scientist) There are two issues I see with closed source research software. It seems to me like someone outside the research group should review all code run for a paper as part of peer review. If there are glaring off-by-one errors, race conditions, etc., then why should we trust the results produced by the software? There are a lot of things that should get caught in peer review that would only be red flags if someone experienced was able to look at the program source. Also, I've always been under the impression that science should be reproducible. If no one can replicate your results, then how can we trust your findings? While this is technically possible without access to source code, closing source code off from the community really makes reproducing results harder. I'm not sure that it would be fair for journals to require open source availability for publishing, but if they don't then we need a creative solution because these are real problems.
bachmeier 11y ago
> If computation is a necessary element of the research, it ought to be a necessary element of publication as well. Correct. Imagine reading an article in a journal that said, "We were able to prove that Theorem 3.1 is true. The proof is omitted because we want to use similar techniques to prove theorems in the future, and by not publishing any details of our proof, we will have an advantage over rival researchers. That will allow us to recover the significant investment of time that we put into doing this proof."
- zevets 11y ago
  I can easily imagine such an article, because it happens all the damn time. "Using the argument of [1] and [2] it can be shown, with brief step x, it can be shown that this theorem holds in xxxx" and then moving onto the conclusions, is exactly the same as "Using the software developed in [3] to solve Eq. 1, we show that our result is statistically significant." If the claim is probable, it will and does get through review.
shepardrtc 11y ago
Especially if the work was funded by public money. What should be done in the case of when a researcher creates a private company as a result of publicly-funded research, but the research isn't fully released or is obfuscated?
digikata 11y ago
I lean towards more open code and data, but could one argue that if the code uses some fundamentally new, non-trivial methodology, a replication of the research should write new code implementing that method from it's description. Otherwise, if the original code is re-used, a bug in the code may very well cause the replication of an erroneous result.
- bachmeier 11y ago
  There are two reasons to look at the code. The first is to look for bugs. Obviously that is easier if you have the code. Testing is much easier than writing from scratch. The second is to see how robust the results are to changes in assumptions and other aspects of the methodology. When you are talking about a big computational problem, it is not realistic to expect others to do a full, independent replication. And if you can't reproduce the original results, you need access to the original code anyway, to find the source of the differences.
- coliveira 11y ago
  In fact this is the main reason why I don't subscribe to the idea of research papers including code (as a requirement). Independent replication should rely on independent implementation. Unless the software is too generic, the original researcher should have the advantage of using the implemented software for new discoveries, as much as a traditional scientist has the advantage of using his own lab for additional research.
  - irixusr 11y ago
    Be able to reimplement the code < That's an interesting perspective. However I think the vast majority of simulation work never gets that sort of scrutiny, no matter if the code is open source Personally though I think we should give up on papers and everything should be published on an ipython notebook style page where others can play with data and code.
- dikaiosune 11y ago
  Conversely, if the original code isn't available, the community may never know why they can't replicate the results. This could especially be a problem if the original researcher is more known and respected than those attempting replications (i.e. devolving into he said she said).
cafebeen 11y ago
I think the article is talking about a bigger issue, which is expanding our idea of a "research contribution" to include software. The current attitude essentially considers software to be equipment for getting to a scientific result. I think that if we started thinking about research software as a research contribution in itself, it would be a good way to accomplish what you talk about, e.g. by making software (and data) something that is published and cited, rather than attached as a supplement to every publication.
stared 11y ago
I concur. Once I wrote a similar answer http://academia.stackexchange.com/questions/23237/why-are-papers-without-code-but-with-results-accepted/23238#23238 http://academia.stackexchange.com/questions/23237/why-are-pa... In short: benefits for scientists (career, prestige) are misaligned with benefits for science and society (reproducibility, progress, openness).
jpolitz 11y ago
There's progress on this front in computer science conferences in software-focused disciplines: http://www.artifact-eval.org/ http://www.artifact-eval.org/ https://plus.google.com/+JanVitek/posts/19BH96G8rrw https://plus.google.com/+JanVitek/posts/19BH96G8rrw The artifact evaluation workflow is a reasonable starting point for what you suggest.

by s_q_b 11y ago

gww 11y ago

I could not agree with you more. It is a disconcerting trend to see high impact papers using an "in house program" or "a modified version of a previously described tool", without providing any details about software versions, analysis steps and any custom filtering applied to the data. There could be bugs, unintentionally (or intentionally) introduced biases, or other things such as outlier removal that are not explained in the manuscripts methods. This is also very true for statistical analysis, many papers do not provide R scripts for their analysis and only a brief overview of the analysis.

noname123 11y ago

>Failure to distribute the data and code not only greatly devalues the contributions of many scientists, it makes replication far more difficult, and opens the door for outright fabrication. Reasons why labs don't produce code: (1) Fear of being scooped; suppose you put in tons of man-hours into a custom population genetics association study for malaria in Africa, you don't want your competitor lab to sequence a bunch samples in SE Asia and run your code as-is and publish a paper when you have the sequencers to do the same. (2) Fear of competitors not being able to replicate stochastic ML results; some machine-learning and artificial neural network papers applied to Biology are stochastic. Past authors have been accused of "cherry-picking" the most "optimistic" runs that show positive results, e.g., (https://liorpachter.wordpress.com/2014/02/11/the-network-nonsense-of-manolis-kellis/ https://liorpachter.wordpress.com/2014/02/11/the-network-non...) (3) Focus on science, not tool production; most labs' focus is to produce publications, not open-source software. User base for scientific software is very niche, unlike Web MVC frameworks; making the payoff calculus of packaging and supporting external users not so great. Furthermore, most labs have custom databases making integrating external software, externalizing internal software difficult.

pvnick 11y ago
Just to follow up on your last point, I have found that folks are often happy to send code, but code written in research labs is generally horrendous. I don't think that the "publish or die" culture and a culture of releasing all relevant code is compatible. It takes time to refactor and produce clean code - time that can be spent chasing the next publication.
- kwhitefoot 11y ago
  Then requiring the code to be published and runnable would have a salutary effect on the whole circus.
  - SapphireSun 11y ago
    Not really... I don't see how that would be anything more than adding one more task to a million other tasks. It doesn't change incentives.
- IanCal 11y ago
  I don't see the need for clean code, I just want the code they used. I've repeatedly tried to find the code used to generate various visualisations (which were the entire point of the paper) and then found 1. "It'll be released soon" on a years old page 2. 404'ing personal pages on university sites 3. Finally some code But then the code didn't, and couldn't, generate the images they have. So now I'm not actually sure if the algorithm works as they say. This was also one of the more successful results I've had in trying to find research code outside of ML. I'm sure other disciplines are also good but ML is mostly what I've had to find. There's a part of the experiment which virtually anyone could replicate for a material cost of near 0 and check in detail if they wanted to and yet that seems to be the most poorly explained and shared part of it. You wouldn't get away with just mentioning the protocol you used in the lab without any references or explanations, would you?
ppod 11y ago
(1) You publish the code at the time of publication of the paper, not before. (2) This is a reason to publish the code, not a reason not to publish it. (3) You don't necessarily have to make package it neatly like production software, but some effort is necessary. We just wouldn't accept these kind of excuses for other descriptions of methodology, even though they might apply. There's no reason that code should be an exception.

tdaltonc 11y ago

It would have the side effect of improving and propagating best practices. Most scientists are self taught programers, and it shows. The data models and codebases used by most labs are . . . amateurish.

dikaiosune 11y ago
I really like what this group is doing: http://software-carpentry.org/ http://software-carpentry.org/ I took their instructor training last summer but I haven't had a chance to run a workshop yet. I think it's a great idea, especially given my recent exposure to researcher-written codebases.
Thriptic 11y ago
Seriously, I consider myself lucky when I get code that's even been commented from labmates...It took me about a year to get 50% of people using github
- collyw 11y ago
  One ex-colleague of mine wouldn't even indent her Perl code properly. I showed here there was even a button in eclipse to do it for her. (Using eclipse seems problematic when there is a text editor that doesn't involve any cognitive overhead to many bioinformaticians. The step through debugger did seem to appeal to this one though).

dalke 11y ago

Could you explain what you mean by "open code"? What is the philosophical underpinning? That is, it seems that most of the argument for "open code" in the scientific review sense implies that 1) the software should be available at no cost, and 2) there is no need to support commercialization of said software. These principles are different than the four freedoms of "free software", and I have difficulties in reconciling the differences. For a concrete example, I am self-employed. I sell scientific software. All of my customers receive it under the BSD license, after they pay me a good chunk of money. Thus, I sell free software for scientific research. The FSF says "Selling a copy of a free program is legitimate, and we encourage it. (Quoting https://www.gnu.org/philosophy/selling.html https://www.gnu.org/philosophy/selling.html .) But most of the time when I see people say "open code" or something similar, they want the right to view, use, test, and modify the source code at no cost. If I publish a peer-reviewed paper about the software, should I be required to distribute the software to readers for no cost? Or may I set a fee of, say, $25,000 to get access to the source code under a free license? (I'm well aware of the loophole where I could publish that something is "open", but require a payment of $1 billion. My question is, what is a reasonable and fair price to charge?) On the flip side, if I am required to publish my software under a free/open source license and not charge for access to the code, then that means any reader can take my source code and commercialize it, or even simply give it away. Commercialization is part of the four freedoms of free software, but it ends up reducing my market size and knocking other parts of the four freedoms. I won't have as much money to continue my self-funded development and research. The principles are important because they help set guidelines for other questions. Can I require that people register their use of the software before getting access to a no-cost copy? Can I wait 6 months to respond to those requests? How long am I required to host the software, or will the journal manage all of that? May I include non-free license terms, like a requirement to cite X if someone uses the software? Does minimized/obsfucated code suffice? And many more details that have been resolved in the context of F/OSS software but have not, I believe, been resolve in the context of what's needed for peer reviewed publications.

dikaiosune 11y ago
(I'm not a scientist) There are two issues I see with closed source research software. It seems to me like someone outside the research group should review all code run for a paper as part of peer review. If there are glaring off-by-one errors, race conditions, etc., then why should we trust the results produced by the software? There are a lot of things that should get caught in peer review that would only be red flags if someone experienced was able to look at the program source. Also, I've always been under the impression that science should be reproducible. If no one can replicate your results, then how can we trust your findings? While this is technically possible without access to source code, closing source code off from the community really makes reproducing results harder. I'm not sure that it would be fair for journals to require open source availability for publishing, but if they don't then we need a creative solution because these are real problems.

bachmeier 11y ago

> If computation is a necessary element of the research, it ought to be a necessary element of publication as well. Correct. Imagine reading an article in a journal that said, "We were able to prove that Theorem 3.1 is true. The proof is omitted because we want to use similar techniques to prove theorems in the future, and by not publishing any details of our proof, we will have an advantage over rival researchers. That will allow us to recover the significant investment of time that we put into doing this proof."

zevets 11y ago
I can easily imagine such an article, because it happens all the damn time. "Using the argument of [1] and [2] it can be shown, with brief step x, it can be shown that this theorem holds in xxxx" and then moving onto the conclusions, is exactly the same as "Using the software developed in [3] to solve Eq. 1, we show that our result is statistically significant." If the claim is probable, it will and does get through review.

shepardrtc 11y ago

Especially if the work was funded by public money. What should be done in the case of when a researcher creates a private company as a result of publicly-funded research, but the research isn't fully released or is obfuscated?

digikata 11y ago

I lean towards more open code and data, but could one argue that if the code uses some fundamentally new, non-trivial methodology, a replication of the research should write new code implementing that method from it's description. Otherwise, if the original code is re-used, a bug in the code may very well cause the replication of an erroneous result.

bachmeier 11y ago
There are two reasons to look at the code. The first is to look for bugs. Obviously that is easier if you have the code. Testing is much easier than writing from scratch. The second is to see how robust the results are to changes in assumptions and other aspects of the methodology. When you are talking about a big computational problem, it is not realistic to expect others to do a full, independent replication. And if you can't reproduce the original results, you need access to the original code anyway, to find the source of the differences.
coliveira 11y ago
In fact this is the main reason why I don't subscribe to the idea of research papers including code (as a requirement). Independent replication should rely on independent implementation. Unless the software is too generic, the original researcher should have the advantage of using the implemented software for new discoveries, as much as a traditional scientist has the advantage of using his own lab for additional research.
- irixusr 11y ago
  Be able to reimplement the code < That's an interesting perspective. However I think the vast majority of simulation work never gets that sort of scrutiny, no matter if the code is open source Personally though I think we should give up on papers and everything should be published on an ipython notebook style page where others can play with data and code.
dikaiosune 11y ago
Conversely, if the original code isn't available, the community may never know why they can't replicate the results. This could especially be a problem if the original researcher is more known and respected than those attempting replications (i.e. devolving into he said she said).

cafebeen 11y ago

I think the article is talking about a bigger issue, which is expanding our idea of a "research contribution" to include software. The current attitude essentially considers software to be equipment for getting to a scientific result. I think that if we started thinking about research software as a research contribution in itself, it would be a good way to accomplish what you talk about, e.g. by making software (and data) something that is published and cited, rather than attached as a supplement to every publication.

stared 11y ago

I concur. Once I wrote a similar answer http://academia.stackexchange.com/questions/23237/why-are-papers-without-code-but-with-results-accepted/23238#23238 http://academia.stackexchange.com/questions/23237/why-are-pa... In short: benefits for scientists (career, prestige) are misaligned with benefits for science and society (reproducibility, progress, openness).

jpolitz 11y ago

There's progress on this front in computer science conferences in software-focused disciplines: http://www.artifact-eval.org/ http://www.artifact-eval.org/ https://plus.google.com/+JanVitek/posts/19BH96G8rrw https://plus.google.com/+JanVitek/posts/19BH96G8rrw The artifact evaluation workflow is a reasonable starting point for what you suggest.