5 ms·

People not liking open source (and it's not Oracle)

30 points by Garbage 16y ago

scott_s 16y ago
Another conclusion is that scientists gain very little by publishing their code. In research terms, they're evaluated on published papers, receiving grants and graduating students. The incentives just aren't there.
- cdavid 16y ago
  The article is just really poor, and mostly shows the author's complete lack of knowledge on how science is done, and even what it is about. One the biggest reason for not opening your code is to avoid given an unfair advantage to other researchers: academic research is extremely competitive, and the risk for free ride is significant. I am all for opening tools more, but I don't think reproducibility is the reason for it. In any case, reproducibility does not mean everyone using the same software (with the same bugs) - it means independent experiments.
- ntoshev 16y ago
  Are there incentives to have a lower/higher barrier for someone else to reproduce your findings and build on them?
  - scott_s 16y ago
    No, not really.
  - dkarl 16y ago
    If you have no reputation and results that contradict expectations, you definitely want someone to reproduce your work. Otherwise, people may follow a Bayesian argument and decide the mostly likely conclusion is that your work sucks.
- dkarl 16y ago
  If they are willing to be honest, they can publish the code in the condition it's in. What's the extra work in uploading a tarball or exporting a single version to a site like github or Sourceforge? The paper (and the data, which should be published as well) provides all the context needed. I suspect the extra work involved in publishing code is because the code is in such bad shape that it would decrease confidence in the work that depends on it. If that's the case, then it's all the more important that researchers establish it as a standard, non-optional part of publishing research.
  - scott_s 16y ago
    Again, there's no incentive. You're arguing that it's not a burden, but you're not presenting any benefit it has to the researchers. If you want researchers to start doing this, then you need to figure out how to incentivize it.
    - dkarl 16y ago
      Sure I am. There's general disrespect in academia for work that is incompletely reported in a way that hinders review and reproduction. It's a historical accident that refusing to publish source code has come to be treated as a special exception to the principles of good scientific practice. The more the basis for that exception is undermined, the more justified researchers will feel in demanding source code when they're skeptical of a result. And here's the key incentive: lack of source code can be used as an excuse for ignoring and devaluing competitors' contributions. Scientists may be reluctant to raise a standard that would result in more work for everybody, but they can't resist a means of denigrating their rivals' work. Once the cat is out of the bag, scientists with weak reputations and disruptive results will be forced to publish source code to prevent their work being round-filed. Established scientists will find it untenable to hold themselves to an explicitly lower standard than everybody else, so they will eventually be forced to start publishing their code, too.
      - scott_s 16y ago
        It's not a refusal, it's a neglect. What you're saying is, "If the culture was different, the incentives would be there." But the culture is not different. You can't just wave a magic wand. Changing culture is hard.
JoelMcCracken 16y ago
Investigating the world through computation is a relatively new phenomenon for science. Science is discussed and advanced through papers. These papers aren't an exact recording or rendition of what was accomplished, but a discussion of the idea, methods used, and eventually a conclusion. Methods may be algorithms, but it doesn't necessarily mean the actual source code. The idea is that the actual code that a scientist used isn't terribly valuable. If another scientist wants to verify the conclusions, they should just whip up another bit of code (or, do the same experiment) themselves. Asking for scientists to contribute source code is not wrong, it just isn't done (to my knowledge) with other modes of scientific exploration.
- possibilistic 16y ago
  This isn't something new at all! Chemists have been using computers extensively since the 70's, and this can be evidenced by the fact that several physical chemists at my university still use Fortran. Computers have been used in science from their inception. Only now are they becoming indespensible for molecular biologists--but they still have a long way to go in both terms of parallelism and the related tools to be really practical for the intense intermolecular interactions we deal with. Why don't we publish our code? Do you publish one-off shell scripts or database migration code? The only important code is already open source or is (unfortunately) commercial in nature.
  - josegonzalez 16y ago
    I regularly see dotfiles in github repositories. Maybe there is a barrier to adoption in the form of git. A more polished version of Gist - marketed to scientists and researchers - would increase uptake.
    - eru 16y ago
      What do you mean by "see[ing] dotfiles"?
  - JoelMcCracken 16y ago
    It is new wrt the history of science, or more specifically the history of academic journals. I am aware of the history of scientific computation -- that's not the point. The point is that scientists don't need to publish certain things, one of them being the source code that helped them discover their findings. Shell scripts and database migrations aren't super interesting, but you would want them if you needed to replicate results. I'm not saying they're interesting as published research, but neither is that random piece of scratch paper you used while trying to understand what is going on in some biological process. It seems like the problem is that non-scientists don't understand the way science happens, and that often the source code isn't something that you would include in an academic journal. Should science be done differently? Maybe. But that discussion is different than "why wont they publish their source?"
dododo 16y ago
in my experience, it's common in academia for code to "float around". often it's not officially open source or distributed, but you know someone who has the code for some paper, so you can use it. there's even a few textbooks in this state (for years). i think the real reason for not releasing this code is time---you might be surprised at just how much effort goes into making a paper/talk/article polished. sure, you could just release the hacky code you used to do your calculations, but you'd like so much more---for it to be useful, for it to be a credit to you, for it to also do XYZ. research code is efficient in that it gets exactly what you want for the paper; it's rarely usable by anyone else. there is essentially zero payoff in academia in publishing code, and so there's no time to fulfill these desires.
- possibilistic 16y ago
  I definitely agree with this. Besides, the code itself is an insignificant part of the process. Any paper will include more than enough evidence to back up its assertions and provide a path for duplicating its results.
- araneae 16y ago
  Exactly. In my experience, if someone wants to look at your code, they will e-mail you and ask for it. Then you e-mail it to them. Sharing on something like sourceforge just simple isn't done. It's not because scientists are "cheating," it's because it's not part of the culture.
- eru 16y ago
  There could be some payoff for releasing code: If other people use your code, you could probably get a citation for it.
bbb 16y ago
> I can only conclude that many scientists are not confident at all with their theories, or they are purportedly cheating. That's really not an insightful conclusion. I think it's rather a question of pride and effort: the code probably needs to be cleaned up before release, and the reward for investing that time is very low (career-wise). Researchers are going to publish their source code when it becomes required by most major journals and conferences.
tensor 16y ago
I think it's worth pointing out that taking someones code and running it again on the same data does not constitute an independent verification of the original work. Just like scientists don't go using the same lab to verify an experiment, they shouldn't use the same code. That said, it is often useful to release code for other reasons. For applied areas, releasing the code in the form of a software tool is fairly common. This allows people from the target area, who may not be computer scientists, easy access to the new method. Code review seems to be nearly non-existent in the scientific community, although it does happen occasionally when one researcher uses another's code base for further experimentation.
bhickey 16y ago
I've worked as an academic hacker. Amongst dedicated biologists, there's no motivation to release your source code. It's the job of a biologist to publish papers so that he or she can pull down the next grant. Anything that doesn't further this goal is going to get zero traction. Publishing it doesn't help you get into Nature or Science. Sure, Sean Eddy (www.psc.edu/general/software/packages/hmmer/) is going to release his code, but he's a methods guy. The software is the focus of publication. The only way code is going to get published is if the journals refuse to publish results without code.

30 points by Garbage 16y ago

Another conclusion is that scientists gain very little by publishing their code. In research terms, they're evaluated on published papers, receiving grants and graduating students. The incentives just aren't there.

cdavid 16y ago
The article is just really poor, and mostly shows the author's complete lack of knowledge on how science is done, and even what it is about. One the biggest reason for not opening your code is to avoid given an unfair advantage to other researchers: academic research is extremely competitive, and the risk for free ride is significant. I am all for opening tools more, but I don't think reproducibility is the reason for it. In any case, reproducibility does not mean everyone using the same software (with the same bugs) - it means independent experiments.
ntoshev 16y ago
Are there incentives to have a lower/higher barrier for someone else to reproduce your findings and build on them?
- scott_s 16y ago
  No, not really.
- dkarl 16y ago
  If you have no reputation and results that contradict expectations, you definitely want someone to reproduce your work. Otherwise, people may follow a Bayesian argument and decide the mostly likely conclusion is that your work sucks.
dkarl 16y ago
If they are willing to be honest, they can publish the code in the condition it's in. What's the extra work in uploading a tarball or exporting a single version to a site like github or Sourceforge? The paper (and the data, which should be published as well) provides all the context needed. I suspect the extra work involved in publishing code is because the code is in such bad shape that it would decrease confidence in the work that depends on it. If that's the case, then it's all the more important that researchers establish it as a standard, non-optional part of publishing research.
- scott_s 16y ago
  Again, there's no incentive. You're arguing that it's not a burden, but you're not presenting any benefit it has to the researchers. If you want researchers to start doing this, then you need to figure out how to incentivize it.
  - dkarl 16y ago
    Sure I am. There's general disrespect in academia for work that is incompletely reported in a way that hinders review and reproduction. It's a historical accident that refusing to publish source code has come to be treated as a special exception to the principles of good scientific practice. The more the basis for that exception is undermined, the more justified researchers will feel in demanding source code when they're skeptical of a result. And here's the key incentive: lack of source code can be used as an excuse for ignoring and devaluing competitors' contributions. Scientists may be reluctant to raise a standard that would result in more work for everybody, but they can't resist a means of denigrating their rivals' work. Once the cat is out of the bag, scientists with weak reputations and disruptive results will be forced to publish source code to prevent their work being round-filed. Established scientists will find it untenable to hold themselves to an explicitly lower standard than everybody else, so they will eventually be forced to start publishing their code, too.
    - scott_s 16y ago
      It's not a refusal, it's a neglect. What you're saying is, "If the culture was different, the incentives would be there." But the culture is not different. You can't just wave a magic wand. Changing culture is hard.

JoelMcCracken 16y ago

Investigating the world through computation is a relatively new phenomenon for science. Science is discussed and advanced through papers. These papers aren't an exact recording or rendition of what was accomplished, but a discussion of the idea, methods used, and eventually a conclusion. Methods may be algorithms, but it doesn't necessarily mean the actual source code. The idea is that the actual code that a scientist used isn't terribly valuable. If another scientist wants to verify the conclusions, they should just whip up another bit of code (or, do the same experiment) themselves. Asking for scientists to contribute source code is not wrong, it just isn't done (to my knowledge) with other modes of scientific exploration.

possibilistic 16y ago
This isn't something new at all! Chemists have been using computers extensively since the 70's, and this can be evidenced by the fact that several physical chemists at my university still use Fortran. Computers have been used in science from their inception. Only now are they becoming indespensible for molecular biologists--but they still have a long way to go in both terms of parallelism and the related tools to be really practical for the intense intermolecular interactions we deal with. Why don't we publish our code? Do you publish one-off shell scripts or database migration code? The only important code is already open source or is (unfortunately) commercial in nature.
- josegonzalez 16y ago
  I regularly see dotfiles in github repositories. Maybe there is a barrier to adoption in the form of git. A more polished version of Gist - marketed to scientists and researchers - would increase uptake.
  - eru 16y ago
    What do you mean by "see[ing] dotfiles"?
- JoelMcCracken 16y ago
  It is new wrt the history of science, or more specifically the history of academic journals. I am aware of the history of scientific computation -- that's not the point. The point is that scientists don't need to publish certain things, one of them being the source code that helped them discover their findings. Shell scripts and database migrations aren't super interesting, but you would want them if you needed to replicate results. I'm not saying they're interesting as published research, but neither is that random piece of scratch paper you used while trying to understand what is going on in some biological process. It seems like the problem is that non-scientists don't understand the way science happens, and that often the source code isn't something that you would include in an academic journal. Should science be done differently? Maybe. But that discussion is different than "why wont they publish their source?"

dododo 16y ago

in my experience, it's common in academia for code to "float around". often it's not officially open source or distributed, but you know someone who has the code for some paper, so you can use it. there's even a few textbooks in this state (for years). i think the real reason for not releasing this code is time---you might be surprised at just how much effort goes into making a paper/talk/article polished. sure, you could just release the hacky code you used to do your calculations, but you'd like so much more---for it to be useful, for it to be a credit to you, for it to also do XYZ. research code is efficient in that it gets exactly what you want for the paper; it's rarely usable by anyone else. there is essentially zero payoff in academia in publishing code, and so there's no time to fulfill these desires.

possibilistic 16y ago
I definitely agree with this. Besides, the code itself is an insignificant part of the process. Any paper will include more than enough evidence to back up its assertions and provide a path for duplicating its results.
araneae 16y ago
Exactly. In my experience, if someone wants to look at your code, they will e-mail you and ask for it. Then you e-mail it to them. Sharing on something like sourceforge just simple isn't done. It's not because scientists are "cheating," it's because it's not part of the culture.
eru 16y ago
There could be some payoff for releasing code: If other people use your code, you could probably get a citation for it.

bbb 16y ago

> I can only conclude that many scientists are not confident at all with their theories, or they are purportedly cheating. That's really not an insightful conclusion. I think it's rather a question of pride and effort: the code probably needs to be cleaned up before release, and the reward for investing that time is very low (career-wise). Researchers are going to publish their source code when it becomes required by most major journals and conferences.

tensor 16y ago

I think it's worth pointing out that taking someones code and running it again on the same data does not constitute an independent verification of the original work. Just like scientists don't go using the same lab to verify an experiment, they shouldn't use the same code. That said, it is often useful to release code for other reasons. For applied areas, releasing the code in the form of a software tool is fairly common. This allows people from the target area, who may not be computer scientists, easy access to the new method. Code review seems to be nearly non-existent in the scientific community, although it does happen occasionally when one researcher uses another's code base for further experimentation.

bhickey 16y ago

I've worked as an academic hacker. Amongst dedicated biologists, there's no motivation to release your source code. It's the job of a biologist to publish papers so that he or she can pull down the next grant. Anything that doesn't further this goal is going to get zero traction. Publishing it doesn't help you get into Nature or Science. Sure, Sean Eddy (www.psc.edu/general/software/packages/hmmer/) is going to release his code, but he's a methods guy. The software is the focus of publication. The only way code is going to get published is if the journals refuse to publish results without code.