11 ms·

The unsung heroes of scientific software

151 points by bootload 11y ago

s_q_b 11y ago
Personally I believe that open code, raw data, and cleaned datasets should be requirements for publication in any peer-reviewed journal in which a paper claims that computational mechanisms were involved. Failure to distribute the data and code not only greatly devalues the contributions of many scientists, it makes replication far more difficult, and opens the door for outright fabrication. If computation is a necessary element of the research, it ought to be a necessary element of publication as well.
- gww 11y ago
  I could not agree with you more. It is a disconcerting trend to see high impact papers using an "in house program" or "a modified version of a previously described tool", without providing any details about software versions, analysis steps and any custom filtering applied to the data. There could be bugs, unintentionally (or intentionally) introduced biases, or other things such as outlier removal that are not explained in the manuscripts methods. This is also very true for statistical analysis, many papers do not provide R scripts for their analysis and only a brief overview of the analysis.
- noname123 11y ago
  >Failure to distribute the data and code not only greatly devalues the contributions of many scientists, it makes replication far more difficult, and opens the door for outright fabrication. Reasons why labs don't produce code: (1) Fear of being scooped; suppose you put in tons of man-hours into a custom population genetics association study for malaria in Africa, you don't want your competitor lab to sequence a bunch samples in SE Asia and run your code as-is and publish a paper when you have the sequencers to do the same. (2) Fear of competitors not being able to replicate stochastic ML results; some machine-learning and artificial neural network papers applied to Biology are stochastic. Past authors have been accused of "cherry-picking" the most "optimistic" runs that show positive results, e.g., (https://liorpachter.wordpress.com/2014/02/11/the-network-nonsense-of-manolis-kellis/ https://liorpachter.wordpress.com/2014/02/11/the-network-non...) (3) Focus on science, not tool production; most labs' focus is to produce publications, not open-source software. User base for scientific software is very niche, unlike Web MVC frameworks; making the payoff calculus of packaging and supporting external users not so great. Furthermore, most labs have custom databases making integrating external software, externalizing internal software difficult.
  - pvnick 11y ago
    Just to follow up on your last point, I have found that folks are often happy to send code, but code written in research labs is generally horrendous. I don't think that the "publish or die" culture and a culture of releasing all relevant code is compatible. It takes time to refactor and produce clean code - time that can be spent chasing the next publication.
    - kwhitefoot 11y ago
      Then requiring the code to be published and runnable would have a salutary effect on the whole circus.
      - SapphireSun 11y ago
        Not really... I don't see how that would be anything more than adding one more task to a million other tasks. It doesn't change incentives.
    - IanCal 11y ago
      I don't see the need for clean code, I just want the code they used. I've repeatedly tried to find the code used to generate various visualisations (which were the entire point of the paper) and then found 1. "It'll be released soon" on a years old page 2. 404'ing personal pages on university sites 3. Finally some code But then the code didn't, and couldn't, generate the images they have. So now I'm not actually sure if the algorithm works as they say. This was also one of the more successful results I've had in trying to find research code outside of ML. I'm sure other disciplines are also good but ML is mostly what I've had to find. There's a part of the experiment which virtually anyone could replicate for a material cost of near 0 and check in detail if they wanted to and yet that seems to be the most poorly explained and shared part of it. You wouldn't get away with just mentioning the protocol you used in the lab without any references or explanations, would you?
  - ppod 11y ago
    (1) You publish the code at the time of publication of the paper, not before. (2) This is a reason to publish the code, not a reason not to publish it. (3) You don't necessarily have to make package it neatly like production software, but some effort is necessary. We just wouldn't accept these kind of excuses for other descriptions of methodology, even though they might apply. There's no reason that code should be an exception.
- tdaltonc 11y ago
  It would have the side effect of improving and propagating best practices. Most scientists are self taught programers, and it shows. The data models and codebases used by most labs are . . . amateurish.
  - dikaiosune 11y ago
    I really like what this group is doing: http://software-carpentry.org/ http://software-carpentry.org/ I took their instructor training last summer but I haven't had a chance to run a workshop yet. I think it's a great idea, especially given my recent exposure to researcher-written codebases.
  - Thriptic 11y ago
    Seriously, I consider myself lucky when I get code that's even been commented from labmates...It took me about a year to get 50% of people using github
    - collyw 11y ago
      One ex-colleague of mine wouldn't even indent her Perl code properly. I showed here there was even a button in eclipse to do it for her. (Using eclipse seems problematic when there is a text editor that doesn't involve any cognitive overhead to many bioinformaticians. The step through debugger did seem to appeal to this one though).
- dalke 11y ago
  Could you explain what you mean by "open code"? What is the philosophical underpinning? That is, it seems that most of the argument for "open code" in the scientific review sense implies that 1) the software should be available at no cost, and 2) there is no need to support commercialization of said software. These principles are different than the four freedoms of "free software", and I have difficulties in reconciling the differences. For a concrete example, I am self-employed. I sell scientific software. All of my customers receive it under the BSD license, after they pay me a good chunk of money. Thus, I sell free software for scientific research. The FSF says "Selling a copy of a free program is legitimate, and we encourage it. (Quoting https://www.gnu.org/philosophy/selling.html https://www.gnu.org/philosophy/selling.html .) But most of the time when I see people say "open code" or something similar, they want the right to view, use, test, and modify the source code at no cost. If I publish a peer-reviewed paper about the software, should I be required to distribute the software to readers for no cost? Or may I set a fee of, say, $25,000 to get access to the source code under a free license? (I'm well aware of the loophole where I could publish that something is "open", but require a payment of $1 billion. My question is, what is a reasonable and fair price to charge?) On the flip side, if I am required to publish my software under a free/open source license and not charge for access to the code, then that means any reader can take my source code and commercialize it, or even simply give it away. Commercialization is part of the four freedoms of free software, but it ends up reducing my market size and knocking other parts of the four freedoms. I won't have as much money to continue my self-funded development and research. The principles are important because they help set guidelines for other questions. Can I require that people register their use of the software before getting access to a no-cost copy? Can I wait 6 months to respond to those requests? How long am I required to host the software, or will the journal manage all of that? May I include non-free license terms, like a requirement to cite X if someone uses the software? Does minimized/obsfucated code suffice? And many more details that have been resolved in the context of F/OSS software but have not, I believe, been resolve in the context of what's needed for peer reviewed publications.
  - dikaiosune 11y ago
    (I'm not a scientist) There are two issues I see with closed source research software. It seems to me like someone outside the research group should review all code run for a paper as part of peer review. If there are glaring off-by-one errors, race conditions, etc., then why should we trust the results produced by the software? There are a lot of things that should get caught in peer review that would only be red flags if someone experienced was able to look at the program source. Also, I've always been under the impression that science should be reproducible. If no one can replicate your results, then how can we trust your findings? While this is technically possible without access to source code, closing source code off from the community really makes reproducing results harder. I'm not sure that it would be fair for journals to require open source availability for publishing, but if they don't then we need a creative solution because these are real problems.
- bachmeier 11y ago
  > If computation is a necessary element of the research, it ought to be a necessary element of publication as well. Correct. Imagine reading an article in a journal that said, "We were able to prove that Theorem 3.1 is true. The proof is omitted because we want to use similar techniques to prove theorems in the future, and by not publishing any details of our proof, we will have an advantage over rival researchers. That will allow us to recover the significant investment of time that we put into doing this proof."
  - zevets 11y ago
    I can easily imagine such an article, because it happens all the damn time. "Using the argument of [1] and [2] it can be shown, with brief step x, it can be shown that this theorem holds in xxxx" and then moving onto the conclusions, is exactly the same as "Using the software developed in [3] to solve Eq. 1, we show that our result is statistically significant." If the claim is probable, it will and does get through review.
- shepardrtc 11y ago
  Especially if the work was funded by public money. What should be done in the case of when a researcher creates a private company as a result of publicly-funded research, but the research isn't fully released or is obfuscated?
- digikata 11y ago
  I lean towards more open code and data, but could one argue that if the code uses some fundamentally new, non-trivial methodology, a replication of the research should write new code implementing that method from it's description. Otherwise, if the original code is re-used, a bug in the code may very well cause the replication of an erroneous result.
  - bachmeier 11y ago
    There are two reasons to look at the code. The first is to look for bugs. Obviously that is easier if you have the code. Testing is much easier than writing from scratch. The second is to see how robust the results are to changes in assumptions and other aspects of the methodology. When you are talking about a big computational problem, it is not realistic to expect others to do a full, independent replication. And if you can't reproduce the original results, you need access to the original code anyway, to find the source of the differences.
  - coliveira 11y ago
    In fact this is the main reason why I don't subscribe to the idea of research papers including code (as a requirement). Independent replication should rely on independent implementation. Unless the software is too generic, the original researcher should have the advantage of using the implemented software for new discoveries, as much as a traditional scientist has the advantage of using his own lab for additional research.
    - irixusr 11y ago
      Be able to reimplement the code < That's an interesting perspective. However I think the vast majority of simulation work never gets that sort of scrutiny, no matter if the code is open source Personally though I think we should give up on papers and everything should be published on an ipython notebook style page where others can play with data and code.
  - dikaiosune 11y ago
    Conversely, if the original code isn't available, the community may never know why they can't replicate the results. This could especially be a problem if the original researcher is more known and respected than those attempting replications (i.e. devolving into he said she said).
- cafebeen 11y ago
  I think the article is talking about a bigger issue, which is expanding our idea of a "research contribution" to include software. The current attitude essentially considers software to be equipment for getting to a scientific result. I think that if we started thinking about research software as a research contribution in itself, it would be a good way to accomplish what you talk about, e.g. by making software (and data) something that is published and cited, rather than attached as a supplement to every publication.
- stared 11y ago
  I concur. Once I wrote a similar answer http://academia.stackexchange.com/questions/23237/why-are-papers-without-code-but-with-results-accepted/23238#23238 http://academia.stackexchange.com/questions/23237/why-are-pa... In short: benefits for scientists (career, prestige) are misaligned with benefits for science and society (reproducibility, progress, openness).
- jpolitz 11y ago
  There's progress on this front in computer science conferences in software-focused disciplines: http://www.artifact-eval.org/ http://www.artifact-eval.org/ https://plus.google.com/+JanVitek/posts/19BH96G8rrw https://plus.google.com/+JanVitek/posts/19BH96G8rrw The artifact evaluation workflow is a reasonable starting point for what you suggest.
dikaiosune 11y ago
Very neat, but they're only indexing CRAN and PyPI right now. In my short time doing scientific computing (staff member at a university right now), I've seen a LOT of C and C++, but those don't have canonical package repositories. I think that finding all of those codebases (even if just limited to GitHub) and indexing those would be very interesting and would be necessary to have representation from a few fields.
- zbjornson 11y ago
  They also seem to be missing Bioconductor, which is where a huge percentage of (maybe even most?) R biology packages are located. Allowing submissions and somehow indexing the relevant repositories in GitHub would be huge.
  - jasonpriem 11y ago
    Yep, we chose CRAN for the MVP, since it's the main R package host. But adding other sources is the plan, and Bioconductor is at the top of that list for R.
- jasonpriem 11y ago
  Yes, agreed...we've had lots of requests to add other languages. And as you say, the growing popularity of GitHub gives us all kinds of cool data even when there's no central package manager for the language. In fact, we're mining imports of every Python and R project on GitHub right now to build out the dependency network beyond the (much much smaller) CRAN and PyPi networks. The idea with Depsy has been to launch quickly with two languages, so people could see what it looks like, then iterate and add more as we get feedback. So we'll count your comment as +1 for C and C++ :)
- lutorm 11y ago
  I strongly oppose "just limiting to GitHub". If this is to be used as some sort of merit factor, its inclusion can't be contingent on using a particular centralized repo. For a start, there are several such providers. Larger projects might well prefer self-hosting, too.
amelius 11y ago
What about generic tools, such as webbrowsers? The web was invented at CERN by a researcher, so I guess webbrowsers must have some value to the scientific process :)
- deleted 11y ago
  [deleted]
chenning 11y ago
Soooo... should people get recognition, or not? Re: https://news.ycombinator.com/item?id=10838166 https://news.ycombinator.com/item?id=10838166
jamesblonde 11y ago
I fully agree with the principle that experiments should be reproducible. In systems research conferences (SOSP, Eurosys, OSDI) very few of the systems publish their algorithms. The worst culprits are the companies, such as Google, Facebook, and Microsoft, who don't even publish all of their algorithms or systems properties. Google have published significant papers that leave out the key algorithms on how parts of their systems work (e.g., In Borg how does the scheduler synchronize the cluster state with the replicas? In Spanner, what are the properties of the underlying storage system, Collosus?). I would love to see all distributed systems papers be able to reproduce everything by just downloading a file that allows the paper's results to be reproduced. We can do it with the help of either virtualization or containers and software configuration frameworks (to parameterize the experiments). We can specify the hardware and network programmatically, we can install the software automatically, we can parameterize system software (with Chef or Puppet attributes), and run the experiments reproducibly. Without huge investment and cloud computing or Docker, we could specify systems that have reproducible hardware/network/software. The first attempt I've seen at this is www.karamel.io that allows you to design reproducible experiments by using JClouds to spawn VMs and setup the virtualized hardware, then orhcestrating Chef to install software that can be parameterized. I hope to see more of this line of platform gain adoption in our community. It would be a boon for research.
- dekhn 11y ago
  Lots of people have proposed and implemented VM-based reproducibility environments. 1) it's actually harder to get reproducible computational experiments than they expected. For example, you can run same VM on a different processor and get different results, which makes bitwise reproduction hard, and statistical tests for nonbitwise equality are harder 2) developing and maintaining the VMs and the environment takes a fair amount of effort from skilled people 3) the resulting improvements to science don't seem to exceed the cost thresholds implied by #1 and #2, and nobody's volunteering their time. My conclusion: good idea, but probably not critically necessary.
  - jamesblonde 11y ago
    That's good feedback. Although I'm considering mostly distributed systems, where full reproducability is not possible. For single-threaded, deterministic programs (discrete event simulations) where performance is not a measurement point, you can do it with VMs. The nicest thing would be to be able to parameterize experiments. Not just re-run them, but change the parameters for different runs.
chrisdbaldwin 11y ago
When I think about unsung software heroes I think of Rob Scharein[1]. He is the creator or KnotPlot[2] which has enabled topological research to flourish. He recieves regular acknowledgement among topology researchers, and he's an awesome guy. Do yourself a favor and play with some knots. [1] www.hypnagogic.net/rob/ [2] www.knotplot.com
jcoffland 11y ago
I've been developing research software for +10 years and I experience this all the time. As one example, over a decade ago I wrote a biological simulation program called CompuCell3D it was the successor to CompuCell, a 2D simulator of cells as cellular automata. Since then many papers have been written based on research utilising this software. Not only have I not been credited in any of these publications but my name has been removed from the software, the website never mentions my original contribution and the current maintainers of said software are not responding to my emails. Granted this code has changed a lot over the years but there are still large parts of the code which are still verbatim from my original software. The researchers in this project see it as totally irrelevant that I wrote the original code because they value code creation MUCH less than their research. This particular case is especially egregious but it is a good anecdotal example of the problem. Sometimes code is not as significant or as breakthrough as raw research. It really matters what kind of coding you are talking about. You don't necessarily credit the construction crew when dedicating a new building but you do credit the architect. Coders are under valued in today's research environment.
- dekhn 11y ago
  What role did you have when you wrote the code? If, for example, you were a full time employee, grad student, or postdoc at a university, the copyright belongs to the university (not true for undergrads!) The website isn't required to list your contribution. That would be a courtesy. Also, if you have a problem with this where you think it's important to change the website or get another form of credit, your lawyer should be talking to your ex-employers lawyer. That should have been clear when the current maintainers failed to respond to your email.
- dekhn 11y ago
  The paper for citing CompuCell3D is Multi-Scale Modeling of Tissues Using CompuCell3D – M. Swat, Gilberto L. Thomas, Julio M. Belmonte, A. Shirinifard, D.Hmeljak, J. A. Glazier, Computational Methods in Cell Biology, Methods in Cell Biology 110: 325-366 (2012) I can't see that article online without paying, but I am certain it cites CompuCell, a Multi-Model Framework For Simulation of Morphogenesis – J. A. Izaguirre, R. Chaturvedi, C. Huang, T. Cickovski, J. Coffland, G. Thomas, G. Forgacs, M. Alber, G. Hentschel, S. A. Newman, and J. A. Glazier, Bioinformatics 20: 1129-1137 (2004) that original article includes your name on the author list. So your statement "I have not been credited in any of these publications" isn't really correct; your contribution is credited indirectly through citations. I don't see any problem with this; I doubt papers citing CompuCell3D have any text like "We acknowledge the efforts of <so and so>".
pdm55 11y ago
My unsung software hero is Dr. Jeffrey Lewis Fox (deceased in 1999 at age 51), Associate Professor, Department of Pharmaceutics and Pharmaceutical Chemistry, University of Utah http://pharmacy.utah.edu/pharmaceutics/news/1999.html#fox http://pharmacy.utah.edu/pharmaceutics/news/1999.html#fox He wrote MINSQ (now called Scientist), the only piece of software I have ever fallen in love with. He sold the first version for $10 (if I recall correctly) in 1990. It was so user-friendly that I was using it within 10 mins to solve a system of differential equations that were part of my PhD research. (Unfortunately, I don't find the more recent versions, "updated" by others, anywhere near as user-friendly.) He called his company, at the University of Utah, MicroMath. I assume he used the algorithms from "Numerical Recipes" to write MINSQ - I know that was the alternative facing me till his excellent software appeared. I really think the U of Utah should have kept the rights to the software and maintained it in his honour.
abawany 11y ago
I will use this opportunity to mention Maxima (https://en.wikipedia.org/wiki/Maxima_%28software%29 https://en.wikipedia.org/wiki/Maxima_%28software%29) that was developed by one of the best math professors that I have had the privilege to learn from: Bill Schelter. It was forked from Macsyma and released under the GPL and still appears to be actively maintained.

151 points by bootload 11y ago

s_q_b 11y ago

Personally I believe that open code, raw data, and cleaned datasets should be requirements for publication in any peer-reviewed journal in which a paper claims that computational mechanisms were involved. Failure to distribute the data and code not only greatly devalues the contributions of many scientists, it makes replication far more difficult, and opens the door for outright fabrication. If computation is a necessary element of the research, it ought to be a necessary element of publication as well.

gww 11y ago
I could not agree with you more. It is a disconcerting trend to see high impact papers using an "in house program" or "a modified version of a previously described tool", without providing any details about software versions, analysis steps and any custom filtering applied to the data. There could be bugs, unintentionally (or intentionally) introduced biases, or other things such as outlier removal that are not explained in the manuscripts methods. This is also very true for statistical analysis, many papers do not provide R scripts for their analysis and only a brief overview of the analysis.
noname123 11y ago
>Failure to distribute the data and code not only greatly devalues the contributions of many scientists, it makes replication far more difficult, and opens the door for outright fabrication. Reasons why labs don't produce code: (1) Fear of being scooped; suppose you put in tons of man-hours into a custom population genetics association study for malaria in Africa, you don't want your competitor lab to sequence a bunch samples in SE Asia and run your code as-is and publish a paper when you have the sequencers to do the same. (2) Fear of competitors not being able to replicate stochastic ML results; some machine-learning and artificial neural network papers applied to Biology are stochastic. Past authors have been accused of "cherry-picking" the most "optimistic" runs that show positive results, e.g., (https://liorpachter.wordpress.com/2014/02/11/the-network-nonsense-of-manolis-kellis/ https://liorpachter.wordpress.com/2014/02/11/the-network-non...) (3) Focus on science, not tool production; most labs' focus is to produce publications, not open-source software. User base for scientific software is very niche, unlike Web MVC frameworks; making the payoff calculus of packaging and supporting external users not so great. Furthermore, most labs have custom databases making integrating external software, externalizing internal software difficult.
- pvnick 11y ago
  Just to follow up on your last point, I have found that folks are often happy to send code, but code written in research labs is generally horrendous. I don't think that the "publish or die" culture and a culture of releasing all relevant code is compatible. It takes time to refactor and produce clean code - time that can be spent chasing the next publication.
  - kwhitefoot 11y ago
    Then requiring the code to be published and runnable would have a salutary effect on the whole circus.
    - SapphireSun 11y ago
      Not really... I don't see how that would be anything more than adding one more task to a million other tasks. It doesn't change incentives.
  - IanCal 11y ago
    I don't see the need for clean code, I just want the code they used. I've repeatedly tried to find the code used to generate various visualisations (which were the entire point of the paper) and then found 1. "It'll be released soon" on a years old page 2. 404'ing personal pages on university sites 3. Finally some code But then the code didn't, and couldn't, generate the images they have. So now I'm not actually sure if the algorithm works as they say. This was also one of the more successful results I've had in trying to find research code outside of ML. I'm sure other disciplines are also good but ML is mostly what I've had to find. There's a part of the experiment which virtually anyone could replicate for a material cost of near 0 and check in detail if they wanted to and yet that seems to be the most poorly explained and shared part of it. You wouldn't get away with just mentioning the protocol you used in the lab without any references or explanations, would you?
- ppod 11y ago
  (1) You publish the code at the time of publication of the paper, not before. (2) This is a reason to publish the code, not a reason not to publish it. (3) You don't necessarily have to make package it neatly like production software, but some effort is necessary. We just wouldn't accept these kind of excuses for other descriptions of methodology, even though they might apply. There's no reason that code should be an exception.
tdaltonc 11y ago
It would have the side effect of improving and propagating best practices. Most scientists are self taught programers, and it shows. The data models and codebases used by most labs are . . . amateurish.
- dikaiosune 11y ago
  I really like what this group is doing: http://software-carpentry.org/ http://software-carpentry.org/ I took their instructor training last summer but I haven't had a chance to run a workshop yet. I think it's a great idea, especially given my recent exposure to researcher-written codebases.
- Thriptic 11y ago
  Seriously, I consider myself lucky when I get code that's even been commented from labmates...It took me about a year to get 50% of people using github
  - collyw 11y ago
    One ex-colleague of mine wouldn't even indent her Perl code properly. I showed here there was even a button in eclipse to do it for her. (Using eclipse seems problematic when there is a text editor that doesn't involve any cognitive overhead to many bioinformaticians. The step through debugger did seem to appeal to this one though).
dalke 11y ago
Could you explain what you mean by "open code"? What is the philosophical underpinning? That is, it seems that most of the argument for "open code" in the scientific review sense implies that 1) the software should be available at no cost, and 2) there is no need to support commercialization of said software. These principles are different than the four freedoms of "free software", and I have difficulties in reconciling the differences. For a concrete example, I am self-employed. I sell scientific software. All of my customers receive it under the BSD license, after they pay me a good chunk of money. Thus, I sell free software for scientific research. The FSF says "Selling a copy of a free program is legitimate, and we encourage it. (Quoting https://www.gnu.org/philosophy/selling.html https://www.gnu.org/philosophy/selling.html .) But most of the time when I see people say "open code" or something similar, they want the right to view, use, test, and modify the source code at no cost. If I publish a peer-reviewed paper about the software, should I be required to distribute the software to readers for no cost? Or may I set a fee of, say, $25,000 to get access to the source code under a free license? (I'm well aware of the loophole where I could publish that something is "open", but require a payment of $1 billion. My question is, what is a reasonable and fair price to charge?) On the flip side, if I am required to publish my software under a free/open source license and not charge for access to the code, then that means any reader can take my source code and commercialize it, or even simply give it away. Commercialization is part of the four freedoms of free software, but it ends up reducing my market size and knocking other parts of the four freedoms. I won't have as much money to continue my self-funded development and research. The principles are important because they help set guidelines for other questions. Can I require that people register their use of the software before getting access to a no-cost copy? Can I wait 6 months to respond to those requests? How long am I required to host the software, or will the journal manage all of that? May I include non-free license terms, like a requirement to cite X if someone uses the software? Does minimized/obsfucated code suffice? And many more details that have been resolved in the context of F/OSS software but have not, I believe, been resolve in the context of what's needed for peer reviewed publications.
- dikaiosune 11y ago
  (I'm not a scientist) There are two issues I see with closed source research software. It seems to me like someone outside the research group should review all code run for a paper as part of peer review. If there are glaring off-by-one errors, race conditions, etc., then why should we trust the results produced by the software? There are a lot of things that should get caught in peer review that would only be red flags if someone experienced was able to look at the program source. Also, I've always been under the impression that science should be reproducible. If no one can replicate your results, then how can we trust your findings? While this is technically possible without access to source code, closing source code off from the community really makes reproducing results harder. I'm not sure that it would be fair for journals to require open source availability for publishing, but if they don't then we need a creative solution because these are real problems.
bachmeier 11y ago
> If computation is a necessary element of the research, it ought to be a necessary element of publication as well. Correct. Imagine reading an article in a journal that said, "We were able to prove that Theorem 3.1 is true. The proof is omitted because we want to use similar techniques to prove theorems in the future, and by not publishing any details of our proof, we will have an advantage over rival researchers. That will allow us to recover the significant investment of time that we put into doing this proof."
- zevets 11y ago
  I can easily imagine such an article, because it happens all the damn time. "Using the argument of [1] and [2] it can be shown, with brief step x, it can be shown that this theorem holds in xxxx" and then moving onto the conclusions, is exactly the same as "Using the software developed in [3] to solve Eq. 1, we show that our result is statistically significant." If the claim is probable, it will and does get through review.
shepardrtc 11y ago
Especially if the work was funded by public money. What should be done in the case of when a researcher creates a private company as a result of publicly-funded research, but the research isn't fully released or is obfuscated?
digikata 11y ago
I lean towards more open code and data, but could one argue that if the code uses some fundamentally new, non-trivial methodology, a replication of the research should write new code implementing that method from it's description. Otherwise, if the original code is re-used, a bug in the code may very well cause the replication of an erroneous result.
- bachmeier 11y ago
  There are two reasons to look at the code. The first is to look for bugs. Obviously that is easier if you have the code. Testing is much easier than writing from scratch. The second is to see how robust the results are to changes in assumptions and other aspects of the methodology. When you are talking about a big computational problem, it is not realistic to expect others to do a full, independent replication. And if you can't reproduce the original results, you need access to the original code anyway, to find the source of the differences.
- coliveira 11y ago
  In fact this is the main reason why I don't subscribe to the idea of research papers including code (as a requirement). Independent replication should rely on independent implementation. Unless the software is too generic, the original researcher should have the advantage of using the implemented software for new discoveries, as much as a traditional scientist has the advantage of using his own lab for additional research.
  - irixusr 11y ago
    Be able to reimplement the code < That's an interesting perspective. However I think the vast majority of simulation work never gets that sort of scrutiny, no matter if the code is open source Personally though I think we should give up on papers and everything should be published on an ipython notebook style page where others can play with data and code.
- dikaiosune 11y ago
  Conversely, if the original code isn't available, the community may never know why they can't replicate the results. This could especially be a problem if the original researcher is more known and respected than those attempting replications (i.e. devolving into he said she said).
cafebeen 11y ago
I think the article is talking about a bigger issue, which is expanding our idea of a "research contribution" to include software. The current attitude essentially considers software to be equipment for getting to a scientific result. I think that if we started thinking about research software as a research contribution in itself, it would be a good way to accomplish what you talk about, e.g. by making software (and data) something that is published and cited, rather than attached as a supplement to every publication.
stared 11y ago
I concur. Once I wrote a similar answer http://academia.stackexchange.com/questions/23237/why-are-papers-without-code-but-with-results-accepted/23238#23238 http://academia.stackexchange.com/questions/23237/why-are-pa... In short: benefits for scientists (career, prestige) are misaligned with benefits for science and society (reproducibility, progress, openness).
jpolitz 11y ago
There's progress on this front in computer science conferences in software-focused disciplines: http://www.artifact-eval.org/ http://www.artifact-eval.org/ https://plus.google.com/+JanVitek/posts/19BH96G8rrw https://plus.google.com/+JanVitek/posts/19BH96G8rrw The artifact evaluation workflow is a reasonable starting point for what you suggest.

dikaiosune 11y ago

Very neat, but they're only indexing CRAN and PyPI right now. In my short time doing scientific computing (staff member at a university right now), I've seen a LOT of C and C++, but those don't have canonical package repositories. I think that finding all of those codebases (even if just limited to GitHub) and indexing those would be very interesting and would be necessary to have representation from a few fields.

zbjornson 11y ago
They also seem to be missing Bioconductor, which is where a huge percentage of (maybe even most?) R biology packages are located. Allowing submissions and somehow indexing the relevant repositories in GitHub would be huge.
- jasonpriem 11y ago
  Yep, we chose CRAN for the MVP, since it's the main R package host. But adding other sources is the plan, and Bioconductor is at the top of that list for R.
jasonpriem 11y ago
Yes, agreed...we've had lots of requests to add other languages. And as you say, the growing popularity of GitHub gives us all kinds of cool data even when there's no central package manager for the language. In fact, we're mining imports of every Python and R project on GitHub right now to build out the dependency network beyond the (much much smaller) CRAN and PyPi networks. The idea with Depsy has been to launch quickly with two languages, so people could see what it looks like, then iterate and add more as we get feedback. So we'll count your comment as +1 for C and C++ :)
lutorm 11y ago
I strongly oppose "just limiting to GitHub". If this is to be used as some sort of merit factor, its inclusion can't be contingent on using a particular centralized repo. For a start, there are several such providers. Larger projects might well prefer self-hosting, too.

amelius 11y ago

What about generic tools, such as webbrowsers? The web was invented at CERN by a researcher, so I guess webbrowsers must have some value to the scientific process :)

deleted 11y ago
[deleted]

chenning 11y ago

Soooo... should people get recognition, or not? Re: https://news.ycombinator.com/item?id=10838166 https://news.ycombinator.com/item?id=10838166

jamesblonde 11y ago

I fully agree with the principle that experiments should be reproducible. In systems research conferences (SOSP, Eurosys, OSDI) very few of the systems publish their algorithms. The worst culprits are the companies, such as Google, Facebook, and Microsoft, who don't even publish all of their algorithms or systems properties. Google have published significant papers that leave out the key algorithms on how parts of their systems work (e.g., In Borg how does the scheduler synchronize the cluster state with the replicas? In Spanner, what are the properties of the underlying storage system, Collosus?). I would love to see all distributed systems papers be able to reproduce everything by just downloading a file that allows the paper's results to be reproduced. We can do it with the help of either virtualization or containers and software configuration frameworks (to parameterize the experiments). We can specify the hardware and network programmatically, we can install the software automatically, we can parameterize system software (with Chef or Puppet attributes), and run the experiments reproducibly. Without huge investment and cloud computing or Docker, we could specify systems that have reproducible hardware/network/software. The first attempt I've seen at this is www.karamel.io that allows you to design reproducible experiments by using JClouds to spawn VMs and setup the virtualized hardware, then orhcestrating Chef to install software that can be parameterized. I hope to see more of this line of platform gain adoption in our community. It would be a boon for research.

dekhn 11y ago
Lots of people have proposed and implemented VM-based reproducibility environments. 1) it's actually harder to get reproducible computational experiments than they expected. For example, you can run same VM on a different processor and get different results, which makes bitwise reproduction hard, and statistical tests for nonbitwise equality are harder 2) developing and maintaining the VMs and the environment takes a fair amount of effort from skilled people 3) the resulting improvements to science don't seem to exceed the cost thresholds implied by #1 and #2, and nobody's volunteering their time. My conclusion: good idea, but probably not critically necessary.
- jamesblonde 11y ago
  That's good feedback. Although I'm considering mostly distributed systems, where full reproducability is not possible. For single-threaded, deterministic programs (discrete event simulations) where performance is not a measurement point, you can do it with VMs. The nicest thing would be to be able to parameterize experiments. Not just re-run them, but change the parameters for different runs.

chrisdbaldwin 11y ago

When I think about unsung software heroes I think of Rob Scharein[1]. He is the creator or KnotPlot[2] which has enabled topological research to flourish. He recieves regular acknowledgement among topology researchers, and he's an awesome guy. Do yourself a favor and play with some knots. [1] www.hypnagogic.net/rob/ [2] www.knotplot.com

jcoffland 11y ago

I've been developing research software for +10 years and I experience this all the time. As one example, over a decade ago I wrote a biological simulation program called CompuCell3D it was the successor to CompuCell, a 2D simulator of cells as cellular automata. Since then many papers have been written based on research utilising this software. Not only have I not been credited in any of these publications but my name has been removed from the software, the website never mentions my original contribution and the current maintainers of said software are not responding to my emails. Granted this code has changed a lot over the years but there are still large parts of the code which are still verbatim from my original software. The researchers in this project see it as totally irrelevant that I wrote the original code because they value code creation MUCH less than their research. This particular case is especially egregious but it is a good anecdotal example of the problem. Sometimes code is not as significant or as breakthrough as raw research. It really matters what kind of coding you are talking about. You don't necessarily credit the construction crew when dedicating a new building but you do credit the architect. Coders are under valued in today's research environment.

dekhn 11y ago
What role did you have when you wrote the code? If, for example, you were a full time employee, grad student, or postdoc at a university, the copyright belongs to the university (not true for undergrads!) The website isn't required to list your contribution. That would be a courtesy. Also, if you have a problem with this where you think it's important to change the website or get another form of credit, your lawyer should be talking to your ex-employers lawyer. That should have been clear when the current maintainers failed to respond to your email.
dekhn 11y ago
The paper for citing CompuCell3D is Multi-Scale Modeling of Tissues Using CompuCell3D – M. Swat, Gilberto L. Thomas, Julio M. Belmonte, A. Shirinifard, D.Hmeljak, J. A. Glazier, Computational Methods in Cell Biology, Methods in Cell Biology 110: 325-366 (2012) I can't see that article online without paying, but I am certain it cites CompuCell, a Multi-Model Framework For Simulation of Morphogenesis – J. A. Izaguirre, R. Chaturvedi, C. Huang, T. Cickovski, J. Coffland, G. Thomas, G. Forgacs, M. Alber, G. Hentschel, S. A. Newman, and J. A. Glazier, Bioinformatics 20: 1129-1137 (2004) that original article includes your name on the author list. So your statement "I have not been credited in any of these publications" isn't really correct; your contribution is credited indirectly through citations. I don't see any problem with this; I doubt papers citing CompuCell3D have any text like "We acknowledge the efforts of <so and so>".

pdm55 11y ago

My unsung software hero is Dr. Jeffrey Lewis Fox (deceased in 1999 at age 51), Associate Professor, Department of Pharmaceutics and Pharmaceutical Chemistry, University of Utah http://pharmacy.utah.edu/pharmaceutics/news/1999.html#fox http://pharmacy.utah.edu/pharmaceutics/news/1999.html#fox He wrote MINSQ (now called Scientist), the only piece of software I have ever fallen in love with. He sold the first version for $10 (if I recall correctly) in 1990. It was so user-friendly that I was using it within 10 mins to solve a system of differential equations that were part of my PhD research. (Unfortunately, I don't find the more recent versions, "updated" by others, anywhere near as user-friendly.) He called his company, at the University of Utah, MicroMath. I assume he used the algorithms from "Numerical Recipes" to write MINSQ - I know that was the alternative facing me till his excellent software appeared. I really think the U of Utah should have kept the rights to the software and maintained it in his honour.

abawany 11y ago

I will use this opportunity to mention Maxima (https://en.wikipedia.org/wiki/Maxima_%28software%29 https://en.wikipedia.org/wiki/Maxima_%28software%29) that was developed by one of the best math professors that I have had the privilege to learn from: Bill Schelter. It was forked from Macsyma and released under the GPL and still appears to be actively maintained.