6 ms·

In my field (ecology and evolutionary biology) there's a small but concerted push to get people to publish "executable papers" where all code and data is availa

In my field (ecology and evolutionary biology) there's a small but concerted push to get people to publish "executable papers" where all code and data is available via GitHub, on an iPython notebook, etc. so that all figures and test cases can easily be reproduced by reviewers and researchers. If you're publishing research, it shouldn't be the reader's job to parse your paper and reimplement the research you describe; it should be on you to make your results replicable, and I think this is a standard we need to insist upon. I can't count how many papers I've read lately that were missing either crucial methods or underlying data so that replication was impossible. edit: Here's an example: https://github.com/weecology/white-etal-2012-ecology https://github.com/weecology/white-etal-2012-ecology Run portions of the analysis pipeline: Empirical analyses: python mete_sads.py ./data/ empir Simulation analyses: python mete_sads.py ./data/ sims Figures: python mete_sads.py ./data/ figs

mej10 14y ago
This should be the default for all code and data produced by publicly funded research.
- 46Bit 14y ago
  Yes, although I don't think this is just a question of open access. It's a question of research verification - and expected of all published research.
vidarh 14y ago
That's great... When I did my MSc (my thesis was on reducing error rates in OCR), of the dozens of computer science papers I surveyed, there were perhaps 2-3 where I didn't have to do extensive "reverse engineering" of the results of the papers to figure out a whole lot of unstated assumptions when I wanted to test their algorithms. It was a tremendous waste of time... Especially when pretty much all of these papers described results that meant the authors had already implemented their algorithms in executable form. But almost none of them made the code available in any form whatsoever.
- kitsune_ 14y ago
  I don't understand how academic authors get away with this. Papers that I implemented or tried to implement in the past were often sorely lacking in details. Your point about unstated assumptions is absolutely true. Some papers are nothing more than glorified abstracts. Free access to PUBLICLY funded research should be the default.
  - dhimes 14y ago
    Free access to PUBLICLY funded research should be the default I agree. But some research is privately funded.
  - esrauch 14y ago
    I think the issue is that the current incentive system make it generally much more favorable to not publish actual source. -A trivial bug in your program can go a long way to discrediting you if someone wants to. -If your methods get inlined into a popular library, the number of people who cite your work will drop to 0. Popular libraries (for Machine Learning at least) typically have a set of authors who will be cited, and if you implement some new method or improvement in that library, they will get the credit and you won't. Since being cited is the most common measure of your net worth, having your work be accessible only with your name attached is significantly better for your career than having your work be maximally accessible.
  - blablabla123 14y ago
    When I did my diploma thesis my mentoring professor explicitly told me to only include short excerpts of the code, if any. Okay, it was Physics, but still. Having read the thesis 1 year after I handed it in, I realized the code fragments are mostly usely/trivial for the reader. Either you publish the whole code or none of it.
  - michael_h 14y ago
    I've had more than a few situations where the author does release the code...and it doesn't match the paper. The equations are different, restrictions are tighter, etc. It's the single most frustrating aspect of my job (excepting autotools). Releasing the code is another way that people can attack your conclusions, so maybe people are reluctant to do it?
fusiongyro 14y ago
I agree completely, but as the resident Dijkstra-head I am compelled to furnish the following quote: "In the good old days physicists repeated each other's experiments, just to be sure. Today they stick to FORTRAN, so that they can share each other's programs, bugs included." -- EWD 498 ("How do we tell truths that might hurt?")
- stingraycharles 14y ago
  And what's wrong with that? That only makes it easier to spot the errors in their research.
  - fusiongyro 14y ago
    If someone goes looking through the code, it's a terrific improvement. But I work with astronomers and I haven't seen that tendency here. I expect this is changing across all science, but it's probably happening faster in biology.
    - czr80 14y ago
      Two "clean room" implementations that achieve the same result is much, much stronger evidence than any degree of code reviews on one piece of code (or reviews of one experiment). Independent reproducibility is a cornerstone of science.
      - fusiongyro 14y ago
        I absolutely agree. I also have about zero faith in untrained scientists ability to perform a clean room implementation. (That's neither here nor there though.)
    - stingraycharles 14y ago
      Good point, I hadn't considered that. But isn't the source code a part of the methodology used? As in, if someone doesn't review the source code, then he's skipping a certain part of the methodology?
tellarin 14y ago
The Reproducible Research movement and the like have been trying to gain ground in many different areas since at least 2009 [1]. Too bad its adoption hasn't grown faster. Especially with all the recent focus on "big data", applied machine learning, and computational science papers. It's hard to actually measure the quality and contributions of most of them. Not having an accepted method to cite and share datasets is also part of the problem. 1- http://www.computer.org/csdl/mags/cs/2009/01/mcs2009010005.pdf http://www.computer.org/csdl/mags/cs/2009/01/mcs2009010005.p... [PDF]
- bendmorris 14y ago
  http://figshare.com/ http://figshare.com/ is gaining traction as a place to deposit data and figures, and it provides DOIs for citation.
jgrahamc 14y ago
This is what I (with co-authors) argued for in Nature last year: http://blog.jgc.org/2012/02/case-for-open-computer-programs.html http://blog.jgc.org/2012/02/case-for-open-computer-programs....
jmilloy 14y ago
I currently work in method development for proteomics, and I think about this all the time. Of course methods must be reproducible, or the paper largely useless. But lately I've come to disagree that source code should be required. If the research relies on a piece of "in-house" code, which isn't described, then that's a problem. But the OP is about research where the algorithm is the advance in the field. In a way, the algorithm is the result and not the method (even though the algorithm will probably be applied to a sample problem and the results thereof validated). Now, if I develop a new method at the bench, you are expected to do the method at your own bench if you want to apply it to your research. This often starts as a pilot experiment just trying out the method, which can take weeks to months, before you integrate it fully into your methods. I don't have to provide a kit which consistently implements the method along with the paper. Certainly once the method gains some traction, an outside company may begin selling kits. Until then, it absolutely should be "the reader's job to ... reimplement the research" - that's an essential part of the process. Similarly, an algorithmic method can be reproducible even if code isn't provided. Just like with bleeding edge wet-lab methods, if you want to use bleeding edge algorithms you will need to be able to code. You will need to be able to read a description of the algorithm and implement it accurately. The publicly funded result is the algorithm as an idea, and providing that the review process works, you're getting that idea. Later on, if the algorithm gains any traction, someone will implement it in a usable, robust package and more labs will be able to use it. Finally, it takes significant time and experience to produce (and maintain) code that others can use and expect to work most of the time. That's a waste of time/money for a research lab, and an inefficient use of public funds. If you can't code yourself, leave code production to those who do it well and don't whine that you can't just plug-in whatever hot new method just came out without any effort or proper understanding.
- FrojoS 14y ago
  > Finally, it takes significant time and experience to produce (and maintain) code that others can use and expect to work most of the time. That's a waste of time/money for a research lab, and an inefficient use of public funds. I haven't seen people asking for maintained, (re)usable code. We just want the crappy code that was used to produce the results. There is even an appropriate license, the Community Research and Academic Programming License (CRAPL). [1] http://matt.might.net/articles/crapl/ http://matt.might.net/articles/crapl/
adolgert 14y ago
Reproduction of work sometimes goes by the name "provenance." 1. Record what you want to do. 2. Record how, when, where, by whom it was done. 3. Include ability to run it again. (Exact reproduction is understood to be impossible given everything that can vary in the hardware, OS, and software.) There is an encoding format for provenance, called Open Provenance Model, that can function as a guideline for how to record actions faithfully. A good example is VisTrails, a workflow application. If you use it to make an image, then you can click on that figure in a PDF, and the URL leads to an online record of what made the image, which downloads to your local machine and runs. You can pick up where the author left off (software, data, internet permitting). Running every program under such a workflow is cumbersome or impossible, though, but it's work in the right direction.
dakimov 14y ago
It is generally a good idea, but I am not sure how efficient it would be, because scientists generally suck at programming, and wandering through a lamer's messy code is not that better than wandering through a poorly formulated scientific paper, very often it is even worse. Such a code written by nonprofessional programmers usually has very bad API design, very bad naming, poor to unacceptably poor performance, is often written in a high-level non-efficient language like Python, so you cannot just insert it into your program on C++. The scientists should better develop a standard for scientific papers, because they do not even name them consistently. Damn, people, the most advanced human knowledge is a collection of PDFs that are not good for anything except printing and forgetting. There is a program 'Mendeley', and all it does is it finds PDFs where you point it at, and renames it depending on how relatively successful it has parsed the article's author and the title. And it does suck, because it cannot even parse the author and title in 100% of cases. Will there be anything better in the XXI century for knowledge sharing and scientific collaboration except as writing a bunch of disconnected PDFs? At leats hyperlinks, for god's sake! Semantic web? Also, the traditional mathematical notation where any entity is marked with a single letter from Latin, Greek or another alphabet is exactly as very poor programming style where all your identifiers are like a, b, c, a1, a2, i3. For me, this makes parsing the scientific papers much harder.
- deleted 14y ago
  [deleted]
- kanzure 14y ago
  the most advanced human knowledge is a collection of PDFs that are not good for anything except printing and forgetting Yep, and we have no backup of science. If science is so important, why is there no backup of all these files? How important it must be that nobody is bothering- despite the moral implications- to make a complete backup? And why can't scientists access papers?
  - Kliment 14y ago
    The last person who tried is now dead after being threatened to rot in jail for life by his state.

mej10 14y ago

This should be the default for all code and data produced by publicly funded research.

46Bit 14y ago
Yes, although I don't think this is just a question of open access. It's a question of research verification - and expected of all published research.

vidarh 14y ago

That's great... When I did my MSc (my thesis was on reducing error rates in OCR), of the dozens of computer science papers I surveyed, there were perhaps 2-3 where I didn't have to do extensive "reverse engineering" of the results of the papers to figure out a whole lot of unstated assumptions when I wanted to test their algorithms. It was a tremendous waste of time... Especially when pretty much all of these papers described results that meant the authors had already implemented their algorithms in executable form. But almost none of them made the code available in any form whatsoever.

kitsune_ 14y ago
I don't understand how academic authors get away with this. Papers that I implemented or tried to implement in the past were often sorely lacking in details. Your point about unstated assumptions is absolutely true. Some papers are nothing more than glorified abstracts. Free access to PUBLICLY funded research should be the default.
- dhimes 14y ago
  Free access to PUBLICLY funded research should be the default I agree. But some research is privately funded.
- esrauch 14y ago
  I think the issue is that the current incentive system make it generally much more favorable to not publish actual source. -A trivial bug in your program can go a long way to discrediting you if someone wants to. -If your methods get inlined into a popular library, the number of people who cite your work will drop to 0. Popular libraries (for Machine Learning at least) typically have a set of authors who will be cited, and if you implement some new method or improvement in that library, they will get the credit and you won't. Since being cited is the most common measure of your net worth, having your work be accessible only with your name attached is significantly better for your career than having your work be maximally accessible.
- blablabla123 14y ago
  When I did my diploma thesis my mentoring professor explicitly told me to only include short excerpts of the code, if any. Okay, it was Physics, but still. Having read the thesis 1 year after I handed it in, I realized the code fragments are mostly usely/trivial for the reader. Either you publish the whole code or none of it.
- michael_h 14y ago
  I've had more than a few situations where the author does release the code...and it doesn't match the paper. The equations are different, restrictions are tighter, etc. It's the single most frustrating aspect of my job (excepting autotools). Releasing the code is another way that people can attack your conclusions, so maybe people are reluctant to do it?

fusiongyro 14y ago

I agree completely, but as the resident Dijkstra-head I am compelled to furnish the following quote: "In the good old days physicists repeated each other's experiments, just to be sure. Today they stick to FORTRAN, so that they can share each other's programs, bugs included." -- EWD 498 ("How do we tell truths that might hurt?")

stingraycharles 14y ago
And what's wrong with that? That only makes it easier to spot the errors in their research.
- fusiongyro 14y ago
  If someone goes looking through the code, it's a terrific improvement. But I work with astronomers and I haven't seen that tendency here. I expect this is changing across all science, but it's probably happening faster in biology.
  - czr80 14y ago
    Two "clean room" implementations that achieve the same result is much, much stronger evidence than any degree of code reviews on one piece of code (or reviews of one experiment). Independent reproducibility is a cornerstone of science.
    - fusiongyro 14y ago
      I absolutely agree. I also have about zero faith in untrained scientists ability to perform a clean room implementation. (That's neither here nor there though.)
  - stingraycharles 14y ago
    Good point, I hadn't considered that. But isn't the source code a part of the methodology used? As in, if someone doesn't review the source code, then he's skipping a certain part of the methodology?

tellarin 14y ago

The Reproducible Research movement and the like have been trying to gain ground in many different areas since at least 2009 [1]. Too bad its adoption hasn't grown faster. Especially with all the recent focus on "big data", applied machine learning, and computational science papers. It's hard to actually measure the quality and contributions of most of them. Not having an accepted method to cite and share datasets is also part of the problem. 1- http://www.computer.org/csdl/mags/cs/2009/01/mcs2009010005.pdf http://www.computer.org/csdl/mags/cs/2009/01/mcs2009010005.p... [PDF]

bendmorris 14y ago
http://figshare.com/ http://figshare.com/ is gaining traction as a place to deposit data and figures, and it provides DOIs for citation.

jgrahamc 14y ago

This is what I (with co-authors) argued for in Nature last year: http://blog.jgc.org/2012/02/case-for-open-computer-programs.html http://blog.jgc.org/2012/02/case-for-open-computer-programs....

jmilloy 14y ago

I currently work in method development for proteomics, and I think about this all the time. Of course methods must be reproducible, or the paper largely useless. But lately I've come to disagree that source code should be required. If the research relies on a piece of "in-house" code, which isn't described, then that's a problem. But the OP is about research where the algorithm is the advance in the field. In a way, the algorithm is the result and not the method (even though the algorithm will probably be applied to a sample problem and the results thereof validated). Now, if I develop a new method at the bench, you are expected to do the method at your own bench if you want to apply it to your research. This often starts as a pilot experiment just trying out the method, which can take weeks to months, before you integrate it fully into your methods. I don't have to provide a kit which consistently implements the method along with the paper. Certainly once the method gains some traction, an outside company may begin selling kits. Until then, it absolutely should be "the reader's job to ... reimplement the research" - that's an essential part of the process. Similarly, an algorithmic method can be reproducible even if code isn't provided. Just like with bleeding edge wet-lab methods, if you want to use bleeding edge algorithms you will need to be able to code. You will need to be able to read a description of the algorithm and implement it accurately. The publicly funded result is the algorithm as an idea, and providing that the review process works, you're getting that idea. Later on, if the algorithm gains any traction, someone will implement it in a usable, robust package and more labs will be able to use it. Finally, it takes significant time and experience to produce (and maintain) code that others can use and expect to work most of the time. That's a waste of time/money for a research lab, and an inefficient use of public funds. If you can't code yourself, leave code production to those who do it well and don't whine that you can't just plug-in whatever hot new method just came out without any effort or proper understanding.

FrojoS 14y ago
> Finally, it takes significant time and experience to produce (and maintain) code that others can use and expect to work most of the time. That's a waste of time/money for a research lab, and an inefficient use of public funds. I haven't seen people asking for maintained, (re)usable code. We just want the crappy code that was used to produce the results. There is even an appropriate license, the Community Research and Academic Programming License (CRAPL). [1] http://matt.might.net/articles/crapl/ http://matt.might.net/articles/crapl/

adolgert 14y ago

Reproduction of work sometimes goes by the name "provenance." 1. Record what you want to do. 2. Record how, when, where, by whom it was done. 3. Include ability to run it again. (Exact reproduction is understood to be impossible given everything that can vary in the hardware, OS, and software.) There is an encoding format for provenance, called Open Provenance Model, that can function as a guideline for how to record actions faithfully. A good example is VisTrails, a workflow application. If you use it to make an image, then you can click on that figure in a PDF, and the URL leads to an online record of what made the image, which downloads to your local machine and runs. You can pick up where the author left off (software, data, internet permitting). Running every program under such a workflow is cumbersome or impossible, though, but it's work in the right direction.

dakimov 14y ago

It is generally a good idea, but I am not sure how efficient it would be, because scientists generally suck at programming, and wandering through a lamer's messy code is not that better than wandering through a poorly formulated scientific paper, very often it is even worse. Such a code written by nonprofessional programmers usually has very bad API design, very bad naming, poor to unacceptably poor performance, is often written in a high-level non-efficient language like Python, so you cannot just insert it into your program on C++. The scientists should better develop a standard for scientific papers, because they do not even name them consistently. Damn, people, the most advanced human knowledge is a collection of PDFs that are not good for anything except printing and forgetting. There is a program 'Mendeley', and all it does is it finds PDFs where you point it at, and renames it depending on how relatively successful it has parsed the article's author and the title. And it does suck, because it cannot even parse the author and title in 100% of cases. Will there be anything better in the XXI century for knowledge sharing and scientific collaboration except as writing a bunch of disconnected PDFs? At leats hyperlinks, for god's sake! Semantic web? Also, the traditional mathematical notation where any entity is marked with a single letter from Latin, Greek or another alphabet is exactly as very poor programming style where all your identifiers are like a, b, c, a1, a2, i3. For me, this makes parsing the scientific papers much harder.

deleted 14y ago
[deleted]
kanzure 14y ago
the most advanced human knowledge is a collection of PDFs that are not good for anything except printing and forgetting Yep, and we have no backup of science. If science is so important, why is there no backup of all these files? How important it must be that nobody is bothering- despite the moral implications- to make a complete backup? And why can't scientists access papers?
- Kliment 14y ago
  The last person who tried is now dead after being threatened to rot in jail for life by his state.