5 ms·
I currently work in method development for proteomics, and I think about this all the time. Of course methods must be reproducible, or the paper largely useless
by jmilloy 14y ago
I currently work in method development for proteomics, and I think about this all the time. Of course methods must be reproducible, or the paper largely useless. But lately I've come to disagree that source code should be required.
If the research relies on a piece of "in-house" code, which isn't described, then that's a problem. But the OP is about research where the algorithm is the advance in the field. In a way, the algorithm is the result and not the method (even though the algorithm will probably be applied to a sample problem and the results thereof validated).
Now, if I develop a new method at the bench, you are expected to do the method at your own bench if you want to apply it to your research. This often starts as a pilot experiment just trying out the method, which can take weeks to months, before you integrate it fully into your methods. I don't have to provide a kit which consistently implements the method along with the paper. Certainly once the method gains some traction, an outside company may begin selling kits. Until then, it absolutely should be "the reader's job to ... reimplement the research" - that's an essential part of the process.
Similarly, an algorithmic method can be reproducible even if code isn't provided. Just like with bleeding edge wet-lab methods, if you want to use bleeding edge algorithms you will need to be able to code. You will need to be able to read a description of the algorithm and implement it accurately. The publicly funded result is the algorithm as an idea, and providing that the review process works, you're getting that idea. Later on, if the algorithm gains any traction, someone will implement it in a usable, robust package and more labs will be able to use it.
Finally, it takes significant time and experience to produce (and maintain) code that others can use and expect to work most of the time. That's a waste of time/money for a research lab, and an inefficient use of public funds. If you can't code yourself, leave code production to those who do it well and don't whine that you can't just plug-in whatever hot new method just came out without any effort or proper understanding.
- FrojoS 14y ago> Finally, it takes significant time and experience to produce (and maintain) code that others can use and expect to work most of the time. That's a waste of time/money for a research lab, and an inefficient use of public funds. I haven't seen people asking for maintained, (re)usable code. We just want the crappy code that was used to produce the results. There is even an appropriate license, the Community Research and Academic Programming License (CRAPL). [1] http://matt.might.net/articles/crapl/ http://matt.might.net/articles/crapl/
- bostonpete 14y agoI would think most people would be unwilling to make their "crappy" code public because no matter how many disclaimers they provide with it, they will be judged by others on it.
- beagle3 14y agoWhy on earth would anyone trust descriptions that they cannot verify? Trusting without the ability to verify goes against everything scientific. If you think your code is too "crappy" for publication, why do you believe it is bug free enough to produce dependable answers?
- dagw 14y agoRe-running their crappy code and getting the same result they got doesn't really prove or verify anything. Re-implementing the algorithm they describe in the paper and getting the same result (or not) is far more interesting.
- beagle3 14y ago> Re-running their crappy code and getting the same result they got doesn't really prove or verify anything. Yes it does. Very often, the data selected for publication is cherry-picked. Running the same crappy code on a more complete data set, (or alternatively, on a partial data set) would give a very quick indication of the robustness of the results - and unlike re-implementing, might be doable in a day rather than months of effort. Furthermore, when you actually re-implement (if you do), it is extremely helpful to compare intermediate results, which is impossible unless you have the original everything. > Re-implementing the algorithm they describe in the paper and getting the same result (or not) is far more interesting. Yes, but very rarely done in fields that are not CS or EE (and not very common in these either). Usually, results are just taken as gospel. Also, there is a ridiculous amount of negligence (and even fraud) in publications. just running the crappy code, seeing the results, and having a cursory look at the code and data would reveal a lot of that.
- 14y ago
- tokipin 14y agoi would imagine that the algorithm would gain a lot more traction if it actually came with an implementation
- bendmorris 14y ago>If you can't code yourself, leave code production to those who do it well and don't whine that you can't just plug-in whatever hot new method just came out without any effort or proper understanding. This is missing the point. I can code, and I'm not promoting code sharing to more easily use other people's research. What I want is to be able to verify that the research I read is accurate, and there's simply not enough time available to reproduce everything I read from scratch. Reproducibility is important; I shouldn't have to just accept the authors' word that their results are exactly as described. Independent verification should be happening at the review stage at the very least, and authors should be required to make that possible if they want to publish.