4 ms·
While I strongly believe that scientists should publish their code along with their papers, I do think it has one potential disadvantage. Suppose a scientist w
by splat 16y ago
While I strongly believe that scientists should publish their code along with their papers, I do think it has one potential disadvantage. Suppose a scientist writes a big piece of code to some complicated calculations, but makes a subtle mistake somewhere in it (perhaps some parentheses are nested incorrectly). If the code is not published and another scientist comes along to extend the results from the first paper, the first thing he will do is try to replicate the results of the original. If the code is unpublished, the second scientist will have to write the code from scratch, and in doing so will likely catch the original error. But if the code is published, the second scientist will just use the original code and probably won't catch the error. Consequently, the mistake will take much longer to catch, if it's ever caught at all.
The advantages still outweigh the disadvantages, but it's important to remember that there are always trade-offs.
- code_duck 16y agoI think that to the contrary, that it would allow people to determine whether the original results were due to flawed software by analyzing the code. Subsequent efforts would remain free to create their own implementations from scratch if desired.
- substack 16y agoFurther, open code means that other researchers can come along and write unit tests. It amazes me how much code is still verified manually if at all in the sciences.
- whyenot 16y agoA lot of code in the sciences is used once or only a few times. You crunch the data, write the paper and move on to the next research project. You don't get tenure for writing unit tests.
- _sh 16y agoAnd you don't make friends with salad, i.e. you certainly won't make tenure when your paper is demonstrated as flawed due to errors in crunching the data that would have turned up with testing.
- joe_the_user 16y agoAh, and that brings us back to why many scientists don't publish at all now... As long as not publishing code is standard, there's a incentive to do it since it might open you to career-withering criticism...
- deleted 16y ago[deleted]
- nervechannel 16y agoConsider this though: I have tried to explain (With limited success) to my colleagues in bioinformatics that not unit-testing your code is like using an instrument that hasn't been calibrated, and blindly trusting the results to be right.
- jonhendry 16y agoThey may not unit test it, in the software development sense, but they probably hand-check a new bit of calculation code on a few data points before setting it loose on a whole data set.
- nervechannel 16y agoYes, or sometimes even less rigour than this: run some data and see if the results "look right". This is the standard procedure for when you've developed a complex algorithm to do something novel, and actually working through it by hand would take hours. I should know, this is how I used to do it before I knew better, and as a result one of my published papers has (very slight) numerical errors in.
- splat 16y agoTrue, and that's the definite advantage to publishing code. My point is just that if there's a lot of code and error is obscure, it will take a long time to identify the error and the mistake will be propagated in further research. Most scientists using the code will just skim it, see if it gets the results published in the original paper, and then move on. This has been my experience with my own research anyway.
- gridspy 16y agoIt seems more likely that the scientist re-writing the code would blame their own implementation or some part of the input data. A lot of time might be wasted going down that path too.
- jerf 16y agoI disagree, because your mental model of a large code base that only has one error is absurd. Scientific code is not magically easier to write than real code. Get two random code hacks to write two significantly-sized codebases, say, about the size of a decent web framework, and you won't get one perfect code base and one with a subtle bug that has ramifications down the line. You'll have two code bases so shot through with bugs that they will never reconcile, ever. The quality issues aren't all that different than a web framework, either. Release one or a small number of code bases, and have everybody pound on and improve them, and you might get somewhere. Have everybody write their own code bases from scratch every time and you'll get yourself the scientific equivalent of http://osvdb.org/search?search[vuln_title]=php&search[text_type]=alltext http://osvdb.org/search?search[vuln_title]=php&search[te... . To be honest at this point when I see a news article that says anything about a "computer model" I almost immediately tune out. The exception is that I read for some sign that the model has been verified against the real world; for instance, protein folding models don't bother me for that reason. But this is the exception, not the rule. When it became acceptable "science" to build immense computer models with no particular need to check them against reality before running off and announcing immense world-shattering results I'm not exactly sure, but it was a great loss to humanity.
- deleted 16y ago[deleted]
- wisty 16y agoExcept in the cases where: someone tries to port the code to another language, or they decide to use a different formula / algorithm; and they notice the result. In the real world, most scientists will keep modifying their program until it gets an expected result (either because the program is correct, or because multiple mistakes cancel each-other out, or the standard theory is wrong but they feel obliged to keep debugging until they can prove it to be correct). Sure, there's times when a computational model proves something interesting (unexpected), but then the researchers may have to open source it just to prove they didn't screw it up.
- xiaoma 16y agoIt's just as likely that the second researcher would make a subtle mistake in the process of replicating the results of a paper that didn't include code. And unlike in the scenario in which the first scientist publishes a mistake, it would be impossible for a third party to find or correct it.
- jonhendry 16y agoIf a scientist says their results have a certain p-value, they don't need to release the source code for how they obtained that p-value, for another scientist to be able to tell if the p-value is incorrect given the data. A lot of code is going to be like that: applying known functions that can be identified by name. If a paper says it used an FFT on some data, there are plenty of well-tested FFT implementations that can be used by another researcher to try to replicate the original result. The original researcher's FFT code, if any, isn't really necessary as long as the function, the inputs, and the outputs are well-documented.