6 ms·
I wrote to the lead author asking them to release the source code of the software that they used for the data analysis. I have yet to receive a reply, but it'
by jgrahamc 15y ago
I wrote to the lead author asking them to release the source code of the software that they used for the data analysis. I have yet to receive a reply, but it's pretty important to eliminate a programming error from this.
For example, there were small errors in the Met Office's climate change software that I detected. The scientific papers were correct, but the translation into code was not. This could have happened here and it would be better if they released the code.
http://blog.jgc.org/2010/02/something-odd-in-crutem3-station-errors.html http://blog.jgc.org/2010/02/something-odd-in-crutem3-station...
- JonnieCache 15y agoI find it very strange that the source code is not routinely released in these circumstances. What possible reason is there for not doing so? The code and the ideas within have no direct commercial value, surely? At the very least it should be as available as the papers themselves.
- jamesrom 15y agoThe only advantage I can see is that when the theory, formulas and algorithms are published, not the specific implementation itself, it forces people who want to who want to verify your work to write their own implementation. It kind of makes sure that a bug in your own implementation doesn't pollute the results of those trying to verify your work.
- jonhendry 15y agoExactly. It shouldn't matter if Lab A implemented the algorithm in Matlab and Lab B implemented the algorithm in NumPy. Take the data, apply the described algorithms, get the same result. If Lab B doesn't get the same result, and Lab B has confidence in their implementation, then there's probably something funny with Lab A's code or data. Reusing Lab A's code on Lab A's data just reproduces Lab A's errors, if any.
- JonnieCache 15y agoThis is very true. However if I can just look at Lab A's code and spot an error in it, that will allow me to discount their results without having to redo their experiment, potentially saving years of work and millions of dollars. Maybe that's what researchers are really afraid of?
- jonhendry 15y ago" discount their results without having to redo their experiment, potentially saving years of work and millions of dollars." Or you could just analyze their data using their described methods and your own implementation.
- alextp 15y agoThe problem is more complex. Some errors will damage the scientific conclusions, others will show up as fractions of a percentage point inside a much larger confidence interval, and will ultimately not matter much. In machine learning research, for example, most evaluations consist of running a new program on soem data, getting results back, and from these results computing some aggregate measure of performance. A bug on the code that computes this measure of performance is _really bad_ and can invalidate all your conclusions. If that code is right, however, a but on the code that trains your model is completely meaningless, because as long as your results are good you can argue that you actually meant to write a paper about the model actually implemented rather than the model you were supposed to implement. I'm sure other scientific areas have similar distinctions, and a naive code reader might fail to notice that a bug is harmless (and there's also the fact that scientific code carries within it a lot of assumptions about the data which, if broken, can be buggy, but are not broken by real data).
- hugh3 15y agoWhat possible reason is there for not doing so? The code and the ideas within have no direct commercial value, surely? Laziness, lack of time, and the fact that most scientific code is not suitable for public consumption. For instance, my code includes error messages such as "What the fuck????" and "This never happens" which I'd need to take out in order to prepare it for publication. In addition, there's the assumption that source code is a pretty trivial implementation detail; we publish the algorithm but not the details of the implementation, just as the experimentalist tells us what he built but doesn't tell us what brand of screwdriver he used to build it.
- JonnieCache 15y agoThe thing is though, as I'm sure you're aware, the algorithm in abstract is utterly irrelevant. The algorithm as released is never actually executed, it is the implementation that gets executed. One "trivial" detail in the implementation, one tiny little floating point rounding error, can throw the entire thing. We all know this. So why is the scientific community so unwilling to face this elephant in the room? And why is nobody else confronting them about it? Of course there's jgc, with his admirable record of confronting people about things, but that's the only example I can think of. You would've thought the "Al Gore made up AGW to devalue my exxon shares" angst-brigade would've jumped on this, but they seem more concerned with digging through departmental email gossip.
- jonhendry 15y ago"The algorithm as released is never actually executed, it is the implementation that gets executed." If you use a different implementation of the algorithm, on the same data, and get a different result, you will find that there was an error. That's a lot easier than combing through someone else's code base. A simple example: If Lab A says they took the square root of 16, you don't need their source code to know there's a problem, if their result is 5 and your result is 4. If you use their code, you might not notice that it's borked.
- rbanffy 15y ago
- jonhendry 15y agoBiology researchers don't get to reuse the other guy's mice, why should they reuse their code? As long as the paper describes the algorithms used, anyone should be able to duplicate the work using their own preferred implementation. Also, a lot of the code may well be specific to the apparatus used, which may be unique to a lab, consisting of custom-built hardware.
- ajuc 15y agoMouse can't be perfectly copied. Code can. If biologists could copy mice from other experiment to give them their own drugs and compare results, they would surely do this along with using their own mice. The more ways to compare experiments, the more sure we will be about results. I understand cleaning up code is a lot of work, so don't do it, just release it as it is. I know it will be ugly, will include obscenity, etc. If somebody need it, it will do the rest of work.
- jonhendry 15y ago"The more ways to compare experiments, the more sure we will be about results." No. The more exactly you duplicate the original experiment, the more likely you duplicate any confounding issues. If someone observes something through a telescope, it's best to verify it through a different telescope somewhere else, to exclude the possibility of an optical quirk of the first telescope.
- ajuc 15y agoSome people will run the same code on the same data. They will get the same results, but so what. Some other people will run the same code on different data (maybe from simulation, or when checking their own theory). They will get some other results, or the same. Some people will recreate experiment from scratch, and get code B and data B. Some people will run code A on data B. Some people will run code B on data A. Now we can't do that. The code is already written, it's waste to hide it, it's not like publishing code prohibit other people from writing their own code. That's how you check for errors - by changing as little as possible and observing results.
- hugh3 15y agoIf I were involved in this project, I'd be sitting around saying "please don't let it be my fault, please don't let it be my fault". Somewhere, somebody screwed up... and if the problem is in my code then I want to find it and correct it myself, rather than have it found and corrected by some random stranger who emailed me, and will then post the bug all over the internet so the world can point and laugh at the exact line of my source code that sent the world on a wild goose chase for FTL neutrinos. So I don't blame this guy for not sending you his source code.
- jgrahamc 15y agoI don't blame him either. Random British guy writes to you in French and ask for your source code. Sure I'd be skeptical too (although personally I err on the side of releasing all my code). The problem is that suppose it is a bug and suppose the code is being kept behind so they can find the bug themselves. A huge amount of time and money is wasted whereas a programmer could look at the code and compare with the paper pretty quickly. Also in the case of high-energy physics its not exactly easy to duplicate experiments because of the eye-watering cost of the equipment.
- jonhendry 15y ago"A huge amount of time and money is wasted whereas a programmer could look at the code and compare with the paper pretty quickly." Maybe. It might be faster and easier to analyze the data using their described methods and your own code and see if the results are the same.
- mattheww 15y agoAsking for the source code of the OPERA analysis is similar in scope to asking for the source code to gmail. Unless you made a very specific request and explained how you would be qualified to review it, I doubt they are taking you very seriously. Furthermore, I doubt anyone at OPERA is in a position to even fulfill your request. The minimum number of separate systems with exclusive sets of code is three, and probably more than ten. However, physicists do make extensive use of unit tests, so that may be some consolation.
- tripzilch 15y ago> However, physicists do make extensive use of unit tests, so that may be some consolation. Ehm, well some do. And those that do, endlessly complain about most of them that don't. Fact of the matter is, most physicists aren't trained in proper software engineering methods. Quite a few, but not nearly a majority are very interested in computers and computing so they may pick up a thing or two or actually bite their teeth in doing things the right way, and then there's most of them who love Physics for being Physics and Science for Science and see the computer as just another tool next to their pencil and paper, and just "get things done" and screwed be the next PHD that has to pick up on their research. At least, that's what I hear from a friend doing a PHD involving analysing huge datasets coming from apparatus measuring high energy cosmic particles. Of course the LHC is a much more prestigious project, so you might expect somewhat higher quality, but you can't change the culture. Physicists are generally not Software Engineers.
- jonhendry 15y agoIt's simple. For programmers, code is king. For scientists, data is king. Code can be recreated. Data is irreplaceable, so that's what is important, along with method.
- ajuc 15y agoData depends on the code. You can't write neutrino arrival time to file on hard disk without some code in between the equipment and the file. That code can be wrong, too. Some code has to be trusted, and scientists can't check every assumption every time, so I don't blame them. But why don't they allow others to check the code? It's the opposite of scientific method.
- jonhendry 15y agoPerhaps an electrician should ask to check all the solder joints on all the cables? How is that any different from a random programmer assuming the physicists are incompetent and presuming that what is really necessary is a code review?
- ajuc 15y agoIf an electrician offered to check them for free, in his free time, why should they refuse? EDIT: publishing code don't require dismantling equipment. It's not my fault electrician isn't very good analogy.
- jonhendry 15y agoBecause it would probably require dismantling vast amounts of equipment, and it would probably be detectable in the data anyway. EDIT: True, not a great analogy.
- deleted 15y ago[deleted]