4 ms·

I work in a fairly large scientific collaboration. Our data sets are on the scale of a few hundred terabytes per year. So, there's a lot of software to convert

I work in a fairly large scientific collaboration. Our data sets are on the scale of a few hundred terabytes per year. So, there's a lot of software to convert from the bits that stream out of the detectors to something that end users (grad students) can analyze. We have a software group devoted to maintaining the core code. Our code has a nightly autobuild for development code and libraries are released every couple months. The code is based on the ROOT package (about one million lines of code) maintained by folks at CERN and is well established within the community and elsewhere. Our libraries probably come to about a million more lines of code, which is maintained scrupulously by our collaboration. Now neither of these sets of code actually do any analysis, they just make it so that the data is usable. In my analysis, there is probably ~10k lines of code to do the analysis and make the figures. In total, there's probably 500k or more lines of analysis code that is not officially maintained. That doesn't include all of the code loaded onto FPGAs and custom chips reading out the detectors or the code in the trigger system. Nor does it include code written into various simulators used to determine "expected" response of the detectors. So for peer review to look at code at the level suggested, people would have to look over literally millions of lines of code. To even be able to make it run, a user would have to set up an environment that would take between a few days and a week. Instead, when I go to publish something, I lay out both my method and how I verified that it works. Reviewers then check that my method is sound and that my verification looks right. They have to trust that I implemented the method as I said I did. In my collaboration, there is actually an internal review process that verifies code runs and for which a much longer private note must be written to explain all of the details of the analysis. However, there is still a high level of trust that everything was implemented as stated. This review does not qualify as "peer" review because it's conducted by people who will be listed as authors. While I agree that more review is better than less, I hope this comment illustrates why software peer review is not a reasonable expectation.

lutorm 17y ago
The code I wrote for my simulations is not that big, but it's certainly big enough to be complicated and buggy. And it's GPL and available on my website. And while I certainly don't expect any of my referees to go over the code line-by-line, one of the rationales I had for GPLing the code was to make it possible for someone to replicate my study. One of the basic tenets about publishing research is that articles should contain all the information needed to reproduce the results. This is not possible for simulation papers unless the source code is public (or maybe a binary is provided, but that has many pitfalls by itself). Results generated with proprietary codes could just as well be hokus-pokus in my mind. It's not clear how it adds to the field unless others can reproduce and build on the results. And with reasonably complicated codes, it's not realistic to think that someone will implement their own version (though that is of course the absolute best thing). So while I don't necessarily think scientific software should be peer reviewed, I'm very hesitant about results obtained with proprietary software.
- spamizbad 17y ago
  What's your view on the mountains of scientific code that depend on proprietary math kernels and in some cases proprietary compiler platforms?
  - lutorm 17y ago
    As you may guess, I'm unhappy about it. In astronomy, almost everyone uses IDL, which is a proprietary, closed-source language. (It also sucks, but that's beside the point.) I think this is bad, because it locks practically the entire research field into a closed platform. (Though GDL is working to be a replacement. I don't know how conformant it is, though.) Still, I think this is less of a problem than the high-level scientific code. Most of the low-level functionality is pretty simple and (hopefully) much better tested than most scientific codes.
- aristoxenus 17y ago
  I'd just throw in that if someone uses your code to try to reproduce what you've done, they may actually just be reproducing a bug -- or in the sinister case, reproducing a faked result. So a more general point in this whole discussion would be that making the code available isn't always as important as making the mathematics or other theory behind the code available, and "reproduction" becomes "reimplementation", or some other corroborating element of the process as described by timr a few comments down (http://news.ycombinator.com/item?id=968456 http://news.ycombinator.com/item?id=968456). (Of course for large software endeavors that isn't practical either. So mattheww's point about trust unfortunately resonates.) As for the proprietary issue, I think sometimes it might actually help reproducibility: if you can just drop a few lines of script into Matlab and examine them, it's potentially a much smaller task with a lot more battle-worn, common-ground code factored out than examining or re-implementing a full library of open code.
joe_the_user 17y ago
The parent hopefully reflects the workings of large, established scientific organizations. I did only a bit of work for a very small shop and where I worked coding standards were completely non-existent. I think that using Matlab is a good way destroy the ability of anyone else to review one's work because of Matlab's nature as a closed, ad-hoc environment. Some scientists could learn a lot from software best practices.

lutorm 17y ago

The code I wrote for my simulations is not that big, but it's certainly big enough to be complicated and buggy. And it's GPL and available on my website. And while I certainly don't expect any of my referees to go over the code line-by-line, one of the rationales I had for GPLing the code was to make it possible for someone to replicate my study. One of the basic tenets about publishing research is that articles should contain all the information needed to reproduce the results. This is not possible for simulation papers unless the source code is public (or maybe a binary is provided, but that has many pitfalls by itself). Results generated with proprietary codes could just as well be hokus-pokus in my mind. It's not clear how it adds to the field unless others can reproduce and build on the results. And with reasonably complicated codes, it's not realistic to think that someone will implement their own version (though that is of course the absolute best thing). So while I don't necessarily think scientific software should be peer reviewed, I'm very hesitant about results obtained with proprietary software.

spamizbad 17y ago
What's your view on the mountains of scientific code that depend on proprietary math kernels and in some cases proprietary compiler platforms?
- lutorm 17y ago
  As you may guess, I'm unhappy about it. In astronomy, almost everyone uses IDL, which is a proprietary, closed-source language. (It also sucks, but that's beside the point.) I think this is bad, because it locks practically the entire research field into a closed platform. (Though GDL is working to be a replacement. I don't know how conformant it is, though.) Still, I think this is less of a problem than the high-level scientific code. Most of the low-level functionality is pretty simple and (hopefully) much better tested than most scientific codes.
aristoxenus 17y ago
I'd just throw in that if someone uses your code to try to reproduce what you've done, they may actually just be reproducing a bug -- or in the sinister case, reproducing a faked result. So a more general point in this whole discussion would be that making the code available isn't always as important as making the mathematics or other theory behind the code available, and "reproduction" becomes "reimplementation", or some other corroborating element of the process as described by timr a few comments down (http://news.ycombinator.com/item?id=968456 http://news.ycombinator.com/item?id=968456). (Of course for large software endeavors that isn't practical either. So mattheww's point about trust unfortunately resonates.) As for the proprietary issue, I think sometimes it might actually help reproducibility: if you can just drop a few lines of script into Matlab and examine them, it's potentially a much smaller task with a lot more battle-worn, common-ground code factored out than examining or re-implementing a full library of open code.

joe_the_user 17y ago

The parent hopefully reflects the workings of large, established scientific organizations. I did only a bit of work for a very small shop and where I worked coding standards were completely non-existent. I think that using Matlab is a good way destroy the ability of anyone else to review one's work because of Matlab's nature as a closed, ad-hoc environment. Some scientists could learn a lot from software best practices.