5 ms·

I find it very strange that the source code is not routinely released in these circumstances. What possible reason is there for not doing so? The code and the i

I find it very strange that the source code is not routinely released in these circumstances. What possible reason is there for not doing so? The code and the ideas within have no direct commercial value, surely? At the very least it should be as available as the papers themselves.

jamesrom 15y ago
The only advantage I can see is that when the theory, formulas and algorithms are published, not the specific implementation itself, it forces people who want to who want to verify your work to write their own implementation. It kind of makes sure that a bug in your own implementation doesn't pollute the results of those trying to verify your work.
- jonhendry 15y ago
  Exactly. It shouldn't matter if Lab A implemented the algorithm in Matlab and Lab B implemented the algorithm in NumPy. Take the data, apply the described algorithms, get the same result. If Lab B doesn't get the same result, and Lab B has confidence in their implementation, then there's probably something funny with Lab A's code or data. Reusing Lab A's code on Lab A's data just reproduces Lab A's errors, if any.
  - JonnieCache 15y ago
    This is very true. However if I can just look at Lab A's code and spot an error in it, that will allow me to discount their results without having to redo their experiment, potentially saving years of work and millions of dollars. Maybe that's what researchers are really afraid of?
    - jonhendry 15y ago
      " discount their results without having to redo their experiment, potentially saving years of work and millions of dollars." Or you could just analyze their data using their described methods and your own implementation.
    - alextp 15y ago
      The problem is more complex. Some errors will damage the scientific conclusions, others will show up as fractions of a percentage point inside a much larger confidence interval, and will ultimately not matter much. In machine learning research, for example, most evaluations consist of running a new program on soem data, getting results back, and from these results computing some aggregate measure of performance. A bug on the code that computes this measure of performance is _really bad_ and can invalidate all your conclusions. If that code is right, however, a but on the code that trains your model is completely meaningless, because as long as your results are good you can argue that you actually meant to write a paper about the model actually implemented rather than the model you were supposed to implement. I'm sure other scientific areas have similar distinctions, and a naive code reader might fail to notice that a bug is harmless (and there's also the fact that scientific code carries within it a lot of assumptions about the data which, if broken, can be buggy, but are not broken by real data).
hugh3 15y ago
What possible reason is there for not doing so? The code and the ideas within have no direct commercial value, surely? Laziness, lack of time, and the fact that most scientific code is not suitable for public consumption. For instance, my code includes error messages such as "What the fuck????" and "This never happens" which I'd need to take out in order to prepare it for publication. In addition, there's the assumption that source code is a pretty trivial implementation detail; we publish the algorithm but not the details of the implementation, just as the experimentalist tells us what he built but doesn't tell us what brand of screwdriver he used to build it.
- JonnieCache 15y ago
  The thing is though, as I'm sure you're aware, the algorithm in abstract is utterly irrelevant. The algorithm as released is never actually executed, it is the implementation that gets executed. One "trivial" detail in the implementation, one tiny little floating point rounding error, can throw the entire thing. We all know this. So why is the scientific community so unwilling to face this elephant in the room? And why is nobody else confronting them about it? Of course there's jgc, with his admirable record of confronting people about things, but that's the only example I can think of. You would've thought the "Al Gore made up AGW to devalue my exxon shares" angst-brigade would've jumped on this, but they seem more concerned with digging through departmental email gossip.
  - jonhendry 15y ago
    "The algorithm as released is never actually executed, it is the implementation that gets executed." If you use a different implementation of the algorithm, on the same data, and get a different result, you will find that there was an error. That's a lot easier than combing through someone else's code base. A simple example: If Lab A says they took the square root of 16, you don't need their source code to know there's a problem, if their result is 5 and your result is 4. If you use their code, you might not notice that it's borked.
    - rbanffy 15y ago
jonhendry 15y ago
Biology researchers don't get to reuse the other guy's mice, why should they reuse their code? As long as the paper describes the algorithms used, anyone should be able to duplicate the work using their own preferred implementation. Also, a lot of the code may well be specific to the apparatus used, which may be unique to a lab, consisting of custom-built hardware.
- ajuc 15y ago
  Mouse can't be perfectly copied. Code can. If biologists could copy mice from other experiment to give them their own drugs and compare results, they would surely do this along with using their own mice. The more ways to compare experiments, the more sure we will be about results. I understand cleaning up code is a lot of work, so don't do it, just release it as it is. I know it will be ugly, will include obscenity, etc. If somebody need it, it will do the rest of work.
  - jonhendry 15y ago
    "The more ways to compare experiments, the more sure we will be about results." No. The more exactly you duplicate the original experiment, the more likely you duplicate any confounding issues. If someone observes something through a telescope, it's best to verify it through a different telescope somewhere else, to exclude the possibility of an optical quirk of the first telescope.
    - ajuc 15y ago
      Some people will run the same code on the same data. They will get the same results, but so what. Some other people will run the same code on different data (maybe from simulation, or when checking their own theory). They will get some other results, or the same. Some people will recreate experiment from scratch, and get code B and data B. Some people will run code A on data B. Some people will run code B on data A. Now we can't do that. The code is already written, it's waste to hide it, it's not like publishing code prohibit other people from writing their own code. That's how you check for errors - by changing as little as possible and observing results.

jamesrom 15y ago

The only advantage I can see is that when the theory, formulas and algorithms are published, not the specific implementation itself, it forces people who want to who want to verify your work to write their own implementation. It kind of makes sure that a bug in your own implementation doesn't pollute the results of those trying to verify your work.

jonhendry 15y ago
Exactly. It shouldn't matter if Lab A implemented the algorithm in Matlab and Lab B implemented the algorithm in NumPy. Take the data, apply the described algorithms, get the same result. If Lab B doesn't get the same result, and Lab B has confidence in their implementation, then there's probably something funny with Lab A's code or data. Reusing Lab A's code on Lab A's data just reproduces Lab A's errors, if any.
- JonnieCache 15y ago
  This is very true. However if I can just look at Lab A's code and spot an error in it, that will allow me to discount their results without having to redo their experiment, potentially saving years of work and millions of dollars. Maybe that's what researchers are really afraid of?
  - jonhendry 15y ago
    " discount their results without having to redo their experiment, potentially saving years of work and millions of dollars." Or you could just analyze their data using their described methods and your own implementation.
  - alextp 15y ago
    The problem is more complex. Some errors will damage the scientific conclusions, others will show up as fractions of a percentage point inside a much larger confidence interval, and will ultimately not matter much. In machine learning research, for example, most evaluations consist of running a new program on soem data, getting results back, and from these results computing some aggregate measure of performance. A bug on the code that computes this measure of performance is _really bad_ and can invalidate all your conclusions. If that code is right, however, a but on the code that trains your model is completely meaningless, because as long as your results are good you can argue that you actually meant to write a paper about the model actually implemented rather than the model you were supposed to implement. I'm sure other scientific areas have similar distinctions, and a naive code reader might fail to notice that a bug is harmless (and there's also the fact that scientific code carries within it a lot of assumptions about the data which, if broken, can be buggy, but are not broken by real data).

hugh3 15y ago

What possible reason is there for not doing so? The code and the ideas within have no direct commercial value, surely? Laziness, lack of time, and the fact that most scientific code is not suitable for public consumption. For instance, my code includes error messages such as "What the fuck????" and "This never happens" which I'd need to take out in order to prepare it for publication. In addition, there's the assumption that source code is a pretty trivial implementation detail; we publish the algorithm but not the details of the implementation, just as the experimentalist tells us what he built but doesn't tell us what brand of screwdriver he used to build it.

JonnieCache 15y ago
The thing is though, as I'm sure you're aware, the algorithm in abstract is utterly irrelevant. The algorithm as released is never actually executed, it is the implementation that gets executed. One "trivial" detail in the implementation, one tiny little floating point rounding error, can throw the entire thing. We all know this. So why is the scientific community so unwilling to face this elephant in the room? And why is nobody else confronting them about it? Of course there's jgc, with his admirable record of confronting people about things, but that's the only example I can think of. You would've thought the "Al Gore made up AGW to devalue my exxon shares" angst-brigade would've jumped on this, but they seem more concerned with digging through departmental email gossip.
- jonhendry 15y ago
  "The algorithm as released is never actually executed, it is the implementation that gets executed." If you use a different implementation of the algorithm, on the same data, and get a different result, you will find that there was an error. That's a lot easier than combing through someone else's code base. A simple example: If Lab A says they took the square root of 16, you don't need their source code to know there's a problem, if their result is 5 and your result is 4. If you use their code, you might not notice that it's borked.
  - rbanffy 15y ago

jonhendry 15y ago

Biology researchers don't get to reuse the other guy's mice, why should they reuse their code? As long as the paper describes the algorithms used, anyone should be able to duplicate the work using their own preferred implementation. Also, a lot of the code may well be specific to the apparatus used, which may be unique to a lab, consisting of custom-built hardware.

ajuc 15y ago
Mouse can't be perfectly copied. Code can. If biologists could copy mice from other experiment to give them their own drugs and compare results, they would surely do this along with using their own mice. The more ways to compare experiments, the more sure we will be about results. I understand cleaning up code is a lot of work, so don't do it, just release it as it is. I know it will be ugly, will include obscenity, etc. If somebody need it, it will do the rest of work.
- jonhendry 15y ago
  "The more ways to compare experiments, the more sure we will be about results." No. The more exactly you duplicate the original experiment, the more likely you duplicate any confounding issues. If someone observes something through a telescope, it's best to verify it through a different telescope somewhere else, to exclude the possibility of an optical quirk of the first telescope.
  - ajuc 15y ago
    Some people will run the same code on the same data. They will get the same results, but so what. Some other people will run the same code on different data (maybe from simulation, or when checking their own theory). They will get some other results, or the same. Some people will recreate experiment from scratch, and get code B and data B. Some people will run code A on data B. Some people will run code B on data A. Now we can't do that. The code is already written, it's waste to hide it, it's not like publishing code prohibit other people from writing their own code. That's how you check for errors - by changing as little as possible and observing results.