6 ms·

"The algorithm as released is never actually executed, it is the implementation that gets executed." If you use a different implementation of the algorithm, on

"The algorithm as released is never actually executed, it is the implementation that gets executed." If you use a different implementation of the algorithm, on the same data, and get a different result, you will find that there was an error. That's a lot easier than combing through someone else's code base. A simple example: If Lab A says they took the square root of 16, you don't need their source code to know there's a problem, if their result is 5 and your result is 4. If you use their code, you might not notice that it's borked.

rbanffy 15y ago
Still, this is not a good reason not to release code. We can deal with funny comments, bad jokes and dead code. Why not let people examine the code? They are not programmers who will be judged by the neatness of their code.
- jonhendry 15y ago
  What if they don't have code, because they're using a commercial product to do the recording?
  - rbanffy 15y ago
    Well... That part has to be certified as a black box, but even if your code is a Mathematica notebook, it would be useful to have it published along the raw data. Again, there is no reason to be embarrassed of comments like "this should never happen". My own code has parts that raise "ThisShouldNeverHappenError" when something that should never happen disregards my opinion and happens anyway. ;-)
jerf 15y ago
You've implicitly accepted that the first implementation is correct, which I've found is one of the mental blocks that people defending this practice seem to have with frequency that surprises me. It isn't true. It isn't even the sort of thing you can try to debate your way out of, because it simply isn't true. Rather complicated numeric code written by non-software engineers is not something I'm willing to presume correct. I wouldn't presume it correct if it were written by software engineers, either, but at least then we'd probably have some sort of halfway reasonable testing to point at. Further, the Kolmogorov complexity of a simulation is typically a wee bit larger than the Kolmogorov complexity of a square root implementation. That metaphor is beyond useless, it's deceptive.
- jonhendry 15y ago
  "You've implicitly accepted that the first implementation is correct" Not at all. The questions are 1) does the method described sound reasonable and 2) if I implement the same methods, with my toolset, and process the same data, do I get the same result? If the results are different, using different implementations of the same methods, then it's possible that either party could have the error. But if the second party is reusing well-tested code of their own, or something from Matlab or whatever, then it is more likely that the unknown code of the original lab is the problem. You might have a point about a simulation, but an awful lot of science isn't simulations, it's analyzing recorded data.
  - jerf 15y ago
    s/simulation/any real code/. It's still not a square root, which is a total straw man. If it were that simple, there'd be no code to write in the first place.
    - jonhendry 15y ago
      Okay, instead of 'square root' try spike sorting algorithms for recorded neuron activity. There are various ways of doing it, various software packages, free and commercial, ways of doing it in Matlab, etc. (Ex: http://www.plexon.com/product/Offline_Sorter.html#Features http://www.plexon.com/product/Offline_Sorter.html#Features) Given a terabyte of recording data, I'd think it should be sufficient to specify the parameters they used for spike sorting. Someone else ought to be able to use a different package on the same data and obtain results similar enough to know if the first person fudged their paper. Given a set of data, and a named algorithm, it ought to be possible to obtain the same result with any correct implementation of that algorithm.
      - jerf 15y ago
        That's begging the question. Yes, of course a correct implementation will by definition obtain the correct result. I'm questioning the casual way people assume they've got the correct algorithm implemented at all. Is your paper nothing but off-the-shelf "spike sorting", or is that used as a component of something larger? Rolling back around to the original point of source code release being a desirable thing, if it's the latter, just knowing that this one library was used from what is probably literally a single sentence in the paper that reads "We used spike sorting software X to obtain sorted spikes", when presumably the library has knobs whose settings you don't know and you still don't even have their raw data to check the settings with, you still know virtually nothing about what was actually done. Source release ought to be standard. The degree to which people who are putatively scientists will go to bat to defend making it difficult or impossible to replicate their experiments boggles my mind.
jasonwatkinspdx 15y ago
This assumes errors by implementors are uncorrelated, which is almost certainly false. As programmers we often all make similar mistakes, particularly in exacting numerical code.
- jonhendry 15y ago
  True, but the second implementation may use different, well-tested, mature code. If the first lab used C++, and the second lab used Matlab and its libraries, and the third lab used trusted code based on NumPy, I would be surprised if their errors were strongly correlated. Brand new code? Sure. But not code they've used for several years through multiple experiments, or code that's part of a widely-used software package. The thing is, labs working in similar areas most likely have ready, tested, trusted software toolkits for doing similar tasks, especially when it comes to data analysis. It may seem to a programmer that the fastest, easiest thing to do would be to read the code, in fact the fastest, easiest thing to do is probably for another lab in the same field to do an independent analysis of the data with their own code.
  - saurik 15y ago
    "trusted code based on NumPy" <- (Just to note, a number of NumPy's routines have seemingly known numerical instability issues; I would not trust it, and was burned by it badly a few months ago. If you've used a specific function before and it seemed to work, then that's great; but if you haven't: watch out.)

rbanffy 15y ago

Still, this is not a good reason not to release code. We can deal with funny comments, bad jokes and dead code. Why not let people examine the code? They are not programmers who will be judged by the neatness of their code.

jonhendry 15y ago
What if they don't have code, because they're using a commercial product to do the recording?
- rbanffy 15y ago
  Well... That part has to be certified as a black box, but even if your code is a Mathematica notebook, it would be useful to have it published along the raw data. Again, there is no reason to be embarrassed of comments like "this should never happen". My own code has parts that raise "ThisShouldNeverHappenError" when something that should never happen disregards my opinion and happens anyway. ;-)

jerf 15y ago

You've implicitly accepted that the first implementation is correct, which I've found is one of the mental blocks that people defending this practice seem to have with frequency that surprises me. It isn't true. It isn't even the sort of thing you can try to debate your way out of, because it simply isn't true. Rather complicated numeric code written by non-software engineers is not something I'm willing to presume correct. I wouldn't presume it correct if it were written by software engineers, either, but at least then we'd probably have some sort of halfway reasonable testing to point at. Further, the Kolmogorov complexity of a simulation is typically a wee bit larger than the Kolmogorov complexity of a square root implementation. That metaphor is beyond useless, it's deceptive.

jonhendry 15y ago
"You've implicitly accepted that the first implementation is correct" Not at all. The questions are 1) does the method described sound reasonable and 2) if I implement the same methods, with my toolset, and process the same data, do I get the same result? If the results are different, using different implementations of the same methods, then it's possible that either party could have the error. But if the second party is reusing well-tested code of their own, or something from Matlab or whatever, then it is more likely that the unknown code of the original lab is the problem. You might have a point about a simulation, but an awful lot of science isn't simulations, it's analyzing recorded data.
- jerf 15y ago
  s/simulation/any real code/. It's still not a square root, which is a total straw man. If it were that simple, there'd be no code to write in the first place.
  - jonhendry 15y ago
    Okay, instead of 'square root' try spike sorting algorithms for recorded neuron activity. There are various ways of doing it, various software packages, free and commercial, ways of doing it in Matlab, etc. (Ex: http://www.plexon.com/product/Offline_Sorter.html#Features http://www.plexon.com/product/Offline_Sorter.html#Features) Given a terabyte of recording data, I'd think it should be sufficient to specify the parameters they used for spike sorting. Someone else ought to be able to use a different package on the same data and obtain results similar enough to know if the first person fudged their paper. Given a set of data, and a named algorithm, it ought to be possible to obtain the same result with any correct implementation of that algorithm.
    - jerf 15y ago
      That's begging the question. Yes, of course a correct implementation will by definition obtain the correct result. I'm questioning the casual way people assume they've got the correct algorithm implemented at all. Is your paper nothing but off-the-shelf "spike sorting", or is that used as a component of something larger? Rolling back around to the original point of source code release being a desirable thing, if it's the latter, just knowing that this one library was used from what is probably literally a single sentence in the paper that reads "We used spike sorting software X to obtain sorted spikes", when presumably the library has knobs whose settings you don't know and you still don't even have their raw data to check the settings with, you still know virtually nothing about what was actually done. Source release ought to be standard. The degree to which people who are putatively scientists will go to bat to defend making it difficult or impossible to replicate their experiments boggles my mind.

jasonwatkinspdx 15y ago

This assumes errors by implementors are uncorrelated, which is almost certainly false. As programmers we often all make similar mistakes, particularly in exacting numerical code.

jonhendry 15y ago
True, but the second implementation may use different, well-tested, mature code. If the first lab used C++, and the second lab used Matlab and its libraries, and the third lab used trusted code based on NumPy, I would be surprised if their errors were strongly correlated. Brand new code? Sure. But not code they've used for several years through multiple experiments, or code that's part of a widely-used software package. The thing is, labs working in similar areas most likely have ready, tested, trusted software toolkits for doing similar tasks, especially when it comes to data analysis. It may seem to a programmer that the fastest, easiest thing to do would be to read the code, in fact the fastest, easiest thing to do is probably for another lab in the same field to do an independent analysis of the data with their own code.
- saurik 15y ago
  "trusted code based on NumPy" <- (Just to note, a number of NumPy's routines have seemingly known numerical instability issues; I would not trust it, and was burned by it badly a few months ago. If you've used a specific function before and it seemed to work, then that's great; but if you haven't: watch out.)