4 ms·

There is the factor that a lot of the code scientists write is hacky, one-off, and fragile. The kinds of people who care about releasing their code also feel at

by rflrob 15y ago

There is the factor that a lot of the code scientists write is hacky, one-off, and fragile. The kinds of people who care about releasing their code also feel at least a little embarassed about the code quality. There's at least one license that recognizes and embraces this fact: http://matt.might.net/articles/crapl/ http://matt.might.net/articles/crapl/

_delirium 15y ago
A more serious problem with that than embarrassment is that such code sometimes really shouldn't be uncritically reused, if we care about reproducibility. An independent reimplementation that reaches the same results is more convincing to me than a 2nd scientist getting the same results when they re-run the 1st scientist's hacky code. It's even more of a problem in the case of code that gets passed around and slowly accumulates ad-hoc additions because nobody wants to reimplement it.
- mjwalshe 15y ago
  Precisely you have to be able to reproduce the experiment with out using the first ones code
  - bo1024 15y ago
    This is true on its own, but seems like a bad standard to hold science to. With normal experiments, it's not enough to say "we ran an experiment that tested the pliability of different materials and found X is most pliable", then say, "you should be able to find that X is most pliable without reusing our method." The point of publishing a method is so it can be critiqued; I think the same should hold with source code. This should not at all excuse people from trying to reproduce simulations with separate code. Also, source code lies kind of halfway between experimental measures and mathematical proofs. Again, you are usually expected to give proofs of non-obvious mathematical results, at least in the supplementary section. Similarly, saying "there exists code which produces this result" shouldn't be sufficient unless it's very obvious.
    - jonhendry 15y ago
      "The point of publishing a method is so it can be critiqued; I think the same should hold with source code. " Except that source code can sometimes obfuscate the intent. It's probably better to provide pseudocode. Don't provide source code for your binary sort, say you sorted the data and say on what it was sorted, and let other people use their own preferred sort implementation. Especially since other labs may not use the same equipment, libraries, languages, etc, so source code may be useless.
      - bo1024 15y ago
        I definitely think it would be great to require/expect pseudocode. I'm not sure if it would be better or worse than providing the original code (hey, why not both?), but it would be a very good standard to adopt.
      - rflrob 15y ago
        "Except that source code can sometimes obfuscate the intent." The source code, no matter how opaque and poorly written, can never really make things less clear. That's because it must be able to be interpreted by a computer. Good pseudocode and high level descriptions can help illuminate the code, but, as the saying goes, "If the code and the comments disagree, then both are probably wrong."
Craiggybear 15y ago
"There is the factor that a lot of the code scientists write is hacky, one-off, and fragile" Oh, absolutely. But the same goes for expensive, commercial software, too. In all walks of life, not just scientific software.

by rflrob 15y ago

_delirium 15y ago

A more serious problem with that than embarrassment is that such code sometimes really shouldn't be uncritically reused, if we care about reproducibility. An independent reimplementation that reaches the same results is more convincing to me than a 2nd scientist getting the same results when they re-run the 1st scientist's hacky code. It's even more of a problem in the case of code that gets passed around and slowly accumulates ad-hoc additions because nobody wants to reimplement it.

mjwalshe 15y ago
Precisely you have to be able to reproduce the experiment with out using the first ones code
- bo1024 15y ago
  This is true on its own, but seems like a bad standard to hold science to. With normal experiments, it's not enough to say "we ran an experiment that tested the pliability of different materials and found X is most pliable", then say, "you should be able to find that X is most pliable without reusing our method." The point of publishing a method is so it can be critiqued; I think the same should hold with source code. This should not at all excuse people from trying to reproduce simulations with separate code. Also, source code lies kind of halfway between experimental measures and mathematical proofs. Again, you are usually expected to give proofs of non-obvious mathematical results, at least in the supplementary section. Similarly, saying "there exists code which produces this result" shouldn't be sufficient unless it's very obvious.
  - jonhendry 15y ago
    "The point of publishing a method is so it can be critiqued; I think the same should hold with source code. " Except that source code can sometimes obfuscate the intent. It's probably better to provide pseudocode. Don't provide source code for your binary sort, say you sorted the data and say on what it was sorted, and let other people use their own preferred sort implementation. Especially since other labs may not use the same equipment, libraries, languages, etc, so source code may be useless.
    - bo1024 15y ago
      I definitely think it would be great to require/expect pseudocode. I'm not sure if it would be better or worse than providing the original code (hey, why not both?), but it would be a very good standard to adopt.
    - rflrob 15y ago
      "Except that source code can sometimes obfuscate the intent." The source code, no matter how opaque and poorly written, can never really make things less clear. That's because it must be able to be interpreted by a computer. Good pseudocode and high level descriptions can help illuminate the code, but, as the saying goes, "If the code and the comments disagree, then both are probably wrong."

Craiggybear 15y ago

"There is the factor that a lot of the code scientists write is hacky, one-off, and fragile" Oh, absolutely. But the same goes for expensive, commercial software, too. In all walks of life, not just scientific software.