3 ms·
> These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. Or maybe these things aren't actu
by chrisseaton 7y ago
> These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability.
Or maybe these things aren't actually as important as we think they are in professional software development?
If they're able to produce useful scientific results (in general, not specifically in this case) without those things then maybe they don't matter as much as we think they do?
- lalaithion 7y agoThe problem is, we need confidence that the software is correct in order to trust the scientific results.
- chrisseaton 7y agoDo you need tests to do that? Are tests the only way? Are you doing formal verification of your software? Why not? If you're not bothering to do that why are you criticising researchers for not bothering to use tests?
- zozbot234 7y ago> Or maybe these things aren't actually as important as we think they are in professional software development? There's something to this. Most academic software is a simple one-off development, with little consideration for long-term maintenance. OTOH it's not likely that undocumented, untested and unfixable software can produce actual "useful scientific results".
- csours 7y agoThese qualities are important for exactly the same reasons they are in production: Without those qualities, your code is brittle, your deploys are brittle, changes are brittle. It's just like saying "It runs on my machine". The scientific term for this is "Replication crisis [0]" 0 - https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
- Balgair 7y agoThe Replication Crisis is much larger in scope than just software reproducibility. But yes, it does include that too.
- csours 7y agoCertainly. I think the "it works on my machine" attitude is reminiscent of the problems associated with the replication crisis.
- chrisseaton 7y ago> Without those qualities, your code is brittle, your deploys are brittle, changes are brittle. Is this brittleness stopping the scientists achieving what they need to achieve? Are you sure that writing tests makes science better? Or are you just assuming that? They aren't idiots and they aren't ignorant of how professional software developers work.
- csours 7y ago> Are you sure that writing tests makes science better? Or are you just assuming that? It is perfectly acceptable to write code without tests. Proof-of-Concept or Minimum Viable Product are a great place to write code without tests. It is less acceptable if other people will run or use that code. It is even less acceptable if anyone (including oneself) ever updates or extends the code. --- You could take this analogy to scientific instruments. Imagine you make a novel particle detector. You get a scientific result with your detector. A colleague uses your detector, but they don't clean it properly before use, and they use a power supply with lower voltage. They don't detect any particles! Was your science bad? Would the "science be better" if there were clearer instructions and pre-requisites? Now imagine another scientist makes a copy of your detector from the description in your paper. They get some stuff wrong because your description was ambiguous. Was the science bad? --- By the way, all of these things are real problems with scientific investigations, and not just in the software realm.
- mcv 7y agoIf the results aren't reproducible, they can't be assumed to be true. Then they're only useful if you only care about publication and not about whether the results are actually true. And yes, this is a serious problem in science.
- 0xdeadbeefbabe 7y ago> maybe these things aren't actually as important as we think they are in professional software development? Maybe for professional professionals they are. It is funny that Hinsen's insight is so mundane and obvious: "the code itself is merely a means to this end".
- pushingice 7y agoAs a former academic, the point is to get the research published. Whether the code continues to work after that, is irrelevant.
- brigandish 7y agoThis is, indeed, the whole truth and nothing but the truth. My own anecdote: through happenstance I'm co-author on a paper, a large number of my friends are PIs/PhDs etc and one of them needed help after the software written for them by the CS student they'd been assigned didn't work (he got a masters for it, I can't believe anyone in the CS department looked at his work). I asked about publishing my code after the paper was published and no one seemed bothered. I've also seen my friends on the mill of, shall we call it paper chasing? That seems most apt. It reminded me of borrowing someone's homework and making minor changes to get it past teacher - except the homework was the same paper being constantly rejigged for submission to a new journal. Not much truly new work seemed to be going on.