3 ms·

> If it ain't broke, don't fix it. But scientific programming is deeply broken. Code presented along with publications often doesn't work, or is an incomplete

> If it ain't broke, don't fix it. But scientific programming is deeply broken. Code presented along with publications often doesn't work, or is an incomplete subpart/toy example that's supposed to be invoked within some larger framework. That sounds great until you realize that "some larger framework" doesn't refer to a standardized tool, but some deeply customized setup (a la the one you're responding to, that uses e.g. ad hoc code generators across two--or sometimes more--languages because the original authors didn't know how to format a string in one of them). Even if you do get lucky enough to find a paper with all requisite code included, in many cases it was only ever invoked on extensively customized, hand-configured environments. And that configuration was done by non tech folks with a "just get it to where I can run the damn simulation" attitude, so configs are neither documented nor automated. And when I say configs, I'm talking about vital stuff--e.g. env vars that control whether real arithmetic or floating point is used. Often as not, you hack your way to try to get something--anything--running, and it either fails catastrophically or produces the wrong result. Now you have to figure out which of several situations you're in: is the research bad? Were the authors just so non-technical they accidentally omitted a vital piece of code? Was the omission deliberate and profit-motivated (e.g. the PI behind the paper plans on patenting some of the software at some point, so didn't want to publish a special sauce)? Was the omission deliberate and shame-motivated (i.e. researchers didn't want to publish their insane pile of hacks written to backfill an incomplete understanding of the tools being used)? Is it an environment-dependent thing? And all of that is just as pertains to code in published work--usually the higher-quality stuff. Assuming ownership of in-house code from other scientific programmers is much, much worse. This isn't abstract moaning about best practices. The failure of labs, companies, publications, and universities to combat this phenomenon has direct, significant, and negative effects on the quality of research and scientific advancement in many fields. TL;dr it is "broke". When programmers complain about reproducibility crises in soft-science fields, they're throwing rocks from glass houses.

klyrs 6y ago
You're bringing in a whole host of issues inapplicable to the snippet OC found questionable. Don't disagree with ya, but "lack of obvious abstraction" isn't one of these "extreme sensitivity to environment vars" cases. In fact, vociferously complaining about such cases is a great way to turn scientists away from code review as a concept. Fold the code away in your head (or edit your local copy), and dig for subtle issues like numerical sensitivity, environment, etc. That's the way to bring actual value to the process. For the code in question, "oh by the way this can be done simpler", with the simplified snippet, is an appropriate approach to the review. But in my experience it's best to save your breath for actual problems.

klyrs 6y ago

You're bringing in a whole host of issues inapplicable to the snippet OC found questionable. Don't disagree with ya, but "lack of obvious abstraction" isn't one of these "extreme sensitivity to environment vars" cases. In fact, vociferously complaining about such cases is a great way to turn scientists away from code review as a concept. Fold the code away in your head (or edit your local copy), and dig for subtle issues like numerical sensitivity, environment, etc. That's the way to bring actual value to the process. For the code in question, "oh by the way this can be done simpler", with the simplified snippet, is an appropriate approach to the review. But in my experience it's best to save your breath for actual problems.