4 ms·

Many senior members of the academic community rely on their reputation as researchers to brush aside basic issues with the software that they develop for scient

by ak217 6y ago

Many senior members of the academic community rely on their reputation as researchers to brush aside basic issues with the software that they develop for scientific purposes. These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. The lack of focus on research software quality among senior PIs, funding committees, and article reviewers is a huge problem in academia. The problems with the Ferguson model and codebase are just the latest prominent example of this. The tools that academics have access to are excellent. If you compare the free support academic software developers receive from the rest of their community to other engineering disciplines, it's beyond great. I think the OP is wrong to suggest otherwise. The problems with the Ferguson model are an opportunity to educate more members of the community about the fact that good software development practices are not optional for good science, and that senior members of the community jeopardize their own reputation by not paying attention to them.

veddox 6y ago
> If you compare the free support academic software developers receive from the rest of their community to other engineering disciplines, it's beyond great. Could you elaborate on what you mean by that?
- rjsw 6y ago
  Not the OP, but I know of at least one University that runs "Programming for scientists" courses at low or no cost.
  - fsh 6y ago
    These courses are usually short introductions into python or Java. Most scientists simply don't have time in their curriculum for a full software engineering course.
    - rjsw 6y ago
      I didn't claim they were good. My father is doing a Python one right now and I'm not very impressed. He is retired from the University but is still eligible to take the course.
  - veddox 6y ago
    Oh we teach programming alright, but software development is another pair of shoes entirely. Unfortunately, that's where the problem lies...
    - zozbot234 6y ago
      That's the whole problem with the "Learn to code" movement, tbh. Plus Python and Java are terrible languages when it comes to those broader swdev/softeng concerns.
      - 0xdeadbeefbabe 6y ago
        What is better than Java?
chrisseaton 6y ago
> These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. Or maybe these things aren't actually as important as we think they are in professional software development? If they're able to produce useful scientific results (in general, not specifically in this case) without those things then maybe they don't matter as much as we think they do?
- lalaithion 6y ago
  The problem is, we need confidence that the software is correct in order to trust the scientific results.
  - chrisseaton 6y ago
    Do you need tests to do that? Are tests the only way? Are you doing formal verification of your software? Why not? If you're not bothering to do that why are you criticising researchers for not bothering to use tests?
- zozbot234 6y ago
  > Or maybe these things aren't actually as important as we think they are in professional software development? There's something to this. Most academic software is a simple one-off development, with little consideration for long-term maintenance. OTOH it's not likely that undocumented, untested and unfixable software can produce actual "useful scientific results".
- csours 6y ago
  These qualities are important for exactly the same reasons they are in production: Without those qualities, your code is brittle, your deploys are brittle, changes are brittle. It's just like saying "It runs on my machine". The scientific term for this is "Replication crisis [0]" 0 - https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
  - Balgair 6y ago
silveraxe93 6y ago
I couldn't agree more. We're in the middle of a reproducibility crisis. It's extremely important that researchers start putting more effort into quality research instead of quantity. "Publish or Perish" is killing academia.
Balgair 6y ago
Academics are not software people, by and large. The code is custom built for a specific graph or statistical run. Typically the people writing the software are grad students with little to no interest in code; they just want the answer. Their interest is in the thing they are studying, not code. They may not even have coded before grad school, nor know matrix algebra or calculus. Excel is a big step for them. I have experience in this issue. A fellow grad student of mine asked for some help with his code. I said sure, but it'll cost you a 12 pack of beer. After the 11th nested 'if' statement in Matlab, I upped it to a 24 pack. We never did get it working right.
Joeboy 6y ago
> These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability Have you actually checked to what extent the Ferguson codebase is guilty of these things? It seems to me there are people spreading misinformation, and too many people taking their word for it.

by ak217 6y ago

veddox 6y ago

> If you compare the free support academic software developers receive from the rest of their community to other engineering disciplines, it's beyond great. Could you elaborate on what you mean by that?

rjsw 6y ago
Not the OP, but I know of at least one University that runs "Programming for scientists" courses at low or no cost.
- fsh 6y ago
  These courses are usually short introductions into python or Java. Most scientists simply don't have time in their curriculum for a full software engineering course.
  - rjsw 6y ago
    I didn't claim they were good. My father is doing a Python one right now and I'm not very impressed. He is retired from the University but is still eligible to take the course.
- veddox 6y ago
  Oh we teach programming alright, but software development is another pair of shoes entirely. Unfortunately, that's where the problem lies...
  - zozbot234 6y ago
    That's the whole problem with the "Learn to code" movement, tbh. Plus Python and Java are terrible languages when it comes to those broader swdev/softeng concerns.
    - 0xdeadbeefbabe 6y ago
      What is better than Java?

chrisseaton 6y ago

> These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. Or maybe these things aren't actually as important as we think they are in professional software development? If they're able to produce useful scientific results (in general, not specifically in this case) without those things then maybe they don't matter as much as we think they do?

lalaithion 6y ago
The problem is, we need confidence that the software is correct in order to trust the scientific results.
- chrisseaton 6y ago
  Do you need tests to do that? Are tests the only way? Are you doing formal verification of your software? Why not? If you're not bothering to do that why are you criticising researchers for not bothering to use tests?
zozbot234 6y ago
> Or maybe these things aren't actually as important as we think they are in professional software development? There's something to this. Most academic software is a simple one-off development, with little consideration for long-term maintenance. OTOH it's not likely that undocumented, untested and unfixable software can produce actual "useful scientific results".
csours 6y ago
These qualities are important for exactly the same reasons they are in production: Without those qualities, your code is brittle, your deploys are brittle, changes are brittle. It's just like saying "It runs on my machine". The scientific term for this is "Replication crisis [0]" 0 - https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
- Balgair 6y ago

silveraxe93 6y ago

I couldn't agree more. We're in the middle of a reproducibility crisis. It's extremely important that researchers start putting more effort into quality research instead of quantity. "Publish or Perish" is killing academia.

Balgair 6y ago

Academics are not software people, by and large. The code is custom built for a specific graph or statistical run. Typically the people writing the software are grad students with little to no interest in code; they just want the answer. Their interest is in the thing they are studying, not code. They may not even have coded before grad school, nor know matrix algebra or calculus. Excel is a big step for them. I have experience in this issue. A fellow grad student of mine asked for some help with his code. I said sure, but it'll cost you a 12 pack of beer. After the 11th nested 'if' statement in Matlab, I upped it to a 24 pack. We never did get it working right.

Joeboy 6y ago

> These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability Have you actually checked to what extent the Ferguson codebase is guilty of these things? It seems to me there are people spreading misinformation, and too many people taking their word for it.