3 ms·

While you are right regarding hard data an evidence, I think the core idea is Problems arise when that software is designed by researchers who really don’t

by stinos 12y ago

While you are right regarding hard data an evidence, I think the core idea is Problems arise when that software is designed by researchers who really don’t know what they’re doing when it comes to coding. A single mistake in the code can lead to a result that appears innocuous enough, but is actually incorrect. and I am actually sure that is, unfortunatley, very very much a real problem. The reasons I believe so are anecdotal, but so overwhelming I cannot ignore them. I personally prevented already a couple of false results of making it into a paper. And that is just me, in just one research group, in just one university. And that also happened by accident mostly (just staring at a guy's screen during a conversation and noticing errors all over the place, redoing an analysis for more subjects and figuring out it is completely wrong, ...). The mistakes I've seen are close to terrifying. Scripts ignoring all input but instead using a generated fixed dataset. Scripts always yielding 'good' results even if you give them white noise as input. Basically, all typcial programming errors and antipatterns ever invented come together in huge god scripts which produce the results, while in the background Matlab is spitting out one warning after the other, console windows are screaming errors, and all that is just being ignored by the researcher who thinks he/she just mastered the skill of programming. Or knows he/she doesn't, but just doesn't care. As long as there are good results, you know. Oh yes I'm getting a bit sentimental now and it's not like that everywhere and not every researcher writes code like that. But if even me, with over 10 years of programming experience and all my automated builds and tests and whatnot cannot reliably produce code without one mistake, how on earth are researchers for which the programming part is often just a byproduct, a necessary evil, going to do that?

danieltillett 12y ago
This is really just bad science - software is just allowing an aspect of poor science to be revealed.
- stinos 12y ago
  In this particular field software is just one of the many means to get to results. Is every scientist who happens to not be particularily good at one of those means a bad scientist?
  - spuz 12y ago
    I don't think so. Imagine you are a good scientist but a poor coder. How would you verify that the results of your research were valid? You could ask another team to review your software, or you could ask another team to recreate your software, or you could hire a professional software development team to do the job of coding for you.
  - danieltillett 12y ago
    Part of being a good scientist is knowing the areas you are an expert in and asking for help in those areas that you are not. If you don't know how to write code then you need to collaborate with someone who does. In my experience these sort of bad outcome occurs when a senior supervisor gives a student or junior post doc a project that involves programming, but is not able to provide adequate supervision. Instead of finding a collaborator that can help they just let the junior scientist thrash around on their own.
venomsnake 12y ago
If a scientist is good enough to design a proper experiment, he is good enough to design a good program. The skillset is the same.
- stinos 12y ago
  The skilset is similar maybe, but putting it in practice is something else. In your claim you skipped the part where the scientist used his skills to become good enough to design a good program. As long as that learning process didn't happen, and that takes time, he/she will not be good enough. As such I know excellent scientists which are known and praised in their field and come up with inventive designs. Yet their code to do something as simple as plotting a histogram is a mess.
- angersock 12y ago
  "Good enough" and "actually carries through" are different. Especially in programming, where you want to actively avoid reinventing the wheel as much as is possible.
- ScottBurson 12y ago
  This is not remotely true, to the point that I don't even understand how you could assert it. What skills taught to physicists or chemists or biologists are even related to notions like modularity and data abstraction? Hell, I've seen people come out of computer science PhD programs with poor software engineering skills and habits. If CS programs don't reliably teach these things, how can you claim researchers in other fields automatically have them?
swatow 12y ago
I'm totally on board with the idea that research software is a problem. My question is whether formal training (which the Sustainable Software Institute who created this study seem to advocate) is the best or most important solution. Two other possibilities: - Research code is uniquely difficult in ways that are different to other kinds of software. This is because the logic is often inherently complicated, since you are doing a complex calculation. - Researchers are stuck using old fashioned languages and tools because they don't know better. Note that this doesn't necessarily they need training, only being pointed to the right places.
- collyw 12y ago
  I work with researchers. While I can't say whether they produced reliable results or not, they would regularly grind the network to a half by producing millions of small files. I did talks on how to use a database, and why it would stop these problems, but hardly any were interested.
- stinos 12y ago
  Hard to tell if training by SSI or so is the solution, but any kind of training would probably do way more good than bad. Sure some research code is quite difficult and not all alike your typical we/business app, but that doesn't mean universal programming principles don't apply or wouldn't make for better code. I'd also argue that pointing to the right places, in a way that sticks, i.e. with proper explanation of why those are the right places, is training.

by stinos 12y ago

danieltillett 12y ago

This is really just bad science - software is just allowing an aspect of poor science to be revealed.

stinos 12y ago
In this particular field software is just one of the many means to get to results. Is every scientist who happens to not be particularily good at one of those means a bad scientist?
- spuz 12y ago
  I don't think so. Imagine you are a good scientist but a poor coder. How would you verify that the results of your research were valid? You could ask another team to review your software, or you could ask another team to recreate your software, or you could hire a professional software development team to do the job of coding for you.
- danieltillett 12y ago
  Part of being a good scientist is knowing the areas you are an expert in and asking for help in those areas that you are not. If you don't know how to write code then you need to collaborate with someone who does. In my experience these sort of bad outcome occurs when a senior supervisor gives a student or junior post doc a project that involves programming, but is not able to provide adequate supervision. Instead of finding a collaborator that can help they just let the junior scientist thrash around on their own.

venomsnake 12y ago

If a scientist is good enough to design a proper experiment, he is good enough to design a good program. The skillset is the same.

stinos 12y ago
The skilset is similar maybe, but putting it in practice is something else. In your claim you skipped the part where the scientist used his skills to become good enough to design a good program. As long as that learning process didn't happen, and that takes time, he/she will not be good enough. As such I know excellent scientists which are known and praised in their field and come up with inventive designs. Yet their code to do something as simple as plotting a histogram is a mess.
angersock 12y ago
"Good enough" and "actually carries through" are different. Especially in programming, where you want to actively avoid reinventing the wheel as much as is possible.
ScottBurson 12y ago
This is not remotely true, to the point that I don't even understand how you could assert it. What skills taught to physicists or chemists or biologists are even related to notions like modularity and data abstraction? Hell, I've seen people come out of computer science PhD programs with poor software engineering skills and habits. If CS programs don't reliably teach these things, how can you claim researchers in other fields automatically have them?

swatow 12y ago

I'm totally on board with the idea that research software is a problem. My question is whether formal training (which the Sustainable Software Institute who created this study seem to advocate) is the best or most important solution. Two other possibilities: - Research code is uniquely difficult in ways that are different to other kinds of software. This is because the logic is often inherently complicated, since you are doing a complex calculation. - Researchers are stuck using old fashioned languages and tools because they don't know better. Note that this doesn't necessarily they need training, only being pointed to the right places.

collyw 12y ago
I work with researchers. While I can't say whether they produced reliable results or not, they would regularly grind the network to a half by producing millions of small files. I did talks on how to use a database, and why it would stop these problems, but hardly any were interested.
stinos 12y ago
Hard to tell if training by SSI or so is the solution, but any kind of training would probably do way more good than bad. Sure some research code is quite difficult and not all alike your typical we/business app, but that doesn't mean universal programming principles don't apply or wouldn't make for better code. I'd also argue that pointing to the right places, in a way that sticks, i.e. with proper explanation of why those are the right places, is training.