4 ms·

I'm happy that people are discussing this, but I really don't like they way the article uncritically repeats the message of the Software Sustainability Institut

by swatow 12y ago

I'm happy that people are discussing this, but I really don't like they way the article uncritically repeats the message of the Software Sustainability Institute. The study by the SSI, only proves that people are writing software, and that many of these people have no training in software engineering. The SSI claim this training is necessary to prevent things like the retractions of journal articles they discuss in the article. But their survey doesn't go as far as to provide evidence that there is a link between formal training in software engineering, and errors in scientific journals. If this article was science, I would call it bad science, because the data they present is not sufficient evidence for their hypothesis.

stinos 12y ago
While you are right regarding hard data an evidence, I think the core idea is Problems arise when that software is designed by researchers who really don’t know what they’re doing when it comes to coding. A single mistake in the code can lead to a result that appears innocuous enough, but is actually incorrect. and I am actually sure that is, unfortunatley, very very much a real problem. The reasons I believe so are anecdotal, but so overwhelming I cannot ignore them. I personally prevented already a couple of false results of making it into a paper. And that is just me, in just one research group, in just one university. And that also happened by accident mostly (just staring at a guy's screen during a conversation and noticing errors all over the place, redoing an analysis for more subjects and figuring out it is completely wrong, ...). The mistakes I've seen are close to terrifying. Scripts ignoring all input but instead using a generated fixed dataset. Scripts always yielding 'good' results even if you give them white noise as input. Basically, all typcial programming errors and antipatterns ever invented come together in huge god scripts which produce the results, while in the background Matlab is spitting out one warning after the other, console windows are screaming errors, and all that is just being ignored by the researcher who thinks he/she just mastered the skill of programming. Or knows he/she doesn't, but just doesn't care. As long as there are good results, you know. Oh yes I'm getting a bit sentimental now and it's not like that everywhere and not every researcher writes code like that. But if even me, with over 10 years of programming experience and all my automated builds and tests and whatnot cannot reliably produce code without one mistake, how on earth are researchers for which the programming part is often just a byproduct, a necessary evil, going to do that?
- danieltillett 12y ago
  This is really just bad science - software is just allowing an aspect of poor science to be revealed.
  - stinos 12y ago
    In this particular field software is just one of the many means to get to results. Is every scientist who happens to not be particularily good at one of those means a bad scientist?
    - spuz 12y ago
      I don't think so. Imagine you are a good scientist but a poor coder. How would you verify that the results of your research were valid? You could ask another team to review your software, or you could ask another team to recreate your software, or you could hire a professional software development team to do the job of coding for you.
    - danieltillett 12y ago
      Part of being a good scientist is knowing the areas you are an expert in and asking for help in those areas that you are not. If you don't know how to write code then you need to collaborate with someone who does. In my experience these sort of bad outcome occurs when a senior supervisor gives a student or junior post doc a project that involves programming, but is not able to provide adequate supervision. Instead of finding a collaborator that can help they just let the junior scientist thrash around on their own.
- venomsnake 12y ago
  If a scientist is good enough to design a proper experiment, he is good enough to design a good program. The skillset is the same.
  - stinos 12y ago
    The skilset is similar maybe, but putting it in practice is something else. In your claim you skipped the part where the scientist used his skills to become good enough to design a good program. As long as that learning process didn't happen, and that takes time, he/she will not be good enough. As such I know excellent scientists which are known and praised in their field and come up with inventive designs. Yet their code to do something as simple as plotting a histogram is a mess.

by swatow 12y ago

stinos 12y ago

While you are right regarding hard data an evidence, I think the core idea is Problems arise when that software is designed by researchers who really don’t know what they’re doing when it comes to coding. A single mistake in the code can lead to a result that appears innocuous enough, but is actually incorrect. and I am actually sure that is, unfortunatley, very very much a real problem. The reasons I believe so are anecdotal, but so overwhelming I cannot ignore them. I personally prevented already a couple of false results of making it into a paper. And that is just me, in just one research group, in just one university. And that also happened by accident mostly (just staring at a guy's screen during a conversation and noticing errors all over the place, redoing an analysis for more subjects and figuring out it is completely wrong, ...). The mistakes I've seen are close to terrifying. Scripts ignoring all input but instead using a generated fixed dataset. Scripts always yielding 'good' results even if you give them white noise as input. Basically, all typcial programming errors and antipatterns ever invented come together in huge god scripts which produce the results, while in the background Matlab is spitting out one warning after the other, console windows are screaming errors, and all that is just being ignored by the researcher who thinks he/she just mastered the skill of programming. Or knows he/she doesn't, but just doesn't care. As long as there are good results, you know. Oh yes I'm getting a bit sentimental now and it's not like that everywhere and not every researcher writes code like that. But if even me, with over 10 years of programming experience and all my automated builds and tests and whatnot cannot reliably produce code without one mistake, how on earth are researchers for which the programming part is often just a byproduct, a necessary evil, going to do that?

danieltillett 12y ago
This is really just bad science - software is just allowing an aspect of poor science to be revealed.
- stinos 12y ago
  In this particular field software is just one of the many means to get to results. Is every scientist who happens to not be particularily good at one of those means a bad scientist?
  - spuz 12y ago
    I don't think so. Imagine you are a good scientist but a poor coder. How would you verify that the results of your research were valid? You could ask another team to review your software, or you could ask another team to recreate your software, or you could hire a professional software development team to do the job of coding for you.
  - danieltillett 12y ago
    Part of being a good scientist is knowing the areas you are an expert in and asking for help in those areas that you are not. If you don't know how to write code then you need to collaborate with someone who does. In my experience these sort of bad outcome occurs when a senior supervisor gives a student or junior post doc a project that involves programming, but is not able to provide adequate supervision. Instead of finding a collaborator that can help they just let the junior scientist thrash around on their own.
venomsnake 12y ago
If a scientist is good enough to design a proper experiment, he is good enough to design a good program. The skillset is the same.
- stinos 12y ago
  The skilset is similar maybe, but putting it in practice is something else. In your claim you skipped the part where the scientist used his skills to become good enough to design a good program. As long as that learning process didn't happen, and that takes time, he/she will not be good enough. As such I know excellent scientists which are known and praised in their field and come up with inventive designs. Yet their code to do something as simple as plotting a histogram is a mess.