5 ms·

When I was in my doctoral program I had some pretty promising early results applying network analysis to metabolic networks. My lab boss/PI was happy to adverti

When I was in my doctoral program I had some pretty promising early results applying network analysis to metabolic networks. My lab boss/PI was happy to advertise my work and scheduled a cross-departmental talk to present my research in front of ~100 professors or so. While I was making a last-minute slide for my presentation I realized one chart looked a little off and I started looking into the raw data. I soon realized that I had a bug in my code that invalidated the last 12 months of calculations run on our HPC cluster. My conclusions were flat out wrong and there was nothing to salvage from the data. I went to my lab boss the night before the talk and told him to cancel it and he just told me to lie and present it anyways. I didn't think that was moral or scientifically sound and I refused. It permanently damaged my professional relationship with him. No one else I talked to seemed particularly concerned about this, and I realized that a lot of people around me were bowing to pressure to fudge results here and there to keep up the cycle of publicity, results, and funding that the entire academic enterprise relied upon. It broke a lot of the faith I had been carrying in science as an institution, at least as far as it is practiced in major American research universities.

mike_hearn 2y ago
Coding errors are a really common source of fraud unfortunately. You did the right thing but the vast majority don't. Given a choice between admitting the grant money was wasted, the exciting finding isn't real, everyone who cited your work should retract their papers or just covering it up, the pressure to do the latter is enormous. During COVID I talked to a guy who used to do computational epidemiology. He came to me because I'd written about the fraud that's endemic in that field and wanted to get stuff off his chest. He was a research programmer, assisting scientists. One of the stories he told involved checking the code for a model written in FORTRAN. He discovered it was mis-using an FFI and using pointer values in equations instead of the dereferenced values. Everything the program had ever calculated was garbage. He checked and it had been used in hundreds of papers. After emailing the authors with a bug report, he got a reply half an hour later saying the papers had been checked and the results didn't change so nothing needed to be done. Little known fact: the COVID model that drove lockdowns in the UK and USA was nothing but bugs. None of the numbers it produced can be replicated. When people pointed out that this was a problem academics went on the attack, claimed none of the criticism was legitimate because it didn't come from experts, and of course simp journalists went along with all of it. They got away with it completely and will probably do it again in future. Even in this thread you can see people defending COVID science as the good stuff! It was all riddled with fraud. Part of the issue is that scientists are mostly self taught coders who aren't interested in coding. They frequently use a standard of: if it looks right, it is right. The whole thing becomes a circular exercise in reinforcing their priors.
- jboggan 2y ago
  >the COVID model that drove lockdowns in the UK and USA was nothing but bugs I would love a source but I believe this given my experience with coding standards in the computational biology space, especially given that some of my own previous work and teaching touched on those models, and I couldn't believe any of the models that were publicized because they were at odds with what I thought the scientific consensus was about spread and containment models (96 hours after patient 0 it's pointless to attempt to restrict movement).
  - mike_hearn 2y ago
    For a source, I wrote this a few years ago: https://dailysceptic.org/2020/05/06/code-review-of-fergusons-model/ https://dailysceptic.org/2020/05/06/code-review-of-fergusons... It was based primarily on the model's own github bug tracker.
shusaku 2y ago
> My conclusions were flat out wrong and there was nothing to salvage from the data. Wow that’s pretty crazy. I have to say many times in my career I’ve been writing a paper and realized “** there’s a bug”, and had to redo everything. But the overall conclusion never changed because the idea was grounded from several different angles(usually the pieces fit together even better). One bug might invalidate your result, but even if your code was correct the underlying assumptions behind the code could be wrong! I think the real issue was your boss wasn’t active enough in your work to make it robust to coding mistakes.
- jboggan 2y ago
  >I think the real issue was your boss wasn’t active enough in your work to make it robust to coding mistakes. That's a major issue and it went beyond coding, I picked a very well known and influential advisor and eventually discovered he didn't really direct any research or write papers, that was all the research assistants and postdocs. I was pretty much left on my own and expected to surface with a paper to put his name on.

mike_hearn 2y ago

Coding errors are a really common source of fraud unfortunately. You did the right thing but the vast majority don't. Given a choice between admitting the grant money was wasted, the exciting finding isn't real, everyone who cited your work should retract their papers or just covering it up, the pressure to do the latter is enormous. During COVID I talked to a guy who used to do computational epidemiology. He came to me because I'd written about the fraud that's endemic in that field and wanted to get stuff off his chest. He was a research programmer, assisting scientists. One of the stories he told involved checking the code for a model written in FORTRAN. He discovered it was mis-using an FFI and using pointer values in equations instead of the dereferenced values. Everything the program had ever calculated was garbage. He checked and it had been used in hundreds of papers. After emailing the authors with a bug report, he got a reply half an hour later saying the papers had been checked and the results didn't change so nothing needed to be done. Little known fact: the COVID model that drove lockdowns in the UK and USA was nothing but bugs. None of the numbers it produced can be replicated. When people pointed out that this was a problem academics went on the attack, claimed none of the criticism was legitimate because it didn't come from experts, and of course simp journalists went along with all of it. They got away with it completely and will probably do it again in future. Even in this thread you can see people defending COVID science as the good stuff! It was all riddled with fraud. Part of the issue is that scientists are mostly self taught coders who aren't interested in coding. They frequently use a standard of: if it looks right, it is right. The whole thing becomes a circular exercise in reinforcing their priors.

jboggan 2y ago
>the COVID model that drove lockdowns in the UK and USA was nothing but bugs I would love a source but I believe this given my experience with coding standards in the computational biology space, especially given that some of my own previous work and teaching touched on those models, and I couldn't believe any of the models that were publicized because they were at odds with what I thought the scientific consensus was about spread and containment models (96 hours after patient 0 it's pointless to attempt to restrict movement).
- mike_hearn 2y ago
  For a source, I wrote this a few years ago: https://dailysceptic.org/2020/05/06/code-review-of-fergusons-model/ https://dailysceptic.org/2020/05/06/code-review-of-fergusons... It was based primarily on the model's own github bug tracker.

shusaku 2y ago

> My conclusions were flat out wrong and there was nothing to salvage from the data. Wow that’s pretty crazy. I have to say many times in my career I’ve been writing a paper and realized “** there’s a bug”, and had to redo everything. But the overall conclusion never changed because the idea was grounded from several different angles(usually the pieces fit together even better). One bug might invalidate your result, but even if your code was correct the underlying assumptions behind the code could be wrong! I think the real issue was your boss wasn’t active enough in your work to make it robust to coding mistakes.

jboggan 2y ago
>I think the real issue was your boss wasn’t active enough in your work to make it robust to coding mistakes. That's a major issue and it went beyond coding, I picked a very well known and influential advisor and eventually discovered he didn't really direct any research or write papers, that was all the research assistants and postdocs. I was pretty much left on my own and expected to surface with a paper to put his name on.