4 ms·

I wrote a paper with co-authors asking for open source code in science (https://www.nature.com/articles/nature10836 https://www.nature.com/articles/nature10836)

I wrote a paper with co-authors asking for open source code in science (https://www.nature.com/articles/nature10836 https://www.nature.com/articles/nature10836) but this GitHub issue is stupid. Calling for retraction because you don't like the code smell is dumb. Work on the code and see what its flaws are before asking for retraction. Yes, I wish the original, apparently C code, had been released but let's fix the bugs in the code.

emerongi 7y ago
This is the correct approach. We, the "software engineers", can jump in and help. If it eventually comes out that the results were wrong then the papers should be retracted. At the moment there is no proof of that. That's the beauty of open-sourcing the code: people can help or verify. Needlessly shitting on other people's work without any proof is just disheartening to me.
- einszwei 7y ago
  I'm reminded of the idiom "Don't throw the baby out with the bathwater". And I agree that software engineers should more actively contribute their expertise here. With that said, if a research paper's main contribution is a model whose results were evaluated using a code with major bugs then retraction of the paper is only professional.
  - acqq 7y ago
    > if a research paper's main contribution is a model whose results were evaluated using buggy code There's no proof of that at all in this case. The person complaining didn't find any scientifically valid problem, only that he personally for some not cleanly stated reason doesn't like the tests provided, which is clearly stupid. Since when should any tests be such that some random guy on github must like them? If the code was used by the experts, and they maybe even used it for years, who says that these experts are in any way obliged to publish all their logs of their use of that code (which doesn't have to even exist in publishable form)? Even if all that they possibly iteratively did for potentially years could be condensed to some tests, who says that it wouldn't take too long (as in, months, years) for that? It's practically just a matter of good will that the code is published at all, in any form at all.
    - einszwei 7y ago
      Apologies if my earlier comment came off as insinuating that code used for research paper was buggy. That was not my intent.
KenanSulayman 7y ago
Not only that. It's an awful precedent to shoot down any research based on the implementation of it. That's more than just a misunderstanding; it's a fundamental misunderstanding of how research and scientific processes work. The issue claims the tests are broken because they're checking for checksums instead of actual data -- but what they really do is to check for the sanity of the implementation to be actually implementing the mathematical models that are underlying to it.
rich_sasha 7y ago
I had the displeasure of working with this sort of "scientific" code. My experience is that anything this messy is just wrong. It's not about appearance. It is so complex, messy, untestable and twisted that it is in the end incapable of giving the correct answer. The overall conclusion may be correct, who knows, but based on my experience, I do not believe, at all, any numerical predictions of this code. That does lend itself to retracting the paper.
- SiempreViernes 7y ago
  If that is indeed true you should write a paper about it, obviously if simply checking the source code for poor formatting is enough to confidently state it produces incorrect results that would be hugely important.
  - rich_sasha 7y ago
    Did you read the code? It’s not formatting or minor anal things like that. There are 10s of variables defined, functions, all without comment, cryptic names updating other cryptic names, stateful global vars updated promiscuously by everyone... This I take issue with.
acqq 7y ago
> but this GitHub issue is stupid. Yes, it is a political proclamation without any practical substance, and should be recognized as such.
- lbeltrame 7y ago
  There's some other criticism (which I won't link because the text is partisan, and it's elsewhere on HN already) with some links that point to (potential or perceived) issues: - https://github.com/mrc-ide/covid-sim/issues/116 https://github.com/mrc-ide/covid-sim/issues/116 - https://github.com/mrc-ide/covid-sim/issues/30 https://github.com/mrc-ide/covid-sim/issues/30
  - acqq 7y ago
    Both of these issues don't have any connection at all with any claim that the code could produce scientifically wrong results. The 116 is "if I use it this special way I would expect something else and not what I see" which can be simply answered with "well, don't do THAT." The simulations by design are not expected to produce exactly the same results in different runs. That's why they are simulations. The reporter tried some "partial save" and expected something else. The 30, if I understand correctly, is observed on Cray to behave in some minor detail not exactly the same as on the PC (which again doesn't have to even mean that the output is scientifically wrong). Seriously? If one is a Cray user, he can fix it, so what?
tomohawk 7y ago
The evidence shows that this model did not have good predictive power. For example, the predictions made about Sweden did not come to pass. The predictions made about the US did not come to pass - specifically, 5400 hospitalizations were predicted by by March 19 for NYC, but only 750 actually occurred - off by an order of magnitude before lockdown measures could have any impact. Did this model provide clarity and insight for the decision making process? Or, did it instead induce a false sense of panic? Here's some more info as to what this repo does and does not contain, and why what was released is not enough. https://www.aier.org/article/imperial-college-model-applied-to-sweden-yields-preposterous-results/ https://www.aier.org/article/imperial-college-model-applied-...
yread 7y ago
Yes, first find the serious issues and then take it down with vengeance. So far only serious issue is this: https://github.com/mrc-ide/covid-sim/issues/168 https://github.com/mrc-ide/covid-sim/issues/168 Looks linked to the awful style, though. When you have 700 LOC functions starting with int i, j, k, l, m, i1, i2, j2, l2, m2, tn; it does tend to happen that you reuse a variable that you shouldn't have reused
- lasagnaphil 7y ago
  These variable names are all what you typically see in mathematics-heavy numerical computation code. Sometimes it’s better to let the index symbols be just symbols rather than assigning long names; otherwise it’s going to be even more difficult to read and understand.
  - Gibbon1 7y ago
    > int i, j, k, l, m, i1, i2, j2, l2, m2, tn; Just smells like FORTRAN to me.
    - kyuudou 7y ago
      My experience with scientists who work with data is that they love FORTRAN.
      - Gibbon1 7y ago
        It's been literally 40 years since I touched FORTRAN but I can totally see why they'd like it. Does what they need without getting in the way.

emerongi 7y ago

This is the correct approach. We, the "software engineers", can jump in and help. If it eventually comes out that the results were wrong then the papers should be retracted. At the moment there is no proof of that. That's the beauty of open-sourcing the code: people can help or verify. Needlessly shitting on other people's work without any proof is just disheartening to me.

einszwei 7y ago
I'm reminded of the idiom "Don't throw the baby out with the bathwater". And I agree that software engineers should more actively contribute their expertise here. With that said, if a research paper's main contribution is a model whose results were evaluated using a code with major bugs then retraction of the paper is only professional.
- acqq 7y ago
  > if a research paper's main contribution is a model whose results were evaluated using buggy code There's no proof of that at all in this case. The person complaining didn't find any scientifically valid problem, only that he personally for some not cleanly stated reason doesn't like the tests provided, which is clearly stupid. Since when should any tests be such that some random guy on github must like them? If the code was used by the experts, and they maybe even used it for years, who says that these experts are in any way obliged to publish all their logs of their use of that code (which doesn't have to even exist in publishable form)? Even if all that they possibly iteratively did for potentially years could be condensed to some tests, who says that it wouldn't take too long (as in, months, years) for that? It's practically just a matter of good will that the code is published at all, in any form at all.
  - einszwei 7y ago
    Apologies if my earlier comment came off as insinuating that code used for research paper was buggy. That was not my intent.

KenanSulayman 7y ago

Not only that. It's an awful precedent to shoot down any research based on the implementation of it. That's more than just a misunderstanding; it's a fundamental misunderstanding of how research and scientific processes work. The issue claims the tests are broken because they're checking for checksums instead of actual data -- but what they really do is to check for the sanity of the implementation to be actually implementing the mathematical models that are underlying to it.

rich_sasha 7y ago

I had the displeasure of working with this sort of "scientific" code. My experience is that anything this messy is just wrong. It's not about appearance. It is so complex, messy, untestable and twisted that it is in the end incapable of giving the correct answer. The overall conclusion may be correct, who knows, but based on my experience, I do not believe, at all, any numerical predictions of this code. That does lend itself to retracting the paper.

SiempreViernes 7y ago
If that is indeed true you should write a paper about it, obviously if simply checking the source code for poor formatting is enough to confidently state it produces incorrect results that would be hugely important.
- rich_sasha 7y ago
  Did you read the code? It’s not formatting or minor anal things like that. There are 10s of variables defined, functions, all without comment, cryptic names updating other cryptic names, stateful global vars updated promiscuously by everyone... This I take issue with.

acqq 7y ago

> but this GitHub issue is stupid. Yes, it is a political proclamation without any practical substance, and should be recognized as such.

lbeltrame 7y ago
There's some other criticism (which I won't link because the text is partisan, and it's elsewhere on HN already) with some links that point to (potential or perceived) issues: - https://github.com/mrc-ide/covid-sim/issues/116 https://github.com/mrc-ide/covid-sim/issues/116 - https://github.com/mrc-ide/covid-sim/issues/30 https://github.com/mrc-ide/covid-sim/issues/30
- acqq 7y ago
  Both of these issues don't have any connection at all with any claim that the code could produce scientifically wrong results. The 116 is "if I use it this special way I would expect something else and not what I see" which can be simply answered with "well, don't do THAT." The simulations by design are not expected to produce exactly the same results in different runs. That's why they are simulations. The reporter tried some "partial save" and expected something else. The 30, if I understand correctly, is observed on Cray to behave in some minor detail not exactly the same as on the PC (which again doesn't have to even mean that the output is scientifically wrong). Seriously? If one is a Cray user, he can fix it, so what?

tomohawk 7y ago

The evidence shows that this model did not have good predictive power. For example, the predictions made about Sweden did not come to pass. The predictions made about the US did not come to pass - specifically, 5400 hospitalizations were predicted by by March 19 for NYC, but only 750 actually occurred - off by an order of magnitude before lockdown measures could have any impact. Did this model provide clarity and insight for the decision making process? Or, did it instead induce a false sense of panic? Here's some more info as to what this repo does and does not contain, and why what was released is not enough. https://www.aier.org/article/imperial-college-model-applied-to-sweden-yields-preposterous-results/ https://www.aier.org/article/imperial-college-model-applied-...

yread 7y ago

Yes, first find the serious issues and then take it down with vengeance. So far only serious issue is this: https://github.com/mrc-ide/covid-sim/issues/168 https://github.com/mrc-ide/covid-sim/issues/168 Looks linked to the awful style, though. When you have 700 LOC functions starting with int i, j, k, l, m, i1, i2, j2, l2, m2, tn; it does tend to happen that you reuse a variable that you shouldn't have reused

lasagnaphil 7y ago
These variable names are all what you typically see in mathematics-heavy numerical computation code. Sometimes it’s better to let the index symbols be just symbols rather than assigning long names; otherwise it’s going to be even more difficult to read and understand.
- Gibbon1 7y ago
  > int i, j, k, l, m, i1, i2, j2, l2, m2, tn; Just smells like FORTRAN to me.
  - kyuudou 7y ago
    My experience with scientists who work with data is that they love FORTRAN.
    - Gibbon1 7y ago
      It's been literally 40 years since I touched FORTRAN but I can totally see why they'd like it. Does what they need without getting in the way.