11 ms·

GitHub issue calling for retraction of Imperial College study for codebase flaws

57 points by jMyles 7y ago

jMyles 7y ago
The study in question is the Imperial College study projecting the spread of COVID-19. It used (apparently an earlier version of) this code to generate the data for its analysis. This study was, of course, the basis for lockdowns around the world.
- lidHanteyk 7y ago
  You sound like an anti-scientific snowflake, FWIW. I don't see how your position actually advances the scientific discourse; you didn't open up any PRs to improve the codebase, and without any other options, it sounds like you just want the scientists to go away and stop trying to end the pandemic.
  - d1str0 7y ago
    No he doesn’t. Asking for verification on a study is incredibly scientific.
    - lidHanteyk 7y ago
      Please, go on; what kind of verification do you want? How would you like to see this codebase improved in order to bring about that verification?
    - eejjjj82 7y ago
      No... doing your own analysis to verify the study is incredibly scientific. And the authors have provided you their code. So find a bug in the implementation which can invalidate one of their claims. Lack of tests, is not a bug.
- est 7y ago
  > This study was, of course, the basis for lockdowns around the world. Its model for China predictions seems to be quite fitting.
- toyg 7y ago
  > This study was, of course, the basis for lockdowns around the world. [citation needed] Afaik this was a lone study that might have been relevant only to the UK debate (and very late at that). Italy had already been in full lockdown for weeks before it was published, quite a few other European countries had gone in shutdown or closed borders, and obviously Asian countries had already taken countermeasures for months. Being so cavalier with the truth is, of course, why intelligent people don't take seriously a lot of anti-science attitudes.
  - ubercow13 7y ago
    The FT claim it was: https://www.ft.com/content/16764a22-69ca-11ea-a3c9-1fe6fedcca75 https://www.ft.com/content/16764a22-69ca-11ea-a3c9-1fe6fedcc...
    - toyg 7y ago
      "The UK and the US". It's in the title. As I said, "around the world" they were already taking countermeasures.
    - fceccon 7y ago
      The timelime doesn't match tho. The article is dated 19/03 and mentions a study released the previous week (so 12/03 or late). The italian lockdown started on the 22/02 [0] for a few towns, and a couple of weeks later was extended to the entire peninsula. [0] https://metro.co.uk/2020/02/25/towns-italy-lockdown-coronavirus-12298246/ https://metro.co.uk/2020/02/25/towns-italy-lockdown-coronavi...
      - ubercow13 7y ago
        Indeed, I forgot how late the UK/US policy changed. The claim that this study affected policy around the world doesn't seem to hold up.
  - pnako 7y ago
    According to Le Monde (~ French NY Times), it's this research that prompted the French government to urgently implement this "lock down" solution. It was criticized by some French scientists in the government's own expert group. But I don't really blame Ferguson. If President Macron listens to him, or Alex Jones, or any other crank or random guy with an opinion and a model, instead of his own experts, he is the one to blame. I think the culture of the "safety principle" and excessive caution is also to blame. https://www.lemonde.fr/planete/article/2020/03/15/coronavirus-les-simulations-alarmantes-des-epidemiologistes-pour-la-france_6033149_3244.html https://www.lemonde.fr/planete/article/2020/03/15/coronaviru...
- deleted 7y ago
  [deleted]
- tastroder 7y ago
  I've honestly lost track of all the simulation papers over the last few weeks, is there anything in particular in this one that's not reflected by any one of those? I don't think I've seen a single one that didn't come to suggesting a scale that calls for some lockdown requirements.
- lostmyoldone 7y ago
  At least in Europe it wasn't a major part of decisions. When it was published, it mostly corroborated what people here were already seeing, that urgent care facilities got - or were going to get - overwhelmed in a few weeks if contact tracing failed to contain close to all cases. There is also a "control" group of sorts in us Swedes, as the Swedish health authority choose to act differently, initially. Their recommendation as of now is essentially equivalent to quarantine, except for non-adults. It's important to understand that it is really to instigate anything like a curfew in Sweden, at least unless we get attacked military by another country. The laws as they stand don't really allow that. So the government is also somewhat limited in what they can do. The situation here is that urgent care has been overwhelmed for about a month, and a lot more people have died or been gravely ill than during even really bad flu seasons. We're also probably far from done yet, as it's much easier to get people to stay indoors when it's cold and rainy outside, and as its been getting warmer more people have been disrespecting the recommendations.
KenoFischer 7y ago
I think this sets a bad example. In many fields the current standard is that code isn't published at all. If bad coding practices will cause you to get bullied into retracting your paper, that's just another argument for why people won't publish their code. Of course if somebody does find a big that invalidates the results of the paper given the stated assumptions, that's another matter, but that doesn't seem to be what's alleged here.
- nradov 7y ago
  The standard needs to change. Publication of code and data ought to be required.
  - denzil_correa 7y ago
    Why exactly? We could also probably require a website to test the application, a mobile app on all platforms and then a native OS apps on all platforms. I apologise for being facetious but writing code is a serious, specialized task. It should be left to the people who specialise in it, scientists aren't the specialists for it. I think it's important to understand that we only require information to generate algorithms written in the paper. As such, assumptions and limitations should be listed enough to generate the material in the paper. One way to do so is code and data, it is certainly not the only way.
    - toyg 7y ago
      I empathise with what you say, but it's also true that maybe things could be better. We don't expect universities to build their own laboratories and halls of residence from first principles - they will call on property developers, architects, and contractors. It would be good if there were more IT services, built by specialists, that academics could tap when required - so they would spend more time advancing their fields of expertise and less time writing mediocre code. Things like PythonAnywhere / Notebooks, Lambda etc are a step forward, but there is still ample space for advancement.
      - 7y ago
andrepd 7y ago
Of all the things wrong with this study, bad code is probably the least (bar actual bugs which mess with results). In fact, it's par on course for code in non-cs academia. There's such a thing as the "physicist code" stereotype.
- pnako 7y ago
  It's generally not used to push policy affecting billions of people, though. But as I said in another comment, let's blame whoever listened to that garbage fire of a model, not the model itself. We don't listen to Minecraft modders for structural engineering advice either, and we did, they would not be the ones to blame for collapsing buildings.
  - metalliqaz 7y ago
    The policy is mostly a reaction to what happened in Wuhan, Iran, and Italy. Especially Italy. The mass graves filling up are a pretty good motivator, as it turns out.[1] [1] https://www.theguardian.com/world/2020/mar/12/coronavirus-iran-mass-graves-qom https://www.theguardian.com/world/2020/mar/12/coronavirus-ir...
    - Gibbon1 7y ago
      One could say the model was experimentally verified in Wuhan, Iran, Italy and New York.
rvz 7y ago
Well I would rather have the code already available for transparency and then we can discuss about the quality later than it not to be open-source at all. I wouldn't expect research scientists to write an elegant and well structured program that ticks all the coding practices boxes at the first time as they are not "software engineers" unless that's their area of research or expertise. They want to present their results and the "code quality" comes secondary to them which can be done later. Surely the Linux source code wasn't cleanly structured in its first open-source release. However, a quick skim at the source, one may suggest that the authors were writing C++ in a style of a C programmer. Maybe one can run a clang-analyzer on the source to find all sorts of issues, I guess.
fergonco 7y ago
If only all the studies that do not publish their code got half this level of criticism.
- s9w 7y ago
  Well most of science doesn't matter. This was used to introduce cataclysmic policy changes.
  - toyg 7y ago
    This was hardly the only element that UK policymakers had to take into account. By the time this study was published, numbers were already accelerating and most of Europe was already in lockdown.
    - s9w 7y ago
      Hard to know really. I think it's likely the big question at the time was how bad will this get. And they did run and use this simulation. Frankly people who are willing to use this for anything should have no say in anything more important than their own papers. There is the great (Carl Sagan?) quote "extraordinary claims require extraordinary evidence". If other parts of the puzzle are of the same quality, I want to see heads rolling. The PCR test for example.
  - lostmyoldone 7y ago
    What position are you taking here, and which cataclysmic policy changes are you talking about?
    - s9w 7y ago
      Well, all the corona laws that have been passed - or did I misunderstand?
emerongi 7y ago
You're trying to apply software development techniques to a discipline where testing is usually done in a much different way. Yes, we should all try to be perfect and write the most magnificent code with the most bulletproof testing methods, but reality is different. How about you write tests that clearly prove that the results from this simulation are absolutely wrong? The code is right there! And you're a "software engineer"! Then start talking about retracting papers.
- Gibbon1 7y ago
  > a discipline where testing is usually done in a much different way This reminds me of the horror mainstream code monkeys express when they see the source code for automotive firmware. Thing is automotive people functional test the shit out of everything. And they don't let jr developers refactor proven good code because they don't like the way it looks.
jgrahamc 7y ago
I wrote a paper with co-authors asking for open source code in science (https://www.nature.com/articles/nature10836 https://www.nature.com/articles/nature10836) but this GitHub issue is stupid. Calling for retraction because you don't like the code smell is dumb. Work on the code and see what its flaws are before asking for retraction. Yes, I wish the original, apparently C code, had been released but let's fix the bugs in the code.
- emerongi 7y ago
  This is the correct approach. We, the "software engineers", can jump in and help. If it eventually comes out that the results were wrong then the papers should be retracted. At the moment there is no proof of that. That's the beauty of open-sourcing the code: people can help or verify. Needlessly shitting on other people's work without any proof is just disheartening to me.
  - einszwei 7y ago
    I'm reminded of the idiom "Don't throw the baby out with the bathwater". And I agree that software engineers should more actively contribute their expertise here. With that said, if a research paper's main contribution is a model whose results were evaluated using a code with major bugs then retraction of the paper is only professional.
    - acqq 7y ago
      > if a research paper's main contribution is a model whose results were evaluated using buggy code There's no proof of that at all in this case. The person complaining didn't find any scientifically valid problem, only that he personally for some not cleanly stated reason doesn't like the tests provided, which is clearly stupid. Since when should any tests be such that some random guy on github must like them? If the code was used by the experts, and they maybe even used it for years, who says that these experts are in any way obliged to publish all their logs of their use of that code (which doesn't have to even exist in publishable form)? Even if all that they possibly iteratively did for potentially years could be condensed to some tests, who says that it wouldn't take too long (as in, months, years) for that? It's practically just a matter of good will that the code is published at all, in any form at all.
ageitgey 7y ago
Academic code has always been a mess (from the standpoint of professional software developers). Anyone who has ever worked with code from academia knows this. If you think this code is messy, check out any random Matlab simulation strung together by any random series of grad students who aren't professional developers in any random lab. People in industry don't always do better, either. Look at the notebook code shared on Github by Kevin Systrom for the rt.live site. It's just as messy. But the point is that he shared it and that helps everyone see what is going on and lets anyone improve on the work. Publicly sharing model code on Github is a recent development and is amazing for everyone. It gets the work out faster, allows for feedback on methodology and coding style and generally lets everyone get smarter over time. We shouldn't discourage this. Posting retraction demands in Github issues based on coding style is not helpful and shows a lack of communication skills. What is helpful is providing useful, specific bug reports, submitting improvements, or sharing your own, improved model. If you want to demand a retraction, do it via alternate means. Remember, this crisis appeared suddenly on the world stage and whoever had a model sitting around got thrust into the forefront of public discussion. But scientists don't make policy. Policy makers make policy. A github issue is not the place to blame someone for a policy you disagree with. That only discourages people from sharing their work which would be a loss for everyone.
- toyg 7y ago
  > shows a lack of communication skills Your reading is very charitable. The political debate is a complete gutter, and everyone is ready to jump at absolutely anything that can advance his argument in the slightest. As you can see in this very thread, jMyles is hardly unbiased on the matter. He's just playing politics, code standards are an excuse.
- neuronic 7y ago
  I worked on an academic open source Java project following my BSc and after years of MSc + industry experience as backend dev, I cringe really hard at what I produced back then. So hard in fact, that I am about to invest time to fix that mess (because I like the project).
jspaetzel 7y ago
This is such a dangerous precedent. This type of call out will only discourage the release of source code making legitimate peer review more difficult.
pnako 7y ago
Blame the politicians who listened to some random guy with some code instead of the random guy with some code himself.
- metalliqaz 7y ago
  "random guy" lol
ubercow13 7y ago
What's wrong with comparing the code results to a hash of a known result?
- metreo 7y ago
  You aren't validating the result its self!
  - ubercow13 7y ago
    It is if you know that result is valid, for example if you have validated it some other way and this is a regression test, or you calculated it by hand.
mikekchar 7y ago
The last comment in the issue (as of this posting) posted to a comment from John Carmack on Twitter: https://twitter.com/ID_AA_Carmack/status/1254872368763277313 https://twitter.com/ID_AA_Carmack/status/1254872368763277313 I specialise in legacy code. I've seen my fair share of abysmal code that works. This is pretty awful, but far from the worst I've ever seen in my career -- even in systems where public safety was critical. I've not looked at this code in any real detail, but I suspect John Carmack (who has doubtless seen a fair amount of complex C code in his life) has it right. On the plus side, if anyone wants to practice refactoring gnarly C code: here's your opportunity.
- acqq 7y ago
  The twits point that Carmack personally worked on that very code (to prepare it for the Github release)! He explicitly writes there: "it turned out that it fared a lot better going through the gauntlet of code analysis tools I hit it with than a lot of more modern code. There is something to be said for straightforward C code. Bugs were found and fixed, but generally in paths that weren't enabled or hit." "the performance scaling using OpenMP was already pretty good, and this was not the place for one of my dramatic system refactorings. Mostly, I was just a code janitor for a few weeks, but I was happy to be able to help a little." and "I can’t vouch for the actual algorithms, but the software engineering seems fine." As shown, he even points that the believes "a lot of more modern code" would have been worse than that "straightforward C", which matches my experiences.
- cjhopman 7y ago
  This is typical academic code. The response is utterly embarrassing for the software engineering profession.
cjhopman 7y ago
This is the only time I've ever actually been embarrassed to be part of this profession. Maybe other cases i can just say it's limited to this other subgroup that doesn't include me...
metalliqaz 7y ago
This github issue sounds a lot like it is politically motivated. He is clearly trying to make the leap from "I don't like the tests in this project" to "public policy is wrong." No actual evidence, just cranky poo-pooing on the project.
sm_1024 7y ago
The title is misleading, the GitHub issue doesn't actually point out any flaws in the codebase.
pythondev64 7y ago
Just because unit tests aren't provded in the codebase, it doesn't mean testing hasn't been done. Quite often it's difficult to model certain things in tests, e.g. Linux kernel drivers, embedded devices etc. However, it seems that the author of the GitHub issue has personal/political problems with the Imperial researchers because of the lockdown: https://pastebin.com/LadbM3E1 https://pastebin.com/LadbM3E1.
- tastroder 7y ago
  Thanks for posting that. @OP: So let's get this straight, you created that GitHub issue within one hour of seeing the repository, rallied whatever Discord server that is to join your cause and put it on HN for reach? As an academic and software engineer, please point me to where I can file an issue for you to retract your issue.
yufeng66 7y ago
The issue is crazy. It basically come down to there is no unit test? Unit test traditionally is not done for numerical code in Fortran or C. We sent people to the moon on code without unit test. We build atomic bombs on code without unit test. It doesn’t mean the code is not tested. Those numerical code are just tested in a different way. Also remember, all models are wrong, some are useful. Is this particular model useful? Probably.
chkaloon 7y ago
I suppose it's only fair that if software engineers think they can play epidemiologists, that epidemiologists could think they can play software engineers.
tjmtjmtjm 7y ago
Welp, this is the best argument for regulating the title of “Engineer” I’ve ever seen.
tgsovlerkhgsel 7y ago
> The tests in this project, being limited to broad, "smoke test"-style assertions Doesn't this put the code far above most academic code by having some tests?
nodamage 7y ago
It's worth noting that this issue was posted by a moderator of the /r/LockdownSkepticism/ subreddit to push a particular political agenda, and now this repo is being brigaded by users from said subreddit.
boublepop 6y ago
This is politically motivated and just plain dumb. Someone with the intention of finding holes posted it with the assumption that the code must be wrong because it’s build in a way that makes it hard for that individual to find potential errors. That doesn’t mean that there are critical errors and this “issue” should not have been submitted unless there are, but the author is impatient and has a foregone conclusion so the issue was submitted and pushed in social media instead of doing the actual work of looking for critical errors. And likely there are no critical errors, only minor ones as is typical of heavy simulation code bases. You don’t do testing via unit and integration tests, you do testing through simulating known systems and building confidence in code correctness over time. And by reviewing the mathematics behind the models, and reviewing that the code matches the math. That’s harder for outsiders who want to discredit you to dig into that and point fingers at it. But not at all impossible, however in the science community we don’t start pointing fingers until we have actual criticism to back our critique.

57 points by jMyles 7y ago

The study in question is the Imperial College study projecting the spread of COVID-19. It used (apparently an earlier version of) this code to generate the data for its analysis. This study was, of course, the basis for lockdowns around the world.

lidHanteyk 7y ago
You sound like an anti-scientific snowflake, FWIW. I don't see how your position actually advances the scientific discourse; you didn't open up any PRs to improve the codebase, and without any other options, it sounds like you just want the scientists to go away and stop trying to end the pandemic.
- d1str0 7y ago
  No he doesn’t. Asking for verification on a study is incredibly scientific.
  - lidHanteyk 7y ago
    Please, go on; what kind of verification do you want? How would you like to see this codebase improved in order to bring about that verification?
  - eejjjj82 7y ago
    No... doing your own analysis to verify the study is incredibly scientific. And the authors have provided you their code. So find a bug in the implementation which can invalidate one of their claims. Lack of tests, is not a bug.
est 7y ago
> This study was, of course, the basis for lockdowns around the world. Its model for China predictions seems to be quite fitting.
toyg 7y ago
> This study was, of course, the basis for lockdowns around the world. [citation needed] Afaik this was a lone study that might have been relevant only to the UK debate (and very late at that). Italy had already been in full lockdown for weeks before it was published, quite a few other European countries had gone in shutdown or closed borders, and obviously Asian countries had already taken countermeasures for months. Being so cavalier with the truth is, of course, why intelligent people don't take seriously a lot of anti-science attitudes.
- ubercow13 7y ago
  The FT claim it was: https://www.ft.com/content/16764a22-69ca-11ea-a3c9-1fe6fedcca75 https://www.ft.com/content/16764a22-69ca-11ea-a3c9-1fe6fedcc...
  - toyg 7y ago
    "The UK and the US". It's in the title. As I said, "around the world" they were already taking countermeasures.
  - fceccon 7y ago
    The timelime doesn't match tho. The article is dated 19/03 and mentions a study released the previous week (so 12/03 or late). The italian lockdown started on the 22/02 [0] for a few towns, and a couple of weeks later was extended to the entire peninsula. [0] https://metro.co.uk/2020/02/25/towns-italy-lockdown-coronavirus-12298246/ https://metro.co.uk/2020/02/25/towns-italy-lockdown-coronavi...
    - ubercow13 7y ago
      Indeed, I forgot how late the UK/US policy changed. The claim that this study affected policy around the world doesn't seem to hold up.
- pnako 7y ago
  According to Le Monde (~ French NY Times), it's this research that prompted the French government to urgently implement this "lock down" solution. It was criticized by some French scientists in the government's own expert group. But I don't really blame Ferguson. If President Macron listens to him, or Alex Jones, or any other crank or random guy with an opinion and a model, instead of his own experts, he is the one to blame. I think the culture of the "safety principle" and excessive caution is also to blame. https://www.lemonde.fr/planete/article/2020/03/15/coronavirus-les-simulations-alarmantes-des-epidemiologistes-pour-la-france_6033149_3244.html https://www.lemonde.fr/planete/article/2020/03/15/coronaviru...
deleted 7y ago
[deleted]
tastroder 7y ago
I've honestly lost track of all the simulation papers over the last few weeks, is there anything in particular in this one that's not reflected by any one of those? I don't think I've seen a single one that didn't come to suggesting a scale that calls for some lockdown requirements.
lostmyoldone 7y ago
At least in Europe it wasn't a major part of decisions. When it was published, it mostly corroborated what people here were already seeing, that urgent care facilities got - or were going to get - overwhelmed in a few weeks if contact tracing failed to contain close to all cases. There is also a "control" group of sorts in us Swedes, as the Swedish health authority choose to act differently, initially. Their recommendation as of now is essentially equivalent to quarantine, except for non-adults. It's important to understand that it is really to instigate anything like a curfew in Sweden, at least unless we get attacked military by another country. The laws as they stand don't really allow that. So the government is also somewhat limited in what they can do. The situation here is that urgent care has been overwhelmed for about a month, and a lot more people have died or been gravely ill than during even really bad flu seasons. We're also probably far from done yet, as it's much easier to get people to stay indoors when it's cold and rainy outside, and as its been getting warmer more people have been disrespecting the recommendations.

KenoFischer 7y ago

I think this sets a bad example. In many fields the current standard is that code isn't published at all. If bad coding practices will cause you to get bullied into retracting your paper, that's just another argument for why people won't publish their code. Of course if somebody does find a big that invalidates the results of the paper given the stated assumptions, that's another matter, but that doesn't seem to be what's alleged here.

nradov 7y ago
The standard needs to change. Publication of code and data ought to be required.
- denzil_correa 7y ago
  Why exactly? We could also probably require a website to test the application, a mobile app on all platforms and then a native OS apps on all platforms. I apologise for being facetious but writing code is a serious, specialized task. It should be left to the people who specialise in it, scientists aren't the specialists for it. I think it's important to understand that we only require information to generate algorithms written in the paper. As such, assumptions and limitations should be listed enough to generate the material in the paper. One way to do so is code and data, it is certainly not the only way.
  - toyg 7y ago
    I empathise with what you say, but it's also true that maybe things could be better. We don't expect universities to build their own laboratories and halls of residence from first principles - they will call on property developers, architects, and contractors. It would be good if there were more IT services, built by specialists, that academics could tap when required - so they would spend more time advancing their fields of expertise and less time writing mediocre code. Things like PythonAnywhere / Notebooks, Lambda etc are a step forward, but there is still ample space for advancement.
    - 7y ago

andrepd 7y ago

Of all the things wrong with this study, bad code is probably the least (bar actual bugs which mess with results). In fact, it's par on course for code in non-cs academia. There's such a thing as the "physicist code" stereotype.

pnako 7y ago
It's generally not used to push policy affecting billions of people, though. But as I said in another comment, let's blame whoever listened to that garbage fire of a model, not the model itself. We don't listen to Minecraft modders for structural engineering advice either, and we did, they would not be the ones to blame for collapsing buildings.
- metalliqaz 7y ago
  The policy is mostly a reaction to what happened in Wuhan, Iran, and Italy. Especially Italy. The mass graves filling up are a pretty good motivator, as it turns out.[1] [1] https://www.theguardian.com/world/2020/mar/12/coronavirus-iran-mass-graves-qom https://www.theguardian.com/world/2020/mar/12/coronavirus-ir...
  - Gibbon1 7y ago
    One could say the model was experimentally verified in Wuhan, Iran, Italy and New York.

rvz 7y ago

Well I would rather have the code already available for transparency and then we can discuss about the quality later than it not to be open-source at all. I wouldn't expect research scientists to write an elegant and well structured program that ticks all the coding practices boxes at the first time as they are not "software engineers" unless that's their area of research or expertise. They want to present their results and the "code quality" comes secondary to them which can be done later. Surely the Linux source code wasn't cleanly structured in its first open-source release. However, a quick skim at the source, one may suggest that the authors were writing C++ in a style of a C programmer. Maybe one can run a clang-analyzer on the source to find all sorts of issues, I guess.

fergonco 7y ago

If only all the studies that do not publish their code got half this level of criticism.

s9w 7y ago
Well most of science doesn't matter. This was used to introduce cataclysmic policy changes.
- toyg 7y ago
  This was hardly the only element that UK policymakers had to take into account. By the time this study was published, numbers were already accelerating and most of Europe was already in lockdown.
  - s9w 7y ago
    Hard to know really. I think it's likely the big question at the time was how bad will this get. And they did run and use this simulation. Frankly people who are willing to use this for anything should have no say in anything more important than their own papers. There is the great (Carl Sagan?) quote "extraordinary claims require extraordinary evidence". If other parts of the puzzle are of the same quality, I want to see heads rolling. The PCR test for example.
- lostmyoldone 7y ago
  What position are you taking here, and which cataclysmic policy changes are you talking about?
  - s9w 7y ago
    Well, all the corona laws that have been passed - or did I misunderstand?

emerongi 7y ago

You're trying to apply software development techniques to a discipline where testing is usually done in a much different way. Yes, we should all try to be perfect and write the most magnificent code with the most bulletproof testing methods, but reality is different. How about you write tests that clearly prove that the results from this simulation are absolutely wrong? The code is right there! And you're a "software engineer"! Then start talking about retracting papers.

Gibbon1 7y ago
> a discipline where testing is usually done in a much different way This reminds me of the horror mainstream code monkeys express when they see the source code for automotive firmware. Thing is automotive people functional test the shit out of everything. And they don't let jr developers refactor proven good code because they don't like the way it looks.

jgrahamc 7y ago

I wrote a paper with co-authors asking for open source code in science (https://www.nature.com/articles/nature10836 https://www.nature.com/articles/nature10836) but this GitHub issue is stupid. Calling for retraction because you don't like the code smell is dumb. Work on the code and see what its flaws are before asking for retraction. Yes, I wish the original, apparently C code, had been released but let's fix the bugs in the code.

emerongi 7y ago
This is the correct approach. We, the "software engineers", can jump in and help. If it eventually comes out that the results were wrong then the papers should be retracted. At the moment there is no proof of that. That's the beauty of open-sourcing the code: people can help or verify. Needlessly shitting on other people's work without any proof is just disheartening to me.
- einszwei 7y ago
  I'm reminded of the idiom "Don't throw the baby out with the bathwater". And I agree that software engineers should more actively contribute their expertise here. With that said, if a research paper's main contribution is a model whose results were evaluated using a code with major bugs then retraction of the paper is only professional.
  - acqq 7y ago
    > if a research paper's main contribution is a model whose results were evaluated using buggy code There's no proof of that at all in this case. The person complaining didn't find any scientifically valid problem, only that he personally for some not cleanly stated reason doesn't like the tests provided, which is clearly stupid. Since when should any tests be such that some random guy on github must like them? If the code was used by the experts, and they maybe even used it for years, who says that these experts are in any way obliged to publish all their logs of their use of that code (which doesn't have to even exist in publishable form)? Even if all that they possibly iteratively did for potentially years could be condensed to some tests, who says that it wouldn't take too long (as in, months, years) for that? It's practically just a matter of good will that the code is published at all, in any form at all.

ageitgey 7y ago

Academic code has always been a mess (from the standpoint of professional software developers). Anyone who has ever worked with code from academia knows this. If you think this code is messy, check out any random Matlab simulation strung together by any random series of grad students who aren't professional developers in any random lab. People in industry don't always do better, either. Look at the notebook code shared on Github by Kevin Systrom for the rt.live site. It's just as messy. But the point is that he shared it and that helps everyone see what is going on and lets anyone improve on the work. Publicly sharing model code on Github is a recent development and is amazing for everyone. It gets the work out faster, allows for feedback on methodology and coding style and generally lets everyone get smarter over time. We shouldn't discourage this. Posting retraction demands in Github issues based on coding style is not helpful and shows a lack of communication skills. What is helpful is providing useful, specific bug reports, submitting improvements, or sharing your own, improved model. If you want to demand a retraction, do it via alternate means. Remember, this crisis appeared suddenly on the world stage and whoever had a model sitting around got thrust into the forefront of public discussion. But scientists don't make policy. Policy makers make policy. A github issue is not the place to blame someone for a policy you disagree with. That only discourages people from sharing their work which would be a loss for everyone.

toyg 7y ago
> shows a lack of communication skills Your reading is very charitable. The political debate is a complete gutter, and everyone is ready to jump at absolutely anything that can advance his argument in the slightest. As you can see in this very thread, jMyles is hardly unbiased on the matter. He's just playing politics, code standards are an excuse.
neuronic 7y ago
I worked on an academic open source Java project following my BSc and after years of MSc + industry experience as backend dev, I cringe really hard at what I produced back then. So hard in fact, that I am about to invest time to fix that mess (because I like the project).

jspaetzel 7y ago

This is such a dangerous precedent. This type of call out will only discourage the release of source code making legitimate peer review more difficult.

pnako 7y ago

Blame the politicians who listened to some random guy with some code instead of the random guy with some code himself.

metalliqaz 7y ago
"random guy" lol

ubercow13 7y ago

What's wrong with comparing the code results to a hash of a known result?

metreo 7y ago
You aren't validating the result its self!
- ubercow13 7y ago
  It is if you know that result is valid, for example if you have validated it some other way and this is a regression test, or you calculated it by hand.

mikekchar 7y ago

The last comment in the issue (as of this posting) posted to a comment from John Carmack on Twitter: https://twitter.com/ID_AA_Carmack/status/1254872368763277313 https://twitter.com/ID_AA_Carmack/status/1254872368763277313 I specialise in legacy code. I've seen my fair share of abysmal code that works. This is pretty awful, but far from the worst I've ever seen in my career -- even in systems where public safety was critical. I've not looked at this code in any real detail, but I suspect John Carmack (who has doubtless seen a fair amount of complex C code in his life) has it right. On the plus side, if anyone wants to practice refactoring gnarly C code: here's your opportunity.

acqq 7y ago
The twits point that Carmack personally worked on that very code (to prepare it for the Github release)! He explicitly writes there: "it turned out that it fared a lot better going through the gauntlet of code analysis tools I hit it with than a lot of more modern code. There is something to be said for straightforward C code. Bugs were found and fixed, but generally in paths that weren't enabled or hit." "the performance scaling using OpenMP was already pretty good, and this was not the place for one of my dramatic system refactorings. Mostly, I was just a code janitor for a few weeks, but I was happy to be able to help a little." and "I can’t vouch for the actual algorithms, but the software engineering seems fine." As shown, he even points that the believes "a lot of more modern code" would have been worse than that "straightforward C", which matches my experiences.
cjhopman 7y ago
This is typical academic code. The response is utterly embarrassing for the software engineering profession.

cjhopman 7y ago

This is the only time I've ever actually been embarrassed to be part of this profession. Maybe other cases i can just say it's limited to this other subgroup that doesn't include me...

metalliqaz 7y ago

This github issue sounds a lot like it is politically motivated. He is clearly trying to make the leap from "I don't like the tests in this project" to "public policy is wrong." No actual evidence, just cranky poo-pooing on the project.

sm_1024 7y ago

The title is misleading, the GitHub issue doesn't actually point out any flaws in the codebase.

pythondev64 7y ago

Just because unit tests aren't provded in the codebase, it doesn't mean testing hasn't been done. Quite often it's difficult to model certain things in tests, e.g. Linux kernel drivers, embedded devices etc. However, it seems that the author of the GitHub issue has personal/political problems with the Imperial researchers because of the lockdown: https://pastebin.com/LadbM3E1 https://pastebin.com/LadbM3E1.

tastroder 7y ago
Thanks for posting that. @OP: So let's get this straight, you created that GitHub issue within one hour of seeing the repository, rallied whatever Discord server that is to join your cause and put it on HN for reach? As an academic and software engineer, please point me to where I can file an issue for you to retract your issue.

yufeng66 7y ago

The issue is crazy. It basically come down to there is no unit test? Unit test traditionally is not done for numerical code in Fortran or C. We sent people to the moon on code without unit test. We build atomic bombs on code without unit test. It doesn’t mean the code is not tested. Those numerical code are just tested in a different way. Also remember, all models are wrong, some are useful. Is this particular model useful? Probably.

chkaloon 7y ago

I suppose it's only fair that if software engineers think they can play epidemiologists, that epidemiologists could think they can play software engineers.

tjmtjmtjm 7y ago

Welp, this is the best argument for regulating the title of “Engineer” I’ve ever seen.

tgsovlerkhgsel 7y ago

> The tests in this project, being limited to broad, "smoke test"-style assertions Doesn't this put the code far above most academic code by having some tests?

nodamage 7y ago

It's worth noting that this issue was posted by a moderator of the /r/LockdownSkepticism/ subreddit to push a particular political agenda, and now this repo is being brigaded by users from said subreddit.

boublepop 6y ago

This is politically motivated and just plain dumb. Someone with the intention of finding holes posted it with the assumption that the code must be wrong because it’s build in a way that makes it hard for that individual to find potential errors. That doesn’t mean that there are critical errors and this “issue” should not have been submitted unless there are, but the author is impatient and has a foregone conclusion so the issue was submitted and pushed in social media instead of doing the actual work of looking for critical errors. And likely there are no critical errors, only minor ones as is typical of heavy simulation code bases. You don’t do testing via unit and integration tests, you do testing through simulating known systems and building confidence in code correctness over time. And by reviewing the mathematics behind the models, and reviewing that the code matches the math. That’s harder for outsiders who want to discredit you to dig into that and point fingers at it. But not at all impossible, however in the science community we don’t start pointing fingers until we have actual criticism to back our critique.