6 ms·

Imperial College London have released their Covid-19 epidemic simulation

57 points by bencollier49 6y ago

asplake 6y ago
Alumnus here, it’s Imperial College, London. It is a university (in its own right), but always goes by that name.
- bencollier49 6y ago
  I knew that; I went to Kings! Apparently I haven't woken up properly this morning. Title changed.
  - dancedar 6y ago
    Alum of both, it's comma-less icl on paperwork, website, rankings, and research, see ie https://www.imperial.ac.uk/mrc-global-infectious-disease-analysis/covid-19/ https://www.imperial.ac.uk/mrc-global-infectious-disease-ana... bottom their address.
    - bencollier49 6y ago
      Ah but I somehow managed to write "Imperial University". I have now had a coffee, and changed the title.
- howd00 6y ago
  Only since 2007. Uni of London before that for 100 years. Used to play them at sports as part of the ULU league.
  - asplake 6y ago
    Yes it was part of the University of London, but always “Imperial College”. As far as I’m aware never “Imperial University” as the title initially had it here.
  - gvjddbnvdrbv 6y ago
    Individual colleges were effectively independent universities for many years before. The University of London umbrella was a shell.
  - OJFord 6y ago
    No, Imperial's royal charter (what's needed to be 'a university' and award degrees in your own right) dates to 1907. I believe that's true of most of UoL's constituents, just not the smaller ones perhaps like SOAS or other field-focused colleges.
notkaiho 6y ago
I'm no expert in C++ so not sure if it's meant to look like that, but that code is making my brain hurt.
- easytiger 6y ago
  it is not meant to look like that. The spacing/tabbing is all over the place.
- OJFord 6y ago
  'Academic code' is infamous for having that effect.
- gpderetta 6y ago
  In my very limited experience, that's what a lot of code coming from academia looks like.
  - Gibbon1 6y ago
    That code reminds me of a program that emulated an old Wyse Color Terminal.
- mshook 6y ago
  Just argument parsing makes my eyes bleed... https://github.com/mrc-ide/covid-sim/blob/master/src/CovidSim.cpp#L146 https://github.com/mrc-ide/covid-sim/blob/master/src/CovidSi... That stuff is priceless: else if (argv[i][1] == 'C' && argv[i][2] == 'L' && argv[i][3] == 'P' && argv[i][4] == '1' && argv[i][5] == ':') I guess string comparisons are complicated. I also fail to see why they used ':' as the separator and why they didn't use a proper library to parse argv...
  - tasogare 6y ago
    Because when you have only a very limited number of arguments it’s easier to do it that way than selecting a lib, adding it to the project, reading the doc, trying it and integrating it for real. I add the issue in C#, and after trying 2 or 3 libs (including one by Microsoft) I just gave up.
djhworld 6y ago
Interesting tidbit for HN, Jobn Carmack helped a little with this project apparently https://twitter.com/ID_AA_Carmack/status/1254872368763277313 https://twitter.com/ID_AA_Carmack/status/1254872368763277313
- antupis 6y ago
  Yeah and he said that code is fine so I am trusting him.
abainbridge 6y ago
This code is great fun. Start here to explore the horror: https://github.com/mrc-ide/covid-sim/blob/master/src/CovidSim.cpp https://github.com/mrc-ide/covid-sim/blob/master/src/CovidSi... I like the way InitModel() crashes (I think) if a global pointer called bmh (short for bitmap header) isn't first initialized by calling InitBMHead() from Bitmap.cpp. I guess it's obvious to academics with giant brains that InitModel() depends on a bitmap existing. But it gets worse - the pointer is declared in CovidSim.cpp but Bitmap.cpp reaches across the module boundary to initialize it. Sweet. Edit: I feel bad now. It's great that this code was made public. I should be encouraging that. I think academics writing code for things like this should probably take an undergrad course on software engineering. I think the one that taught me about Abstract Data Types would be most beneficial the author(s) of this code.
- deleted 6y ago
  [deleted]
- JauntyHatAngle 6y ago
  I appreciate you pointing out the flaws, but I fear they sarcasm is not really warranted. Shit code or not, you can call it out without denigrating the guy. It's nice they aired their source code out for people to look at, least we can do is to critique without snark.
  - abainbridge 6y ago
    OK, fair point. Snarkiest comment removed. My comments weren't meant to be serious. The problem I pointed out isn't actually a bug, it's just a style of programming that makes the code harder to read/work on than it needs to be.
- deleted 6y ago
  [deleted]
- cataphract 6y ago
  I've seen a lot of code coming out of academia that exhibits a similar complete lack of structure and completely ignores the most basic software development best practices (such as having tests). There are exceptions, and research software engineers are a thing, but unsurprisingly most academics focus their careers on the science rather than the code.
lbeltrame 6y ago
Some people have raised issues with the model[1][2], or rather, the software implementation of it. There's also a (flagged) submission on HN discussing this[3] referencing [1]. [1] (warning: possibly partisan link) https://lockdownsceptics.org/code-review-of-fergusons-model/ https://lockdownsceptics.org/code-review-of-fergusons-model/ [2] https://github.com/mrc-ide/covid-sim/issues/165 https://github.com/mrc-ide/covid-sim/issues/165 [3] https://news.ycombinator.com/item?id=23099212 https://news.ycombinator.com/item?id=23099212
- te_chris 6y ago
  That issue is unhelpful. Yes, no good tests, but how about suggesting ways it can actually be improved? Why are devs such bores when it comes to things like this? They have released the model, review it and suggest improvements! Don't grandstand "We, the undersigned etc etc" as if that's going to help improve the codebase in the slightest. Carmack is OK with it, he's put his name to reviewing it. That's not to excuse the bad testing, but he hasn't thrown his hands up and run away. He worked with them constructively to make it possible for us to even see it! Be more like Carmack. EDIT: Also be like this person: https://github.com/mrc-ide/covid-sim/issues/161 https://github.com/mrc-ide/covid-sim/issues/161 Helpful, constructive and the developers engaged with them.
  - JauntyHatAngle 6y ago
    I fear that this area is too divisive currently to allow for normal discourse. The downvoting in these kinds of topics have been atrocious on hacker News the last few weeks.
    - Gibbon1 6y ago
      HN is one of the increasing fewer places online where covid19 deniers can congregate.
      - lbeltrame 6y ago
        I personally don't like much the term "deniers", because there are a lot of unknowns on this virus and what it does, and there is no agreement on many fronts. Also, doing these generalizations groups together people with very questionable theories ("It's the 5G") with others that have more nuanced criticism. Personally (and yes, I am a scientist) try to look up whatever is said in the media, either by journalists or experts, no matter if the results end up matching 100% what it is said (often it is less, and on some cases there is no match). I think there should be fairly high standards of scientific rigor even in published code, especially if this might impact public policy actions, like we should expect high rigor in biological and epidemiological studies.
kidintech 6y ago
I guess confidence really is key when making code public...
buboard 6y ago
.
- toyg 6y ago
  > Its results is as useful as simcity is for mayors. I bet the quality of local administration would go through the roof if all mayoral candidates were forced to be proficient at SimCity.
  - deleted 6y ago
    [deleted]
devit 6y ago
Is this thing actually being used for policy decisions? Given the complexity, poor language/framework choice (should have used either Rust or Tensorflow), bad code design, and unclear determination of the parameters (especially the fact they don't seem to be estimated from real data with Bayesian inference, or if they are they didn't release that), as well as the ludicrous CPU and RAM usage making it impossible for most people to run it and thus check it, it doesn't seem like it has any chance of actually being a good model. If this is the state of the art for the most important statistical modelling project in the world, we are in the dark ages.
- bencollier49 6y ago
  It's 13+ years old. There was no Rust or Tensorflow when it was originally produced.
  - drtillberg 6y ago
    13+ years ago the same academic modeler predicted 200 million deaths from a bird flu outbreak that killed 282 people in total.[1]. That's a pretty enormous error-- alarmist publicity-seeking behavior then and now, but the damage was felt more acutely now. If this modeler really wanted a competent simulation it would have been better constructed. That it was not well-constructed is another datapoint indicating how egregious the error was on the part of the goverment in failing to properly vet it. [1].https://www.express.co.uk/comment/columnists/frederick-forsyth/1270245/coronavirus-neil-ferguson-sars-bird-flu-deaths https://www.express.co.uk/comment/columnists/frederick-forsy...
    - zimpenfish 6y ago
      That's an amazingly disingenuous comment. https://www.theguardian.com/world/2005/sep/30/birdflu.jamessturcke https://www.theguardian.com/world/2005/sep/30/birdflu.jamess... He said that if it was the same as the 1918 flu, you could probably scale it up to 200M people. He didn't predict 200M, he speculated that it could be that bad. And he wasn't alone - "A global influenza pandemic is imminent and will kill up to 150 million people" said "David Nabarro, one of the most senior public health experts at the World Health Organisation" "A Department of Health contingency plan states anywhere that there could be between 21,500 and 709,000 deaths in Britain." An unnamed WHO spokeswoman said "best case scenario" would be 7.4 million deaths globally. https://www.newscientist.com/article/dn7787-flu-pandemic-lethal-yet-preventable/ https://www.newscientist.com/article/dn7787-flu-pandemic-let... "And yet, the models show, if targeted action is taken within a critical three-week window, an outbreak could be limited to fewer than 100 individuals within two months." Oh, he also showed how to react and keep the number of casualties to a minimum. Don't believe what you read in dirt rags.
beshrkayali 6y ago
Honestly this is kind of sad. I can't understand how someone can have the gall to use any results from this spaghetti code in a scientific paper, let alone one that changes a country's entire strategy.
- jfnixon 6y ago
  I'll be "that guy" and point out we know the climate simulations have similar issues.
  - deleted 6y ago
    [deleted]
  - deleted 6y ago
    [deleted]
te_chris 6y ago
If this has been flagged off the front page I despair for HN. This should be an apex story for us: open source release of a computer model that has incredible relevance to loads of people globally. Quite sad.
- bencollier49 6y ago
  I'm pretty sure it was. It was at number four and rising then suddenly dropped to the bottom of the front page before vanishing.
deleted 6y ago
[deleted]
jarym 6y ago
For a long time academics would only publish formulas and results as part of their papers - omitting source code and arguing that ‘the formulas are the only thing that matter’. Perhaps the real reason is the code typically smells of Swiss cheese. From now on I vow to trust NO academic results of computer models unless source code is published along with instructions on how to reproduce outputs (or at least similar output!)

57 points by bencollier49 6y ago

asplake 6y ago

Alumnus here, it’s Imperial College, London. It is a university (in its own right), but always goes by that name.

bencollier49 6y ago
I knew that; I went to Kings! Apparently I haven't woken up properly this morning. Title changed.
- dancedar 6y ago
  Alum of both, it's comma-less icl on paperwork, website, rankings, and research, see ie https://www.imperial.ac.uk/mrc-global-infectious-disease-analysis/covid-19/ https://www.imperial.ac.uk/mrc-global-infectious-disease-ana... bottom their address.
  - bencollier49 6y ago
    Ah but I somehow managed to write "Imperial University". I have now had a coffee, and changed the title.
howd00 6y ago
Only since 2007. Uni of London before that for 100 years. Used to play them at sports as part of the ULU league.
- asplake 6y ago
  Yes it was part of the University of London, but always “Imperial College”. As far as I’m aware never “Imperial University” as the title initially had it here.
- gvjddbnvdrbv 6y ago
  Individual colleges were effectively independent universities for many years before. The University of London umbrella was a shell.
- OJFord 6y ago
  No, Imperial's royal charter (what's needed to be 'a university' and award degrees in your own right) dates to 1907. I believe that's true of most of UoL's constituents, just not the smaller ones perhaps like SOAS or other field-focused colleges.

notkaiho 6y ago

I'm no expert in C++ so not sure if it's meant to look like that, but that code is making my brain hurt.

easytiger 6y ago
it is not meant to look like that. The spacing/tabbing is all over the place.
OJFord 6y ago
'Academic code' is infamous for having that effect.
gpderetta 6y ago
In my very limited experience, that's what a lot of code coming from academia looks like.
- Gibbon1 6y ago
  That code reminds me of a program that emulated an old Wyse Color Terminal.
mshook 6y ago
Just argument parsing makes my eyes bleed... https://github.com/mrc-ide/covid-sim/blob/master/src/CovidSim.cpp#L146 https://github.com/mrc-ide/covid-sim/blob/master/src/CovidSi... That stuff is priceless: else if (argv[i][1] == 'C' && argv[i][2] == 'L' && argv[i][3] == 'P' && argv[i][4] == '1' && argv[i][5] == ':') I guess string comparisons are complicated. I also fail to see why they used ':' as the separator and why they didn't use a proper library to parse argv...
- tasogare 6y ago
  Because when you have only a very limited number of arguments it’s easier to do it that way than selecting a lib, adding it to the project, reading the doc, trying it and integrating it for real. I add the issue in C#, and after trying 2 or 3 libs (including one by Microsoft) I just gave up.

djhworld 6y ago

Interesting tidbit for HN, Jobn Carmack helped a little with this project apparently https://twitter.com/ID_AA_Carmack/status/1254872368763277313 https://twitter.com/ID_AA_Carmack/status/1254872368763277313

antupis 6y ago
Yeah and he said that code is fine so I am trusting him.

abainbridge 6y ago

This code is great fun. Start here to explore the horror: https://github.com/mrc-ide/covid-sim/blob/master/src/CovidSim.cpp https://github.com/mrc-ide/covid-sim/blob/master/src/CovidSi... I like the way InitModel() crashes (I think) if a global pointer called bmh (short for bitmap header) isn't first initialized by calling InitBMHead() from Bitmap.cpp. I guess it's obvious to academics with giant brains that InitModel() depends on a bitmap existing. But it gets worse - the pointer is declared in CovidSim.cpp but Bitmap.cpp reaches across the module boundary to initialize it. Sweet. Edit: I feel bad now. It's great that this code was made public. I should be encouraging that. I think academics writing code for things like this should probably take an undergrad course on software engineering. I think the one that taught me about Abstract Data Types would be most beneficial the author(s) of this code.

deleted 6y ago
[deleted]
JauntyHatAngle 6y ago
I appreciate you pointing out the flaws, but I fear they sarcasm is not really warranted. Shit code or not, you can call it out without denigrating the guy. It's nice they aired their source code out for people to look at, least we can do is to critique without snark.
- abainbridge 6y ago
  OK, fair point. Snarkiest comment removed. My comments weren't meant to be serious. The problem I pointed out isn't actually a bug, it's just a style of programming that makes the code harder to read/work on than it needs to be.
deleted 6y ago
[deleted]
cataphract 6y ago
I've seen a lot of code coming out of academia that exhibits a similar complete lack of structure and completely ignores the most basic software development best practices (such as having tests). There are exceptions, and research software engineers are a thing, but unsurprisingly most academics focus their careers on the science rather than the code.

lbeltrame 6y ago

Some people have raised issues with the model[1][2], or rather, the software implementation of it. There's also a (flagged) submission on HN discussing this[3] referencing [1]. [1] (warning: possibly partisan link) https://lockdownsceptics.org/code-review-of-fergusons-model/ https://lockdownsceptics.org/code-review-of-fergusons-model/ [2] https://github.com/mrc-ide/covid-sim/issues/165 https://github.com/mrc-ide/covid-sim/issues/165 [3] https://news.ycombinator.com/item?id=23099212 https://news.ycombinator.com/item?id=23099212

te_chris 6y ago
That issue is unhelpful. Yes, no good tests, but how about suggesting ways it can actually be improved? Why are devs such bores when it comes to things like this? They have released the model, review it and suggest improvements! Don't grandstand "We, the undersigned etc etc" as if that's going to help improve the codebase in the slightest. Carmack is OK with it, he's put his name to reviewing it. That's not to excuse the bad testing, but he hasn't thrown his hands up and run away. He worked with them constructively to make it possible for us to even see it! Be more like Carmack. EDIT: Also be like this person: https://github.com/mrc-ide/covid-sim/issues/161 https://github.com/mrc-ide/covid-sim/issues/161 Helpful, constructive and the developers engaged with them.
- JauntyHatAngle 6y ago
  I fear that this area is too divisive currently to allow for normal discourse. The downvoting in these kinds of topics have been atrocious on hacker News the last few weeks.
  - Gibbon1 6y ago
    HN is one of the increasing fewer places online where covid19 deniers can congregate.
    - lbeltrame 6y ago
      I personally don't like much the term "deniers", because there are a lot of unknowns on this virus and what it does, and there is no agreement on many fronts. Also, doing these generalizations groups together people with very questionable theories ("It's the 5G") with others that have more nuanced criticism. Personally (and yes, I am a scientist) try to look up whatever is said in the media, either by journalists or experts, no matter if the results end up matching 100% what it is said (often it is less, and on some cases there is no match). I think there should be fairly high standards of scientific rigor even in published code, especially if this might impact public policy actions, like we should expect high rigor in biological and epidemiological studies.

kidintech 6y ago

I guess confidence really is key when making code public...

buboard 6y ago

toyg 6y ago
> Its results is as useful as simcity is for mayors. I bet the quality of local administration would go through the roof if all mayoral candidates were forced to be proficient at SimCity.
- deleted 6y ago
  [deleted]

devit 6y ago

Is this thing actually being used for policy decisions? Given the complexity, poor language/framework choice (should have used either Rust or Tensorflow), bad code design, and unclear determination of the parameters (especially the fact they don't seem to be estimated from real data with Bayesian inference, or if they are they didn't release that), as well as the ludicrous CPU and RAM usage making it impossible for most people to run it and thus check it, it doesn't seem like it has any chance of actually being a good model. If this is the state of the art for the most important statistical modelling project in the world, we are in the dark ages.

bencollier49 6y ago
It's 13+ years old. There was no Rust or Tensorflow when it was originally produced.
- drtillberg 6y ago
  13+ years ago the same academic modeler predicted 200 million deaths from a bird flu outbreak that killed 282 people in total.[1]. That's a pretty enormous error-- alarmist publicity-seeking behavior then and now, but the damage was felt more acutely now. If this modeler really wanted a competent simulation it would have been better constructed. That it was not well-constructed is another datapoint indicating how egregious the error was on the part of the goverment in failing to properly vet it. [1].https://www.express.co.uk/comment/columnists/frederick-forsyth/1270245/coronavirus-neil-ferguson-sars-bird-flu-deaths https://www.express.co.uk/comment/columnists/frederick-forsy...
  - zimpenfish 6y ago
    That's an amazingly disingenuous comment. https://www.theguardian.com/world/2005/sep/30/birdflu.jamessturcke https://www.theguardian.com/world/2005/sep/30/birdflu.jamess... He said that if it was the same as the 1918 flu, you could probably scale it up to 200M people. He didn't predict 200M, he speculated that it could be that bad. And he wasn't alone - "A global influenza pandemic is imminent and will kill up to 150 million people" said "David Nabarro, one of the most senior public health experts at the World Health Organisation" "A Department of Health contingency plan states anywhere that there could be between 21,500 and 709,000 deaths in Britain." An unnamed WHO spokeswoman said "best case scenario" would be 7.4 million deaths globally. https://www.newscientist.com/article/dn7787-flu-pandemic-lethal-yet-preventable/ https://www.newscientist.com/article/dn7787-flu-pandemic-let... "And yet, the models show, if targeted action is taken within a critical three-week window, an outbreak could be limited to fewer than 100 individuals within two months." Oh, he also showed how to react and keep the number of casualties to a minimum. Don't believe what you read in dirt rags.

beshrkayali 6y ago

Honestly this is kind of sad. I can't understand how someone can have the gall to use any results from this spaghetti code in a scientific paper, let alone one that changes a country's entire strategy.

jfnixon 6y ago
I'll be "that guy" and point out we know the climate simulations have similar issues.
- deleted 6y ago
  [deleted]
- deleted 6y ago
  [deleted]

te_chris 6y ago

If this has been flagged off the front page I despair for HN. This should be an apex story for us: open source release of a computer model that has incredible relevance to loads of people globally. Quite sad.

bencollier49 6y ago
I'm pretty sure it was. It was at number four and rising then suddenly dropped to the bottom of the front page before vanishing.

deleted 6y ago

[deleted]

jarym 6y ago

For a long time academics would only publish formulas and results as part of their papers - omitting source code and arguing that ‘the formulas are the only thing that matter’. Perhaps the real reason is the code typically smells of Swiss cheese. From now on I vow to trust NO academic results of computer models unless source code is published along with instructions on how to reproduce outputs (or at least similar output!)