4 ms·

Some people have raised issues with the model[1][2], or rather, the software implementation of it. There's also a (flagged) submission on HN discussing this[3]

Some people have raised issues with the model[1][2], or rather, the software implementation of it. There's also a (flagged) submission on HN discussing this[3] referencing [1]. [1] (warning: possibly partisan link) https://lockdownsceptics.org/code-review-of-fergusons-model/ https://lockdownsceptics.org/code-review-of-fergusons-model/ [2] https://github.com/mrc-ide/covid-sim/issues/165 https://github.com/mrc-ide/covid-sim/issues/165 [3] https://news.ycombinator.com/item?id=23099212 https://news.ycombinator.com/item?id=23099212

te_chris 6y ago
That issue is unhelpful. Yes, no good tests, but how about suggesting ways it can actually be improved? Why are devs such bores when it comes to things like this? They have released the model, review it and suggest improvements! Don't grandstand "We, the undersigned etc etc" as if that's going to help improve the codebase in the slightest. Carmack is OK with it, he's put his name to reviewing it. That's not to excuse the bad testing, but he hasn't thrown his hands up and run away. He worked with them constructively to make it possible for us to even see it! Be more like Carmack. EDIT: Also be like this person: https://github.com/mrc-ide/covid-sim/issues/161 https://github.com/mrc-ide/covid-sim/issues/161 Helpful, constructive and the developers engaged with them.
- JauntyHatAngle 6y ago
  I fear that this area is too divisive currently to allow for normal discourse. The downvoting in these kinds of topics have been atrocious on hacker News the last few weeks.
  - Gibbon1 6y ago
    HN is one of the increasing fewer places online where covid19 deniers can congregate.
    - lbeltrame 6y ago
      I personally don't like much the term "deniers", because there are a lot of unknowns on this virus and what it does, and there is no agreement on many fronts. Also, doing these generalizations groups together people with very questionable theories ("It's the 5G") with others that have more nuanced criticism. Personally (and yes, I am a scientist) try to look up whatever is said in the media, either by journalists or experts, no matter if the results end up matching 100% what it is said (often it is less, and on some cases there is no match). I think there should be fairly high standards of scientific rigor even in published code, especially if this might impact public policy actions, like we should expect high rigor in biological and epidemiological studies.
      - Gibbon1 6y ago
        I started using the term deniers because all of them are studiously ignoring something that doesn't pencil out. It doesn't matter if they are loony deniers, sciency deniers, I have a big brain tech bro deniers. The result is all the same. Garbage. Dangerous garbage.
- renewiltord 6y ago
  Carmack has different motivations. He has nothing to prove and simplistically looks to make things better. You can tell in his appearance on Joe Rogan. Most HN users are just publicly preening. It's like a Mechanical Turk GPT2. I actually doubt they can write code.
- thu2111 6y ago
  It's too late. The early buggy code was already used to drive decisions, and they refused to release it for a month and a half whilst they tried to fix the most embarrassing errors. Go click around GitHub. They screwed up a shuffle, there are uninitialised reads, RNG bugs, the works.
SiempreViernes 6y ago
> On a personal level, I’d go further and suggest that all academic epidemiology be defunded. Wow, next they'll review one doctors handwriting and conclude that hospitals should be defunded, with their job handled by horse doctors... Non-intended randomness is of course bad, but it's bad mainly because it makes it harder to track down causes of actually important problems with the produced distributions. The worst problem this all out murder attempt can muster is that the code is hard to debug which frankly is should be the default assumption for all research code, not too persuasive. Models are after all just tools: what is critical is that you've made reliable predictions, not that the tools themselves are easy to use correctly. A more interesting critique would be something along the lines of this: https://www.nicholaslewis.org/imperial-college-uk-covid-19-numbers-dont-seem-to-add-up/ https://www.nicholaslewis.org/imperial-college-uk-covid-19-n... (however it's not by a subject matter expert so the problems they find might well be because the misunderstood some detail).
- jfnixon 6y ago
  If you read the lockdownskeptics cite, "hard to debug" is not the problem. Non-determinism in the output is the issue, and if this is indeed the case, why would anyone trust the results? Do a bunch of runs and average is not a good answer.
  - creato 6y ago
    It's really not enough to say. "Do a bunch of runs and average" is exactly how quite a bit of simulation software works. In this case, a small number of random outcomes early in the "pandemic" will have a large impact on the outcome. Of course, this kind of uncertainty needs to be dealt with, and that may have been done by running the simulation code we are presented with multiple times. It may be necessary to read both the code and the associated papers to judge this correctly.
    - SiempreViernes 6y ago
      Exactly, the individual runs aren't actually the "deliverable" of the code; rather it is the average of many runs that represent the real result of the code. Nothing presented clearly compromised the (supposed) reliability of the distributions produced, so the impact of these bugs beyond the inconvenience they add is unclear. To be clear, it is certainly not true that removing these bugs will somehow prove that the model and its inputs themselves are correct.

te_chris 6y ago

That issue is unhelpful. Yes, no good tests, but how about suggesting ways it can actually be improved? Why are devs such bores when it comes to things like this? They have released the model, review it and suggest improvements! Don't grandstand "We, the undersigned etc etc" as if that's going to help improve the codebase in the slightest. Carmack is OK with it, he's put his name to reviewing it. That's not to excuse the bad testing, but he hasn't thrown his hands up and run away. He worked with them constructively to make it possible for us to even see it! Be more like Carmack. EDIT: Also be like this person: https://github.com/mrc-ide/covid-sim/issues/161 https://github.com/mrc-ide/covid-sim/issues/161 Helpful, constructive and the developers engaged with them.

JauntyHatAngle 6y ago
I fear that this area is too divisive currently to allow for normal discourse. The downvoting in these kinds of topics have been atrocious on hacker News the last few weeks.
- Gibbon1 6y ago
  HN is one of the increasing fewer places online where covid19 deniers can congregate.
  - lbeltrame 6y ago
    I personally don't like much the term "deniers", because there are a lot of unknowns on this virus and what it does, and there is no agreement on many fronts. Also, doing these generalizations groups together people with very questionable theories ("It's the 5G") with others that have more nuanced criticism. Personally (and yes, I am a scientist) try to look up whatever is said in the media, either by journalists or experts, no matter if the results end up matching 100% what it is said (often it is less, and on some cases there is no match). I think there should be fairly high standards of scientific rigor even in published code, especially if this might impact public policy actions, like we should expect high rigor in biological and epidemiological studies.
    - Gibbon1 6y ago
      I started using the term deniers because all of them are studiously ignoring something that doesn't pencil out. It doesn't matter if they are loony deniers, sciency deniers, I have a big brain tech bro deniers. The result is all the same. Garbage. Dangerous garbage.
renewiltord 6y ago
Carmack has different motivations. He has nothing to prove and simplistically looks to make things better. You can tell in his appearance on Joe Rogan. Most HN users are just publicly preening. It's like a Mechanical Turk GPT2. I actually doubt they can write code.
thu2111 6y ago
It's too late. The early buggy code was already used to drive decisions, and they refused to release it for a month and a half whilst they tried to fix the most embarrassing errors. Go click around GitHub. They screwed up a shuffle, there are uninitialised reads, RNG bugs, the works.

SiempreViernes 6y ago

> On a personal level, I’d go further and suggest that all academic epidemiology be defunded. Wow, next they'll review one doctors handwriting and conclude that hospitals should be defunded, with their job handled by horse doctors... Non-intended randomness is of course bad, but it's bad mainly because it makes it harder to track down causes of actually important problems with the produced distributions. The worst problem this all out murder attempt can muster is that the code is hard to debug which frankly is should be the default assumption for all research code, not too persuasive. Models are after all just tools: what is critical is that you've made reliable predictions, not that the tools themselves are easy to use correctly. A more interesting critique would be something along the lines of this: https://www.nicholaslewis.org/imperial-college-uk-covid-19-numbers-dont-seem-to-add-up/ https://www.nicholaslewis.org/imperial-college-uk-covid-19-n... (however it's not by a subject matter expert so the problems they find might well be because the misunderstood some detail).

jfnixon 6y ago
If you read the lockdownskeptics cite, "hard to debug" is not the problem. Non-determinism in the output is the issue, and if this is indeed the case, why would anyone trust the results? Do a bunch of runs and average is not a good answer.
- creato 6y ago
  It's really not enough to say. "Do a bunch of runs and average" is exactly how quite a bit of simulation software works. In this case, a small number of random outcomes early in the "pandemic" will have a large impact on the outcome. Of course, this kind of uncertainty needs to be dealt with, and that may have been done by running the simulation code we are presented with multiple times. It may be necessary to read both the code and the associated papers to judge this correctly.
  - SiempreViernes 6y ago
    Exactly, the individual runs aren't actually the "deliverable" of the code; rather it is the average of many runs that represent the real result of the code. Nothing presented clearly compromised the (supposed) reliability of the distributions produced, so the impact of these bugs beyond the inconvenience they add is unclear. To be clear, it is certainly not true that removing these bugs will somehow prove that the model and its inputs themselves are correct.