4 ms·

> On a personal level, I’d go further and suggest that all academic epidemiology be defunded. Wow, next they'll review one doctors handwriting and conclude tha

> On a personal level, I’d go further and suggest that all academic epidemiology be defunded. Wow, next they'll review one doctors handwriting and conclude that hospitals should be defunded, with their job handled by horse doctors... Non-intended randomness is of course bad, but it's bad mainly because it makes it harder to track down causes of actually important problems with the produced distributions. The worst problem this all out murder attempt can muster is that the code is hard to debug which frankly is should be the default assumption for all research code, not too persuasive. Models are after all just tools: what is critical is that you've made reliable predictions, not that the tools themselves are easy to use correctly. A more interesting critique would be something along the lines of this: https://www.nicholaslewis.org/imperial-college-uk-covid-19-numbers-dont-seem-to-add-up/ https://www.nicholaslewis.org/imperial-college-uk-covid-19-n... (however it's not by a subject matter expert so the problems they find might well be because the misunderstood some detail).

jfnixon 7y ago
If you read the lockdownskeptics cite, "hard to debug" is not the problem. Non-determinism in the output is the issue, and if this is indeed the case, why would anyone trust the results? Do a bunch of runs and average is not a good answer.
- creato 7y ago
  It's really not enough to say. "Do a bunch of runs and average" is exactly how quite a bit of simulation software works. In this case, a small number of random outcomes early in the "pandemic" will have a large impact on the outcome. Of course, this kind of uncertainty needs to be dealt with, and that may have been done by running the simulation code we are presented with multiple times. It may be necessary to read both the code and the associated papers to judge this correctly.
  - SiempreViernes 7y ago
    Exactly, the individual runs aren't actually the "deliverable" of the code; rather it is the average of many runs that represent the real result of the code. Nothing presented clearly compromised the (supposed) reliability of the distributions produced, so the impact of these bugs beyond the inconvenience they add is unclear. To be clear, it is certainly not true that removing these bugs will somehow prove that the model and its inputs themselves are correct.
  - thu2111 7y ago
    The non-determinism that page talks about is coming from bugs like memory corruptions, floating point inaccuracies, initialisation order bugs. It's not an intentional part of the model. Averaging corrupted data is meaningless, it doesn't magically fix the corruption. I really wonder what it would take for some people to lose faith in epidemiology. Has this field ever predicted an epidemic correctly? Is there any level of bugginess that would yield the output of these teams unacceptable, to them?
    - creato 7y ago
      > The non-determinism that page talks about is coming from bugs like memory corruptions, ..., initialisation order bugs. Source? I haven't seen these specifically cited anywhere. > floating point inaccuracies, Combined with (safe) race conditions, this will cause non-determinism that would probably be considered OK. In general: John Carmack looked at the code and thought it was OK for what it is (decade old simulation transpiled from Fortran at some point). Some ex-Google guy thinks its horrible. I looked at the code myself briefly. I haven't formed a strong opinion about the code myself beyond "it's ugly and I don't want to work with it, glad it's not my problem." I am however objecting to some of the comments here that make it sound like it is obviously broken for reasons that they just don't understand. I think a comment on the GitHub is relevant: https://github.com/mrc-ide/covid-sim/issues/175#issuecomment-625377867 https://github.com/mrc-ide/covid-sim/issues/175#issuecomment... > To add to this, please read report 9 properly. The 500k UK prediction was if governments did nothing whatsoever - we never believed governments would do nothing but we modelled it as a base case, because that's part of what you do when you model. > With the full social distancing the report suggestd it might be possible to reduce deaths perhaps to 20k - a death count we have already exceeded. Nobody here is laughing about that. The report was also very frank about the uncertainty involved in trying to predict what might happen at that stage
lbeltrame 7y ago
> Models are after all just tools: what is critical is that you've made reliable predictions, not that the tools themselves are easy to use correctly. Reliable and reproducible. I think this discussion, even if the code ends up being correct and the model OK, is worth having. It might not change anything today, but could set new (hopefully better!) standards tomorrow. That said, that "personal level" statement is absolutely out of the line.
- SiempreViernes 7y ago
  Sure, but for an inherently stochastic model the main results are the average behaviour, and the reproducibility of these are the main concern. Reproducibility of individual runs is more a convenience.

jfnixon 7y ago

If you read the lockdownskeptics cite, "hard to debug" is not the problem. Non-determinism in the output is the issue, and if this is indeed the case, why would anyone trust the results? Do a bunch of runs and average is not a good answer.

creato 7y ago
It's really not enough to say. "Do a bunch of runs and average" is exactly how quite a bit of simulation software works. In this case, a small number of random outcomes early in the "pandemic" will have a large impact on the outcome. Of course, this kind of uncertainty needs to be dealt with, and that may have been done by running the simulation code we are presented with multiple times. It may be necessary to read both the code and the associated papers to judge this correctly.
- SiempreViernes 7y ago
  Exactly, the individual runs aren't actually the "deliverable" of the code; rather it is the average of many runs that represent the real result of the code. Nothing presented clearly compromised the (supposed) reliability of the distributions produced, so the impact of these bugs beyond the inconvenience they add is unclear. To be clear, it is certainly not true that removing these bugs will somehow prove that the model and its inputs themselves are correct.
- thu2111 7y ago
  The non-determinism that page talks about is coming from bugs like memory corruptions, floating point inaccuracies, initialisation order bugs. It's not an intentional part of the model. Averaging corrupted data is meaningless, it doesn't magically fix the corruption. I really wonder what it would take for some people to lose faith in epidemiology. Has this field ever predicted an epidemic correctly? Is there any level of bugginess that would yield the output of these teams unacceptable, to them?
  - creato 7y ago
    > The non-determinism that page talks about is coming from bugs like memory corruptions, ..., initialisation order bugs. Source? I haven't seen these specifically cited anywhere. > floating point inaccuracies, Combined with (safe) race conditions, this will cause non-determinism that would probably be considered OK. In general: John Carmack looked at the code and thought it was OK for what it is (decade old simulation transpiled from Fortran at some point). Some ex-Google guy thinks its horrible. I looked at the code myself briefly. I haven't formed a strong opinion about the code myself beyond "it's ugly and I don't want to work with it, glad it's not my problem." I am however objecting to some of the comments here that make it sound like it is obviously broken for reasons that they just don't understand. I think a comment on the GitHub is relevant: https://github.com/mrc-ide/covid-sim/issues/175#issuecomment-625377867 https://github.com/mrc-ide/covid-sim/issues/175#issuecomment... > To add to this, please read report 9 properly. The 500k UK prediction was if governments did nothing whatsoever - we never believed governments would do nothing but we modelled it as a base case, because that's part of what you do when you model. > With the full social distancing the report suggestd it might be possible to reduce deaths perhaps to 20k - a death count we have already exceeded. Nobody here is laughing about that. The report was also very frank about the uncertainty involved in trying to predict what might happen at that stage

lbeltrame 7y ago

> Models are after all just tools: what is critical is that you've made reliable predictions, not that the tools themselves are easy to use correctly. Reliable and reproducible. I think this discussion, even if the code ends up being correct and the model OK, is worth having. It might not change anything today, but could set new (hopefully better!) standards tomorrow. That said, that "personal level" statement is absolutely out of the line.

SiempreViernes 7y ago
Sure, but for an inherently stochastic model the main results are the average behaviour, and the reproducibility of these are the main concern. Reproducibility of individual runs is more a convenience.