3 ms·

So I understand your view correctly: Are you saying that you expect that in Ferguson's model, changes in intermediate values on the order of one part in 2^53 (a

So I understand your view correctly: Are you saying that you expect that in Ferguson's model, changes in intermediate values on the order of one part in 2^53 (a double mantissa) would change the output by tens of thousands of deaths? If so, how could that model possibly be useful, given that none of the inputs are known to anywhere near that? If you're really that sensitive to numerical error, then just averaging multiple multi-threaded runs may be biased in strange ways, because the process adding the uncertainty (thread scheduling or whatever) is neither deterministic nor random with known statistics--a warm day might cause your CPU to throttle, changing the scheduling, significantly changing your result. (To be clear, I don't think that's the case; I think any floating-point error is negligible except to any extent it prevents bit-exact reproducible runs.) The stuff you're quoting above is pretty standard description of dynamical systems in general. It's true, but I don't see how it applies to Ferguson's model with respect to numerical error; and if it does, then I don't see how his model is useful. I could believe his model is chaotic in the sense that the outputs vary dramatically over the expected uncertainties in the inputs, but the numerical inaccuracy is many orders of magnitude below that. Finally, I keep saying their "deterministic from seed" mode was broken multi-threaded or after restarts, and you keep saying it was fine single-threaded in a single start. I think both are true, so I'm not sure what we're arguing about?

eesmith 6y ago
> would change the output by tens of thousands of deaths? Yes, with the modification of the statement to "I would not be surprised if it changes by many percentage points." The bug report you linked to says only "This mostly manifests as a time- offset, but there are numerical changes beyond that." It shows a different by day 80 of 100,000 deaths, or 20% overall. It does not show if by day (say) 120 those numbers have gotten further together or closer. If they both end at about the same numbers, separated only by a week, then it does not change the overall conclusions in the paper at https://www.imperial.ac.uk/media/imperial-college/medicine/sph/ide/gida-fellowships/Imperial-College-COVID19-NPI-modelling-16-03-2020.pdf https://www.imperial.ac.uk/media/imperial-college/medicine/s... . It does shift some of the dates, and I am annoyed with the lack of error bars or the like in their plots. You asked "how could that model possibly be useful, given that none of the inputs are known to anywhere near that?" You check it for sensitivity across a range of values and see how the overall conclusions change. Quoting that paper I linked to, "Such policies are robust to uncertainty in both the reproduction number, R0(Table 4) and in the severity of the virus ... However, there are very large uncertainties around the transmission of this virus, the likely effectiveness of different policies and the extent to which the population spontaneously adopts risk reducing behaviours. This means it is difficult to be definitive about the likely initial duration of measures which will be required, except that it will be several months." When all the lines in a spaghetti map show the hurricane bearing down on you, you prepare for bad weather, even though the hurricane track is almost certainly not going to follow any of those paths. You write "but the numerical inaccuracy is many orders of magnitude below that." What I know about chaotic systems is that I cannot make that conclusion without much more study about the relevant phase space. If the end result is an attractor, then even intermediate chaotic behavior is not much of an issue, depending of course on the specific details of the policy guidance. You write "I think both are true, so I'm not sure what we're arguing about?" My comment started because of your comments at https://news.ycombinator.com/item?id=23213711 https://news.ycombinator.com/item?id=23213711 where you wrote that discussion on the GitHub comments wasn't "terribly clear" to you. I gave my interpretation. This specific thread started because you wrote "They seem to think the model is fine and just the "deterministic from seed" mode is broken", while I don't see in that bug report where they state the '"deterministic from seed" mode is broken'. Is the GitHub commentary clear to you now, and if not, what is unclear?
- tripletao 6y ago
  We've perhaps not spent our time efficiently in this thread; it looks like they fixed the non-determinism a month ago, https://github.com/mrc-ide/covid-sim/pull/121 https://github.com/mrc-ide/covid-sim/pull/121 Linked directly from the issue, too. It would indeed seem to have all been random numbers in the wrong order.
  - eesmith 6y ago
    C'est la vie. Thanks for tracking that down.

eesmith 6y ago

> would change the output by tens of thousands of deaths? Yes, with the modification of the statement to "I would not be surprised if it changes by many percentage points." The bug report you linked to says only "This mostly manifests as a time- offset, but there are numerical changes beyond that." It shows a different by day 80 of 100,000 deaths, or 20% overall. It does not show if by day (say) 120 those numbers have gotten further together or closer. If they both end at about the same numbers, separated only by a week, then it does not change the overall conclusions in the paper at https://www.imperial.ac.uk/media/imperial-college/medicine/sph/ide/gida-fellowships/Imperial-College-COVID19-NPI-modelling-16-03-2020.pdf https://www.imperial.ac.uk/media/imperial-college/medicine/s... . It does shift some of the dates, and I am annoyed with the lack of error bars or the like in their plots. You asked "how could that model possibly be useful, given that none of the inputs are known to anywhere near that?" You check it for sensitivity across a range of values and see how the overall conclusions change. Quoting that paper I linked to, "Such policies are robust to uncertainty in both the reproduction number, R0(Table 4) and in the severity of the virus ... However, there are very large uncertainties around the transmission of this virus, the likely effectiveness of different policies and the extent to which the population spontaneously adopts risk reducing behaviours. This means it is difficult to be definitive about the likely initial duration of measures which will be required, except that it will be several months." When all the lines in a spaghetti map show the hurricane bearing down on you, you prepare for bad weather, even though the hurricane track is almost certainly not going to follow any of those paths. You write "but the numerical inaccuracy is many orders of magnitude below that." What I know about chaotic systems is that I cannot make that conclusion without much more study about the relevant phase space. If the end result is an attractor, then even intermediate chaotic behavior is not much of an issue, depending of course on the specific details of the policy guidance. You write "I think both are true, so I'm not sure what we're arguing about?" My comment started because of your comments at https://news.ycombinator.com/item?id=23213711 https://news.ycombinator.com/item?id=23213711 where you wrote that discussion on the GitHub comments wasn't "terribly clear" to you. I gave my interpretation. This specific thread started because you wrote "They seem to think the model is fine and just the "deterministic from seed" mode is broken", while I don't see in that bug report where they state the '"deterministic from seed" mode is broken'. Is the GitHub commentary clear to you now, and if not, what is unclear?

tripletao 6y ago
We've perhaps not spent our time efficiently in this thread; it looks like they fixed the non-determinism a month ago, https://github.com/mrc-ide/covid-sim/pull/121 https://github.com/mrc-ide/covid-sim/pull/121 Linked directly from the issue, too. It would indeed seem to have all been random numbers in the wrong order.
- eesmith 6y ago
  C'est la vie. Thanks for tracking that down.