6 ms·

I'm familiar with the concept of chaos. I've also spent a certain amount of time solving DEs for money, and most were not particularly chaotic. My superficial u

I'm familiar with the concept of chaos. I've also spent a certain amount of time solving DEs for money, and most were not particularly chaotic. My superficial understanding of Ferguson's model is that it's an extended SIR model, with different classes of patient with different behaviors, geographical grouping, etc. From my intuition, it doesn't seem like that kind of DE would be too chaotic, and I haven't seen anything in the literature to suggest otherwise (since when systems do become sensitive to limited arithmetic precision, it's common for practitioners to notice and report that, as they first famously did with the weather in your example above). Note that this sensitivity isn't usually just a numerical implementation detail. For example, it's one of the reasons why the weather is so hard to predict, because the physical effects that create the weather are extremely sensitive to the real physical initial conditions, which we can measure only with some uncertainty, typically much worse than our numerical precision. So when such behavior is present, modelers are usually aware that it is, since it has consequences well beyond floating point reductions. I haven't seen that discussed in any of Ferguson's papers, so I strongly suspect that floating point issues may prevent bit-exact results but do not explain the variation seen. But in any case, I think we agree that's probably not the biggest difference, and the biggest difference is probably random numbers in the wrong order? I typically attempt to determine whether a program is bug-free by testing it. That's harder when a program is non-deterministic, so people often include the ability to derive all pseudo-randomness from a fixed seed. It looks like Imperial College tried to do that here, but it didn't work. I agree that's not a flaw in the model, just something that makes it harder to prove that no flaws exists. Would you at least agree that in a perfect world, with unlimited resources, that's something you'd fix?

eesmith 6y ago
Given your expertise in solving DEs for money, and knowledge of chaos theory, why did you earlier mention "Yes, you can get chaotic behavior solving certain PDEs and stuff" and seemingly exclude the ordinary differential equations which started chaos theory? We know that transmissions rates are not fixed. While R0 models the average, sometimes 1 person infects 100 other people, and sometimes no one is infected. These 100+ events are rare, but not extraordinarily rare. If it happens early in a simulation, then your simulated model will result in a higher growth than one where that event occurs later. So sometimes you'll have a large outburst early, and sometimes it will be delayed a week or two. > My superficial understanding of Ferguson's model is that it's an extended SIR model, with different classes of patient with different behaviors, geographical grouping, etc. From my intuition, it doesn't seem like that kind of DE would be too chaotic The paper "Impact of non-pharmaceutical interventions (NPIs) to reduce COVID- 19 mortality and healthcare demand" appears to be the one which has influenced many. It cites https://www.nature.com/articles/nature04795 https://www.nature.com/articles/nature04795 as the model used, which cites https://www.nature.com/articles/nature04017 https://www.nature.com/articles/nature04017 which gives a summary: "We constructed a spatially explicit simulation of the 85 million people residing in Thailand and in a 100-km wide zone of contiguous neighbouring countries. The model explicitly incorporates households, schools and workplaces, as these are known to be the primary contexts of influenza transmission .. Random contacts in the community associated with day-to-day movement and travel were also modelled." I expect that sort of agent-based modeling, by nature, to be chaotic, along the lines I outlined above. You write: "So when such behavior is present, modelers are usually aware that it is, since it has consequences well beyond floating point reductions. I haven't seen that discussed in any of Ferguson's papers" How many papers of his have you read? The paper I quoted from, https://www.nature.com/articles/nature04017 https://www.nature.com/articles/nature04017 , includes: "Multiple assumptions are inevitably made when undertaking preparedness modelling for a future emergent infection. Sensitivity analyses are therefore critical for assessing the robustness of policy conclusions. Here, critical assumptions not already discussed include (1) the ratio of within-place to community transmission, (2) the expected generation time, Tg, of a new pandemic strain (largely determined by the duration of viral shedding and therefore infectiousness), (3) the level of heterogeneity in individual infectiousness (for example, ‘superspreaders’20), (4) antiviral efficacy/take-up, and (5) the sensitivity and specificity of case detection during the control programme." You rhetorically asked "Would you at least agree that in a perfect world, with unlimited resources, that's something you'd fix?" That's a bogus argument. If I had that sort of unlimited resources there are many more things I would rather use it on than making this simulation be the way you want it to be. For one, the noise and induced chaos are certainly far less than which result from the uncertain input values to the program.
- tripletao 6y ago
  I said "PDEs and stuff" above because the cases where chaotic behavior is most known to be a practical problem are PDEs. It's certainly also true that ODEs can exhibit chaotic behavior, and that's even occasionally a practical effect (chaotic orbits, etc.). It's just usually not. And I'm not sure what you intend by those quotes? Sensitivity analysis is a normal thing you do with almost every model. Those sensitivities are usually small enough that numerical error is negligible--your 100:1 example above is such a small share of the dynamic range of typical data types that it wouldn't usually get any special attention. So "sensitivity" usually means the author is talking about the behavior of the ideal model, correctly neglecting numerical effects. In the rare cases where that numerical error isn't negligible, the author would typically remark on that explicitly, and on the additional steps they took to build an accurate model despite that, and on why that extraordinary sensitivity doesn't invalidate their model in the real world (given that real-world measurement uncertainties are usually much bigger than an LSb). That doesn't seem to be the case here. Chaos is real, and it's cool; but in most (not all, most) practical work, it doesn't turn out to be that important. I first mentioned it only to show I was aware of cases where a single-LSb error could get amplified huge, and didn't think this was one of those rare cases. I agree that floating point issues may result in non-bit-exact results, but no one other than you is suggesting they explain anything close to the variation observed. I think we even agree that variation is probably mostly random numbers in the wrong order? So I'm not sure why we're continuing to discuss chaos. In my first post on this page, I wrote: > I'm not sure the "bug uncertainty" is any bigger than all the existing known uncertainty about the assumptions that go in to his (or any such) model, though. So I think I agree with you? I'm just saying that from my experience, it's normal good practice in stochastic code to add a "deterministic from seed" mode. It's certainly possible to develop a good model without that, in the same way that it's possible to write good software without tests or source control--it's just harder.
  - eesmith 6y ago
    > in most (not all, most) practical work, it doesn't turn out to be that important shrug Once upon a time, when I did simulation work, chaotic effects were everywhere. To see if perhaps my views were unusual, I did a search and found https://www.ncbi.nlm.nih.gov/books/NBK305903/ https://www.ncbi.nlm.nih.gov/books/NBK305903/ - "Assessing the Use of Agent-Based Models for Tobacco Regulation", "Appendix B - Agent-Based Models for Policy Analysis" "An agent-based model (ABM) is a computational simulation model of a many-agent system that captures the behaviors of the system’s autonomous agents and their interactions with each other. An ABM is a computational instantiation of a complex adaptive system (CAS). A CAS is a dynamic model that represents individual agents and their collective behavior" That describes the Imperial College model. "ABMs are nonlinear models; highly nonlinear is the usual term. Deterministic nonlinear models have three characteristics that make them difficult to use statistically: " * Sensitive dependence on initial conditions: A map exhibits sensitive dependence on initial conditions at x0 if there is some distance d > 0 such that no matter how close to x0 another initial point x′0 is chosen, xt and x′t will eventually be at least distance d apart. " * Complicated limit dynamics: The limit behavior of typical linear dynamic systems is simple: Either the system converges to a steady state from any initial conditions, or they blow up, diverging to infinity. Nonlinear dynamical systems have much more complicated dynamics, including stable limit cycles of various periodicities, strange attractors, and chaotic behavior. " * Sensitive dependence on parameters: Small changes in parameter values can lead to abrupt changes in the qualitative character of system dynamics. This is in alignment with my understanding of the issues. These are well-known, and I have difficulty understanding your background enough to not have faced them before. Perhaps since your intuition is based on experiences which are distant from this sort of modeling, you can't use your experience as a guideline? You wrote: > it's normal good practice in stochastic code to add a "deterministic from seed" mode There's nothing in the bug report that you linked to which suggests that the result is not "deterministic from seed" on single-threaded code, so long as you repeat the re-run the code exactly. The reported non-determinism was not a result of re-running the code exactly, but of running it in a slightly different state. With this sort of model, "Small changes in parameter values can lead to abrupt changes in the qualitative character of system dynamics." Which is reasonable because in real-life we also expect that small changes in how things turn out may lead to 'abrupt changes in the qualitative character of system dynamics'.

eesmith 6y ago

Given your expertise in solving DEs for money, and knowledge of chaos theory, why did you earlier mention "Yes, you can get chaotic behavior solving certain PDEs and stuff" and seemingly exclude the ordinary differential equations which started chaos theory? We know that transmissions rates are not fixed. While R0 models the average, sometimes 1 person infects 100 other people, and sometimes no one is infected. These 100+ events are rare, but not extraordinarily rare. If it happens early in a simulation, then your simulated model will result in a higher growth than one where that event occurs later. So sometimes you'll have a large outburst early, and sometimes it will be delayed a week or two. > My superficial understanding of Ferguson's model is that it's an extended SIR model, with different classes of patient with different behaviors, geographical grouping, etc. From my intuition, it doesn't seem like that kind of DE would be too chaotic The paper "Impact of non-pharmaceutical interventions (NPIs) to reduce COVID- 19 mortality and healthcare demand" appears to be the one which has influenced many. It cites https://www.nature.com/articles/nature04795 https://www.nature.com/articles/nature04795 as the model used, which cites https://www.nature.com/articles/nature04017 https://www.nature.com/articles/nature04017 which gives a summary: "We constructed a spatially explicit simulation of the 85 million people residing in Thailand and in a 100-km wide zone of contiguous neighbouring countries. The model explicitly incorporates households, schools and workplaces, as these are known to be the primary contexts of influenza transmission .. Random contacts in the community associated with day-to-day movement and travel were also modelled." I expect that sort of agent-based modeling, by nature, to be chaotic, along the lines I outlined above. You write: "So when such behavior is present, modelers are usually aware that it is, since it has consequences well beyond floating point reductions. I haven't seen that discussed in any of Ferguson's papers" How many papers of his have you read? The paper I quoted from, https://www.nature.com/articles/nature04017 https://www.nature.com/articles/nature04017 , includes: "Multiple assumptions are inevitably made when undertaking preparedness modelling for a future emergent infection. Sensitivity analyses are therefore critical for assessing the robustness of policy conclusions. Here, critical assumptions not already discussed include (1) the ratio of within-place to community transmission, (2) the expected generation time, Tg, of a new pandemic strain (largely determined by the duration of viral shedding and therefore infectiousness), (3) the level of heterogeneity in individual infectiousness (for example, ‘superspreaders’20), (4) antiviral efficacy/take-up, and (5) the sensitivity and specificity of case detection during the control programme." You rhetorically asked "Would you at least agree that in a perfect world, with unlimited resources, that's something you'd fix?" That's a bogus argument. If I had that sort of unlimited resources there are many more things I would rather use it on than making this simulation be the way you want it to be. For one, the noise and induced chaos are certainly far less than which result from the uncertain input values to the program.

tripletao 6y ago
I said "PDEs and stuff" above because the cases where chaotic behavior is most known to be a practical problem are PDEs. It's certainly also true that ODEs can exhibit chaotic behavior, and that's even occasionally a practical effect (chaotic orbits, etc.). It's just usually not. And I'm not sure what you intend by those quotes? Sensitivity analysis is a normal thing you do with almost every model. Those sensitivities are usually small enough that numerical error is negligible--your 100:1 example above is such a small share of the dynamic range of typical data types that it wouldn't usually get any special attention. So "sensitivity" usually means the author is talking about the behavior of the ideal model, correctly neglecting numerical effects. In the rare cases where that numerical error isn't negligible, the author would typically remark on that explicitly, and on the additional steps they took to build an accurate model despite that, and on why that extraordinary sensitivity doesn't invalidate their model in the real world (given that real-world measurement uncertainties are usually much bigger than an LSb). That doesn't seem to be the case here. Chaos is real, and it's cool; but in most (not all, most) practical work, it doesn't turn out to be that important. I first mentioned it only to show I was aware of cases where a single-LSb error could get amplified huge, and didn't think this was one of those rare cases. I agree that floating point issues may result in non-bit-exact results, but no one other than you is suggesting they explain anything close to the variation observed. I think we even agree that variation is probably mostly random numbers in the wrong order? So I'm not sure why we're continuing to discuss chaos. In my first post on this page, I wrote: > I'm not sure the "bug uncertainty" is any bigger than all the existing known uncertainty about the assumptions that go in to his (or any such) model, though. So I think I agree with you? I'm just saying that from my experience, it's normal good practice in stochastic code to add a "deterministic from seed" mode. It's certainly possible to develop a good model without that, in the same way that it's possible to write good software without tests or source control--it's just harder.
- eesmith 6y ago
  > in most (not all, most) practical work, it doesn't turn out to be that important shrug Once upon a time, when I did simulation work, chaotic effects were everywhere. To see if perhaps my views were unusual, I did a search and found https://www.ncbi.nlm.nih.gov/books/NBK305903/ https://www.ncbi.nlm.nih.gov/books/NBK305903/ - "Assessing the Use of Agent-Based Models for Tobacco Regulation", "Appendix B - Agent-Based Models for Policy Analysis" "An agent-based model (ABM) is a computational simulation model of a many-agent system that captures the behaviors of the system’s autonomous agents and their interactions with each other. An ABM is a computational instantiation of a complex adaptive system (CAS). A CAS is a dynamic model that represents individual agents and their collective behavior" That describes the Imperial College model. "ABMs are nonlinear models; highly nonlinear is the usual term. Deterministic nonlinear models have three characteristics that make them difficult to use statistically: " * Sensitive dependence on initial conditions: A map exhibits sensitive dependence on initial conditions at x0 if there is some distance d > 0 such that no matter how close to x0 another initial point x′0 is chosen, xt and x′t will eventually be at least distance d apart. " * Complicated limit dynamics: The limit behavior of typical linear dynamic systems is simple: Either the system converges to a steady state from any initial conditions, or they blow up, diverging to infinity. Nonlinear dynamical systems have much more complicated dynamics, including stable limit cycles of various periodicities, strange attractors, and chaotic behavior. " * Sensitive dependence on parameters: Small changes in parameter values can lead to abrupt changes in the qualitative character of system dynamics. This is in alignment with my understanding of the issues. These are well-known, and I have difficulty understanding your background enough to not have faced them before. Perhaps since your intuition is based on experiences which are distant from this sort of modeling, you can't use your experience as a guideline? You wrote: > it's normal good practice in stochastic code to add a "deterministic from seed" mode There's nothing in the bug report that you linked to which suggests that the result is not "deterministic from seed" on single-threaded code, so long as you repeat the re-run the code exactly. The reported non-determinism was not a result of re-running the code exactly, but of running it in a slightly different state. With this sort of model, "Small changes in parameter values can lead to abrupt changes in the qualitative character of system dynamics." Which is reasonable because in real-life we also expect that small changes in how things turn out may lead to 'abrupt changes in the qualitative character of system dynamics'.