3 ms·

I said "PDEs and stuff" above because the cases where chaotic behavior is most known to be a practical problem are PDEs. It's certainly also true that ODEs can

I said "PDEs and stuff" above because the cases where chaotic behavior is most known to be a practical problem are PDEs. It's certainly also true that ODEs can exhibit chaotic behavior, and that's even occasionally a practical effect (chaotic orbits, etc.). It's just usually not. And I'm not sure what you intend by those quotes? Sensitivity analysis is a normal thing you do with almost every model. Those sensitivities are usually small enough that numerical error is negligible--your 100:1 example above is such a small share of the dynamic range of typical data types that it wouldn't usually get any special attention. So "sensitivity" usually means the author is talking about the behavior of the ideal model, correctly neglecting numerical effects. In the rare cases where that numerical error isn't negligible, the author would typically remark on that explicitly, and on the additional steps they took to build an accurate model despite that, and on why that extraordinary sensitivity doesn't invalidate their model in the real world (given that real-world measurement uncertainties are usually much bigger than an LSb). That doesn't seem to be the case here. Chaos is real, and it's cool; but in most (not all, most) practical work, it doesn't turn out to be that important. I first mentioned it only to show I was aware of cases where a single-LSb error could get amplified huge, and didn't think this was one of those rare cases. I agree that floating point issues may result in non-bit-exact results, but no one other than you is suggesting they explain anything close to the variation observed. I think we even agree that variation is probably mostly random numbers in the wrong order? So I'm not sure why we're continuing to discuss chaos. In my first post on this page, I wrote: > I'm not sure the "bug uncertainty" is any bigger than all the existing known uncertainty about the assumptions that go in to his (or any such) model, though. So I think I agree with you? I'm just saying that from my experience, it's normal good practice in stochastic code to add a "deterministic from seed" mode. It's certainly possible to develop a good model without that, in the same way that it's possible to write good software without tests or source control--it's just harder.

eesmith 6y ago
> in most (not all, most) practical work, it doesn't turn out to be that important shrug Once upon a time, when I did simulation work, chaotic effects were everywhere. To see if perhaps my views were unusual, I did a search and found https://www.ncbi.nlm.nih.gov/books/NBK305903/ https://www.ncbi.nlm.nih.gov/books/NBK305903/ - "Assessing the Use of Agent-Based Models for Tobacco Regulation", "Appendix B - Agent-Based Models for Policy Analysis" "An agent-based model (ABM) is a computational simulation model of a many-agent system that captures the behaviors of the system’s autonomous agents and their interactions with each other. An ABM is a computational instantiation of a complex adaptive system (CAS). A CAS is a dynamic model that represents individual agents and their collective behavior" That describes the Imperial College model. "ABMs are nonlinear models; highly nonlinear is the usual term. Deterministic nonlinear models have three characteristics that make them difficult to use statistically: " * Sensitive dependence on initial conditions: A map exhibits sensitive dependence on initial conditions at x0 if there is some distance d > 0 such that no matter how close to x0 another initial point x′0 is chosen, xt and x′t will eventually be at least distance d apart. " * Complicated limit dynamics: The limit behavior of typical linear dynamic systems is simple: Either the system converges to a steady state from any initial conditions, or they blow up, diverging to infinity. Nonlinear dynamical systems have much more complicated dynamics, including stable limit cycles of various periodicities, strange attractors, and chaotic behavior. " * Sensitive dependence on parameters: Small changes in parameter values can lead to abrupt changes in the qualitative character of system dynamics. This is in alignment with my understanding of the issues. These are well-known, and I have difficulty understanding your background enough to not have faced them before. Perhaps since your intuition is based on experiences which are distant from this sort of modeling, you can't use your experience as a guideline? You wrote: > it's normal good practice in stochastic code to add a "deterministic from seed" mode There's nothing in the bug report that you linked to which suggests that the result is not "deterministic from seed" on single-threaded code, so long as you repeat the re-run the code exactly. The reported non-determinism was not a result of re-running the code exactly, but of running it in a slightly different state. With this sort of model, "Small changes in parameter values can lead to abrupt changes in the qualitative character of system dynamics." Which is reasonable because in real-life we also expect that small changes in how things turn out may lead to 'abrupt changes in the qualitative character of system dynamics'.
- tripletao 6y ago
  So I understand your view correctly: Are you saying that you expect that in Ferguson's model, changes in intermediate values on the order of one part in 2^53 (a double mantissa) would change the output by tens of thousands of deaths? If so, how could that model possibly be useful, given that none of the inputs are known to anywhere near that? If you're really that sensitive to numerical error, then just averaging multiple multi-threaded runs may be biased in strange ways, because the process adding the uncertainty (thread scheduling or whatever) is neither deterministic nor random with known statistics--a warm day might cause your CPU to throttle, changing the scheduling, significantly changing your result. (To be clear, I don't think that's the case; I think any floating-point error is negligible except to any extent it prevents bit-exact reproducible runs.) The stuff you're quoting above is pretty standard description of dynamical systems in general. It's true, but I don't see how it applies to Ferguson's model with respect to numerical error; and if it does, then I don't see how his model is useful. I could believe his model is chaotic in the sense that the outputs vary dramatically over the expected uncertainties in the inputs, but the numerical inaccuracy is many orders of magnitude below that. Finally, I keep saying their "deterministic from seed" mode was broken multi-threaded or after restarts, and you keep saying it was fine single-threaded in a single start. I think both are true, so I'm not sure what we're arguing about?
  - eesmith 6y ago
    > would change the output by tens of thousands of deaths? Yes, with the modification of the statement to "I would not be surprised if it changes by many percentage points." The bug report you linked to says only "This mostly manifests as a time- offset, but there are numerical changes beyond that." It shows a different by day 80 of 100,000 deaths, or 20% overall. It does not show if by day (say) 120 those numbers have gotten further together or closer. If they both end at about the same numbers, separated only by a week, then it does not change the overall conclusions in the paper at https://www.imperial.ac.uk/media/imperial-college/medicine/sph/ide/gida-fellowships/Imperial-College-COVID19-NPI-modelling-16-03-2020.pdf https://www.imperial.ac.uk/media/imperial-college/medicine/s... . It does shift some of the dates, and I am annoyed with the lack of error bars or the like in their plots. You asked "how could that model possibly be useful, given that none of the inputs are known to anywhere near that?" You check it for sensitivity across a range of values and see how the overall conclusions change. Quoting that paper I linked to, "Such policies are robust to uncertainty in both the reproduction number, R0(Table 4) and in the severity of the virus ... However, there are very large uncertainties around the transmission of this virus, the likely effectiveness of different policies and the extent to which the population spontaneously adopts risk reducing behaviours. This means it is difficult to be definitive about the likely initial duration of measures which will be required, except that it will be several months." When all the lines in a spaghetti map show the hurricane bearing down on you, you prepare for bad weather, even though the hurricane track is almost certainly not going to follow any of those paths. You write "but the numerical inaccuracy is many orders of magnitude below that." What I know about chaotic systems is that I cannot make that conclusion without much more study about the relevant phase space. If the end result is an attractor, then even intermediate chaotic behavior is not much of an issue, depending of course on the specific details of the policy guidance. You write "I think both are true, so I'm not sure what we're arguing about?" My comment started because of your comments at https://news.ycombinator.com/item?id=23213711 https://news.ycombinator.com/item?id=23213711 where you wrote that discussion on the GitHub comments wasn't "terribly clear" to you. I gave my interpretation. This specific thread started because you wrote "They seem to think the model is fine and just the "deterministic from seed" mode is broken", while I don't see in that bug report where they state the '"deterministic from seed" mode is broken'. Is the GitHub commentary clear to you now, and if not, what is unclear?

eesmith 6y ago

> in most (not all, most) practical work, it doesn't turn out to be that important shrug Once upon a time, when I did simulation work, chaotic effects were everywhere. To see if perhaps my views were unusual, I did a search and found https://www.ncbi.nlm.nih.gov/books/NBK305903/ https://www.ncbi.nlm.nih.gov/books/NBK305903/ - "Assessing the Use of Agent-Based Models for Tobacco Regulation", "Appendix B - Agent-Based Models for Policy Analysis" "An agent-based model (ABM) is a computational simulation model of a many-agent system that captures the behaviors of the system’s autonomous agents and their interactions with each other. An ABM is a computational instantiation of a complex adaptive system (CAS). A CAS is a dynamic model that represents individual agents and their collective behavior" That describes the Imperial College model. "ABMs are nonlinear models; highly nonlinear is the usual term. Deterministic nonlinear models have three characteristics that make them difficult to use statistically: " * Sensitive dependence on initial conditions: A map exhibits sensitive dependence on initial conditions at x0 if there is some distance d > 0 such that no matter how close to x0 another initial point x′0 is chosen, xt and x′t will eventually be at least distance d apart. " * Complicated limit dynamics: The limit behavior of typical linear dynamic systems is simple: Either the system converges to a steady state from any initial conditions, or they blow up, diverging to infinity. Nonlinear dynamical systems have much more complicated dynamics, including stable limit cycles of various periodicities, strange attractors, and chaotic behavior. " * Sensitive dependence on parameters: Small changes in parameter values can lead to abrupt changes in the qualitative character of system dynamics. This is in alignment with my understanding of the issues. These are well-known, and I have difficulty understanding your background enough to not have faced them before. Perhaps since your intuition is based on experiences which are distant from this sort of modeling, you can't use your experience as a guideline? You wrote: > it's normal good practice in stochastic code to add a "deterministic from seed" mode There's nothing in the bug report that you linked to which suggests that the result is not "deterministic from seed" on single-threaded code, so long as you repeat the re-run the code exactly. The reported non-determinism was not a result of re-running the code exactly, but of running it in a slightly different state. With this sort of model, "Small changes in parameter values can lead to abrupt changes in the qualitative character of system dynamics." Which is reasonable because in real-life we also expect that small changes in how things turn out may lead to 'abrupt changes in the qualitative character of system dynamics'.

tripletao 6y ago
So I understand your view correctly: Are you saying that you expect that in Ferguson's model, changes in intermediate values on the order of one part in 2^53 (a double mantissa) would change the output by tens of thousands of deaths? If so, how could that model possibly be useful, given that none of the inputs are known to anywhere near that? If you're really that sensitive to numerical error, then just averaging multiple multi-threaded runs may be biased in strange ways, because the process adding the uncertainty (thread scheduling or whatever) is neither deterministic nor random with known statistics--a warm day might cause your CPU to throttle, changing the scheduling, significantly changing your result. (To be clear, I don't think that's the case; I think any floating-point error is negligible except to any extent it prevents bit-exact reproducible runs.) The stuff you're quoting above is pretty standard description of dynamical systems in general. It's true, but I don't see how it applies to Ferguson's model with respect to numerical error; and if it does, then I don't see how his model is useful. I could believe his model is chaotic in the sense that the outputs vary dramatically over the expected uncertainties in the inputs, but the numerical inaccuracy is many orders of magnitude below that. Finally, I keep saying their "deterministic from seed" mode was broken multi-threaded or after restarts, and you keep saying it was fine single-threaded in a single start. I think both are true, so I'm not sure what we're arguing about?
- eesmith 6y ago
  > would change the output by tens of thousands of deaths? Yes, with the modification of the statement to "I would not be surprised if it changes by many percentage points." The bug report you linked to says only "This mostly manifests as a time- offset, but there are numerical changes beyond that." It shows a different by day 80 of 100,000 deaths, or 20% overall. It does not show if by day (say) 120 those numbers have gotten further together or closer. If they both end at about the same numbers, separated only by a week, then it does not change the overall conclusions in the paper at https://www.imperial.ac.uk/media/imperial-college/medicine/sph/ide/gida-fellowships/Imperial-College-COVID19-NPI-modelling-16-03-2020.pdf https://www.imperial.ac.uk/media/imperial-college/medicine/s... . It does shift some of the dates, and I am annoyed with the lack of error bars or the like in their plots. You asked "how could that model possibly be useful, given that none of the inputs are known to anywhere near that?" You check it for sensitivity across a range of values and see how the overall conclusions change. Quoting that paper I linked to, "Such policies are robust to uncertainty in both the reproduction number, R0(Table 4) and in the severity of the virus ... However, there are very large uncertainties around the transmission of this virus, the likely effectiveness of different policies and the extent to which the population spontaneously adopts risk reducing behaviours. This means it is difficult to be definitive about the likely initial duration of measures which will be required, except that it will be several months." When all the lines in a spaghetti map show the hurricane bearing down on you, you prepare for bad weather, even though the hurricane track is almost certainly not going to follow any of those paths. You write "but the numerical inaccuracy is many orders of magnitude below that." What I know about chaotic systems is that I cannot make that conclusion without much more study about the relevant phase space. If the end result is an attractor, then even intermediate chaotic behavior is not much of an issue, depending of course on the specific details of the policy guidance. You write "I think both are true, so I'm not sure what we're arguing about?" My comment started because of your comments at https://news.ycombinator.com/item?id=23213711 https://news.ycombinator.com/item?id=23213711 where you wrote that discussion on the GitHub comments wasn't "terribly clear" to you. I gave my interpretation. This specific thread started because you wrote "They seem to think the model is fine and just the "deterministic from seed" mode is broken", while I don't see in that bug report where they state the '"deterministic from seed" mode is broken'. Is the GitHub commentary clear to you now, and if not, what is unclear?