After reading about the implementation of this model and experimenting with different a while, I've made some observations about its behavior and especially abo

After reading about the implementation of this model and experimenting with different a while, I've made some observations about its behavior and especially about the values that have been chosen for its for its parameters which I feel are important to consider when interpreting its output. TL;DR: Many of the critical inputs to this simulation are based off of almost entirely unknown values, and relatively small changes to these inputs can swing the results of the simulation by orders of magnitude. The defaults that the website creators have selected seem pessimistic on average compared to even the source research which can have the effect of misleading users as to what the expected outcome of the pandemic is. Since under-reacting and over-reacting to this situation both have huge real-world ramifications, making assessments based off of even a fully valid and well-designed model can lead to incorrect decisions being made if its parameters are set unrealistically. ---- First of all, there are a TON of variables and many of them pull values from extremely thin data. Although it looks like the researchers didn't just make up any of their values, there are several instances of very impactful parameters being hand-waved: ## MITIGATION CURVE The "Mitigation" ratio curve, an unsurprisingly critical variable for the simulation, seems too high even for the "strong" mitigation preset. Even after looking extensively, I couldn't find any explanation or justification from the simulation's creators as to how they created their default curve. As this is possibly the most impactful parameter in the entire simulation, that just makes it more critical to correctly pick its value. Judging by the degree of dramatic measures that society has taken so far and the actions of people that I've observed, I'd personally estimate that ratio to be much lower than even the one included in the "strong" mitigation curve. Of course this is random guesstimation of my own here, but it goes to illustrate the massive changes in results that can stem from small changes to the input values they've chosen. After starting at default settings, setting population to USA, and setting mitigation factor to "strong", further adjusting the March 15 mitigation factor value down 10% from 60% to 50% (a value that seems extremely reasonable and even generous to me) causes the total number of cases to drop by nearly 66% holding all other values constant. Adjusting it to 40% reduces the total infections by another nearly 50% from there. Another paper that is cited by the simulation's authors writes that "[...] by taking drastic social distancing measures and policies of controlling the source of infection, with the tremendous joint efforts from the government, healthcare workers, and the people (Fig. 1), Rt was substantially reduced [from 3.58] to 0.32 in Wuhan after February 2, which was encouraging for the global efforts fighting against the Covid-19 outbreak using traditional non-pharmaceutical measures [...]" [4] This would seem to represent a mitigation ratio of 0.1 after just around a month. Clearly there are differences in the way that the pandemic was handled in China compared to how it's being handled now in the rest of the world, but China's ability to combat it that effectively seems to lend credence to the idea that mitigation ratios of less than 0.5 are possible if not already in place in the US and across the rest of the world. In any case, claiming nearly 80% of the transmissibility of this virus remaining at this point in time and maxing out at 60% as the "moderate" case seems dangerously exaggerated to the upside and potentially skews the results of the whole simulation due to how critical this one variable is in determining the output. ## ANNUAL AVERAGE R0 Another foundational variable is R0: The number of additional infected individuals per infection. The linked research paper states "the early human-to-human transmission of 2019-nCoV was characterized by values of R0 around 2.2 (median value, with 90% high density interval: 1.4–3.8)" [2] Although the distribution from the research paper has a long tail towards higher values, setting a value of 2.7 for "Moderate/North" feels somewhat disingenuous given that the mean is 2.2. Of course other research papers list a wide range of different values for this variable, so perhaps expanding the range of the presets would be a better option. In any case, this variable is dominated by the mitigation factor in cases of high mitigation, so its precise value may not matter as much in those situations. ## SEASONAL FORCING The "seasonal forcing" factor varies from 0 to 0.2 in all of their "Epidemiology" presets. On their about page, the example they provide in their graph seems to have a value of ~0.6, but that may just be an illustration using non-realistic values for visual effect. [1] One other thing to note is that their implementation uses the selected R0 as a mean value for their function, meaning that at the peak month the true R0 is `(1 + <seasonal forcing>) * R0`. This doesn't agree with the data from the research paper, which estimated their R0 value using data from the most infectious period (winter). Assuming a seasonal forcing factor of 0.2, that means The R0 values they provide are actually 20% inflated in January on top of their already high values. Of course, their very choice of a cosine wave to model that impact seems largely unfounded, but of course given a lack of data pointing to a more accurate option it's as good as any. That being said, even a small change to that function could have massive impacts on the simulation's results. ## ICU OVERFLOW In the most pessimistic scenarios, their implementation of additional fatalities caused by ICU overflow fail to take into account potential measures such as emergency hospitals, things like the hospital ship being sent over to New York by the navy [3], and temporary increases to hospital capacity. In most cases, the peak of ICU overflow doesn't occur until months down the line, giving a lot of time to scrape up resources to increase capacity. Perhaps providing an additional value to adjust the hospital capacity over time would be useful for accurately representing the impact of ICU overflow. ## OTHER VARIABLES + NOTES - "Imports" is held constant through the entire simulation. It seems unlikely that it would remain this way; setting this as a curve instead would make more sense in my opinion. Also, I couldn't find any justification or explanation for the values they picked for that parameter. - Improvements to treatment causing ICU/hospital stay time to go down aren't accounted for. Of course there's no guarantee that it will change significantly as the pandemic progresses (or even that it will not get worse due to some mutation of the virus or other circumstance), but recent research and experimentation seems to already be making progress in creating more effective treatments for the virus. - Due to how incredibly vast the system this model models is, there are an uncountable number of possible external events that could, in some situations, invalidate the whole thing entirely. At least mentioning the possibility of things such as a vaccine being released in the coming months, mutations with developed antiviral drug resistance, etc. seems prudent to me. ---- NOTE: I'm not a trained statistician or epidemiologist - I just have experience working with data and compute models. I'd appreciate any feedback about the accuracy of my comments here or expansions/refutations of my thinking. [1] Link from about page: https://neherlab.org/covid19/assets/seasonal_illustration.151cec1.svg; https://neherlab.org/covid19/assets/seasonal_illustration.15... Screenshot at time of writing: https://ameo.link/u/7pk.png https://ameo.link/u/7pk.png [2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7001239/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7001239/ [3] https://www.nbcnewyork.com/news/local/navy-medical-ship-coming-to-relieve-nyc-hospitals-amid-virus-was-last-in-city-after-9-11/2333767/ https://www.nbcnewyork.com/news/local/navy-medical-ship-comi... [4] https://www.medrxiv.org/content/10.1101/2020.03.03.20030593v1.full.pdf https://www.medrxiv.org/content/10.1101/2020.03.03.20030593v...