10 ms·
Why it’s so hard to make a good Covid-19 model
- lonelappde 7y agoThe article is a long listing of ways to say GIGO. It's hard to get good clean accurate data.
- foolzcrow 7y agoExactly. WHO Datasets appear to be causing models to grossly over estimate numbers in various regions. Hmmm.
- ccsnags 7y agoGIGO is a principle i learned at audio engineering school and it applies to all input/output relationships across multiple disciplines. Have a 4k tv but only a 1080 cable box? Your tv will only show 1080. Did you record too much compression on the master track? Now you have to deal with that in the mix. GIGO is any time you get an input that will never give a desired or expected output. In circuits, its easy. It works or it doesn’t. In data science, GIGO can be hidden by hiding the assumptions made to reach the level of knowable information from the data provided. Data scientists have a huge uphill battle right now. It is far more complex than looking at the numbers and trying to find patterns. People can find patterns in clouds.
- paganel 7y agoJust this evening some Italian scientist announced on his twitter account that the number of new positive cases does not correspond to the number of new tests carried out and announced for that day, because those tests could have actually been made 2 or 3 days before or something like that. Which, presumably, instantly invalidated all the charts and data-modelling based on the "number of new positive cases" / "total number of tests made" (with a lower value being seen as better). But the Region of Sicily (or its official twitter account, anyway) replied that in their case the number of new positive cases and the total number of tests made are indeed correlated, which of course means that everything is a mess in terms of data coming in and its significance. Later edit: For those who know Italian this is the tweet [1] I was writing about, and it looks like I was remembering wrong, the guy is not a scientist per se, more like a "data scientist", he seems to be working at a company very similar to fivethirtyeight (but presumably focused on the Italian market). [1] https://twitter.com/lorepregliasco/status/1248279589337645058 https://twitter.com/lorepregliasco/status/124827958933764505...
- tomrod 7y agoThat's right, difference between the date of specimen collection and date of announcement. General approach is to announce as soon as positive confirmation hit. There is a vintage to this data. Source: working in this dataspace as we speak. Keep up the crowd-sourced wisdom HN, it helps prevent us data science folks in thick of it from keeping blinders on!
- 3fe9a03ccd14ca5 7y agoWhat about our models for global warming?
- chrisco255 7y agoCorrect. Non-linear dynamic multivariate systems are very difficult to model and uncertainty is a function of time. That's why we can't predict the weather more than a few days out with any degree of accuracy.
- xienze 7y agoWell those are obviously perfect. /s
- ben_w 7y agoThose two things are unrelated. Unlike a virus, the laws of physics don’t appear to mutate. Also, you can’t measure the quality of your model of virus progression with earth observation satellites or hundreds of thousands of years worth of ice core data. Also, rather unfortunately in this case, the lack of meaningful action by world governments means that none of the climate models have to account for feedback caused by the world actually acting on the results of those the models.
- guscost 7y agoIt’s going to be fun to watch this flood of nonsense convince precisely nobody for the next few years.
- Symmetry 7y agoCompared to a spherical cow model we could make as soon as we realized that CO2 the oceans absorbed didn't stay absorbed we've only done a little better. It's a complicated system, we don't know all the detail, and the calibration of the older models we've been able to robustly test at least just wasn't there. Compared to constant model where the temperature stays the same both the modern sophisticated model and the spherical cow model do much, much better. For simple things like "how much infrared light and in what bands does CO2 absorb" we can measure that in a laboratory and have it nailed down.
- tgafpc2 7y agoBecause people have unrealistic expectations.
- jefftk 7y agoNote that this article is from March 31st, and while that wouldn't normally be very long ago things are moving extremely quickly.
- moate 7y agoIn the 9 days since we still have models that are all over the place. Nate Silver covers the difficulty in predictive modeling of pandemics in his book "The Signal and the Noise" and in the years since it was written not much has changes to they're basically rehashing the points he made then.
- CodeWriter23 7y agoWell, politically things are moving quickly too. Bottom line, all the “science” failed to engage The Cautionary Principle when they should have known that what they did know was likely outweighed by what they didn’t.
- jefftk 7y agoI can't tell what you think should have happened differently here
- mcguire 7y ago"Meanwhile, a report from Imperial College London that made headlines for its dire, modeling-based forecasts predicted about 2.2 million U.S. deaths from the coronavirus, if nobody changes their everyday behavior." On the other hand, they're still quoting that study, which was very out of date well before the 31st.
- Thrymr 7y agoWhat this article misses is that simple models of complex systems in science are most useful for understanding the dynamics of phenomena, not for making accurate quantitative predictions. This is not a model of a mass accelerating in a vacuum where Newton's laws are sufficient to a high degree of accuracy, or even a numerical model of the aerodynamics of an airplane, where the physics are well understood but there are far too many particles to solve analytically. This is a parameterized model for the behavior of millions of people, each one's reaction to exposure to viral particles, legal and social norms, personal and economic situations, and so on. Of course we are not going to be able to predict the future of an unprecedented event like this quantitatively. It's not just that data are hard to obtain accurately; the models themselves are so simplified that they can't capture much of the important dynamics going on.
- jhayward 7y agoAgree. I think a better exposition on COVID-19 models is Zenep Tufecki's article in The Atlantic [1]. TLDR: They're more directional roadmaps, showing you which possible outcome paths you need to prune off the tree of outcomes, not exact numerical predictions. [1] https://www.theatlantic.com/technology/archive/2020/04/coronavirus-models-arent-supposed-be-right/609271/ https://www.theatlantic.com/technology/archive/2020/04/coron...
- dekhn 7y agoIn a sense, much of this work is similar to A* search and reinforcement learning. You have a limited view of the world, the responses to your actions are not directly causal, but if you can use the data you have to make reasoned decisions on what parts of the (extremely large) action space are unlikely to pay off, you can avoid wasting resources on them.
- sesuximo 7y agoIt's hard to use a roadmap with this much variance. Is there anything that can be said with certainty about covid
- 7y ago
- deleted 7y ago[deleted]
- ape4 7y agoWho thought it would be easy. One person can decide to break quarantine and infect other people.
- sgt101 7y agoVery odd little diagrams to show model components in this article - why don't they use digraphs like everyone who knows how to do stats? oh...
- francisofascii 7y agoIn jest, I am imagining a bunch of data scientists working late into the night, running the numbers and various scenarios, and then one person finally stands up and says, "FK it, let's just shut down the whole country and hope for the best."
- sesuximo 7y agoThat's totally unrealistic. A data scientist is never in the room where decisions are made
- ck2 7y agoGood luck trying to predict how much of a population is going to refuse to isolate and interact anyway. Also relies on governments and politicians to not fudge testing and death counts, which is never going to be accurate. btw the financial times has maybe the best graph on the stats however flawed: http://com.ft.imagepublish.upp-prod-us.s3.amazonaws.com/225197b0-79d4-11ea-af44-daa3def9ae03 http://com.ft.imagepublish.upp-prod-us.s3.amazonaws.com/2251...
- watwut 7y agoThe actual model used by UK imperial college estimated compliance with karantene of sick people 75% and compliance with general stay-at-home 50%. What I find frustrating is that actual models used by epidemiologists back in february and March incorporated pretty much all "gotchas" non-epidemiologists discovered today. And it does not matter, because non-epidemiologists still assume they are first ones to ever discover them.
- KaiserPro 7y agoIts pretty simple: there isn't any decent data. The data we have is strongly biased to older and sicker people. There is no systematic surveillance of a geographic area, only panic testing of those who are showing symptoms. Until there is sampling of a a borough, city or town, from start to finish, we will have wildly wrong models. The only thing that we can plot reasonably accurately is the exponent of the fatalities, but even then its because its based on mostly hard data (unless its china...)
- jmoss20 7y agoThere have been a few of those samples. https://www.cebm.net/covid-19/covid-19-what-proportion-are-asymptomatic/ https://www.cebm.net/covid-19/covid-19-what-proportion-are-a...
- deleted 7y ago[deleted]
- GoodJokes 7y agoNate silver loves to attempt to make predictions based on shitty data. I can’t think of worse data set than polling.
- arjun_tina 7y agoBest article I've read through this entire Pandemic.
- systemvoltage 7y agoCompletely tangent observation: What's the point of using sketch scribbles over the diagrams? They could just make it in powerpoint and simplify. It would be easier to read as well. Decoration for the sake of decoration? Why?
- simonsarris 7y agoCharitably: To express the tentative nature of the information or model. "Here's our working theory." (I think it makes it somewhat hard to read in this case)
- _Microft 7y agoI like the look of it.
- YetAnotherNick 7y agoI tried to do some exploration with the data to get some idea on the final number. One of the best plot I found that could tell the final number is plotting the percentage of cases in the next week with the cases till now. It is like negative half parabola and the time it meets the x axis will give the final number per country. This is the final plot: https://i.imgur.com/5p4Xife.png https://i.imgur.com/5p4Xife.png. You could make a good guess from the data how many people it will affect in the lifetime per country by continuing the same pattern till it reaches x axis.
- m0zg 7y ago> Why it’s so hard to make a good Covid-19 model Because there's no downside to the authors for overpredicting deaths and resource use, and _a lot_ of downside for underpredicting. So all the "bad" things get taken into account, and all the "good" things are ignored. One thing I've reinforced in my view of the world is that common sense is very uncommon indeed. "2 million deaths" my ass. I hope people reconsider their trust in other models that are "hard to make".
- socalnate1 7y ago2 Million deaths was without all the drastic measures we've taken. I'm having a hard time understanding what exactly you are suggesting we (society) should have done here. "Used our common sense" to do what?
- m0zg 7y agoSo what you're saying is Donald Trump saved 2 million lives, then? :-) I think once we're through this, you will see that there wouldn't be 2 million deaths no matter what, although the measures did help, of course. The current IHME models get routinely revised downward by a lot (to 1/3-1/4th the initial figures) even though _they assumed "measures" right from the start_. As of this morning, the projected fatalities in the US dropped again by a quarter, to 61K, with the lower bound at 31K. My prediction? We will be nearer to the lower bound. Why do I think that? There are fewer than 10K "serious" cases in the US, and over the past week intubations in NY have dropped so precipitously, Cuomo doesn't mention them in his pressers anymore.
- macintux 7y agoI don’t know that anyone expected (nearly) every state to impose a stay-at-home order. Crashing our economy into a brick wall is not an easy decision. I frankly expected things to get much worse before state governments took action. Obviously we could have done much better, much sooner, but I certainly can believe why early estimates were much more pessimistic. And that’s all disregarding the fact that we still don’t know how/when this ends. We might still be in the early days.
- tunesmith 7y agoI'm still curious about the relationship between R0 and doubling period, because you can kind of see a relationship between the two by examining contagion period. In general we've seen doubling periods that seem to suggest 5-6 days, but occasionally as fast as 3 days, unclear how distorted those numbers are by testing and mitigation and misattributed deaths. If it has a natural R0 of 6, then it means that an infection will infect 6 others within that contagion period. If we're thinking R0 is not 2-3 but is instead 5-6, then to make consistent with the doubling periods we are seeing, it'd mean that people are contagious for a longer period of time than we first thought.
- thedudeabides5 7y agoWe know it's hard Nate, it's hard for everyone else too. That being said, I'd pay good money to hear his latest numbers...
- scared2 7y agoIt's novelty. Basically all models are based on assumptions in closed system. As one gets more information they add it to the closed system eventually making it easier to predict. As we know reality is extremely heterogeneous and an open system with too many interactions. This in principle makes all practically wrong. But it does not mean that they are not useful. (At least they are useful to make scary graphs that convince presidents). I had a lame model that is currently predicting lower death rate but in a similar trend [1]. In this model my assumption is the lockdown to continue for 45 days. The result which I regret to have seen shows a scary number of 600k deaths after 39 days. So I'm hoping for something spectacular to happen such as vaccine, a drug, the sun etc that I would use to change my prediction. [1] https://news.ycombinator.com/item?id=22814927 https://news.ycombinator.com/item?id=22814927
- brummm 7y agoGarbage in, garbage out! Can't build good models with data that's terrible. Inconsistent testing, different reporting methods, lying from governments.
- sesuximo 7y agoGarbage in policy out
- mrfusion 7y agoDon’t all these models ignore the effects of warmer weather? A few studies have suggested there’s at least some effect.
- oehpr 7y agoI've found this topic pretty interesting, and I've enjoyed trying my hand at it myself. One of the things I've been playing with is Insight Maker https://insightmaker.com/ https://insightmaker.com/ This site is a totally free platform where you can set up the kinds of simulations this article describes (stock and flow models). You can even specify your uncertainty in your baseline assumptions and run sensitivity analysis to see what the relative impact of each factor is on the model, and the range of potential outputs you could have. This system is very much like https://www.getguesstimate.com/ https://www.getguesstimate.com/, except much more flexible and way less intuitive. Insight maker isn't a professional tool, it's really more of an advocacy and outreach platform, but despite that it's really quite powerful. I think after you've read this article and internalized the difficulties in modeling pandemics (and have re-affirmed to yourself that you are not an epidemiologist, unless you are, more power to you if so), you might have some fun trying to build the model this article describes.
- deleted 7y ago[deleted]
- nowweknow 7y agoThe results of lockdown in Spain and Italy are not in this post and they provide crucial information. The lockdown in Spain has reduced in two weeks the daily infection rate from 42% to 4%, so there is hope for the future. But the problem is that the economy can't cope with the lockdown so we have a big problem. Also since the R0 can be reduced so much with political decisions, the emphasis should be on the political and economical ground.
- fxj 7y agoI really like the neher lab prediction tool. https://neherlab.org/covid19 https://neherlab.org/covid19 You can choose different scenarios and compare it with real data. So you can match the parameters of your model to the actual measured data (number of deaths) and it also has data about the population and the state of ICUs and hospitals that you can use in your prediction. In the end it is just a tool which can give completely wrong predictions, but you get a felling for what could and could not happen.