3 ms·

"No model is ever "fixed" if that means 100% correct." One definition of broken for a proposed alternative to the status quo is if the alternative under-perfor

"No model is ever "fixed" if that means 100% correct." One definition of broken for a proposed alternative to the status quo is if the alternative under-performs it. Kind of makes one ask why anyone would adopt it to begin with. There's a simple model using temperature that works pretty well. Google's solution is said to perform worse than that with more false positives. Google's isn't "fixed" or "working" until they show it outperforms the simple solution that works with similar error margin and cost. In other words, it isn't good until people would want to give up existing method to get extra benefits or cost savings new one brings.

nl 9y ago
In other words, it isn't good until people would want to give up existing method to get extra benefits or cost savings new one brings. They do. The current state of the art methods used "in production" today all use Flu Trends data from Google[1], other forms of digital data[2], ensemble methods incorporating them all, or human-based "crowdsourced" forecasting[3] One definition of broken for a proposed alternative to the status quo is if the alternative under-performs it. Which is not what happened here. Here's the report into the 2014 CDC's Flu Forecasting competition: https://bmcinfectdis.biomedcentral.com/articles/10.1186/s12879-016-1669-x https://bmcinfectdis.biomedcentral.com/articles/10.1186/s128... Note that there is no mention of the "simple mean temperature" model. That's because it isn't very useful. That model predicts flu increases in winter, and picks up minor variations because of weather patterns. To simplify even further, you can average all the CDC flu data and use that as your prediction and on an average year you'll have a decently performing model. This isn't useful as a forecast, because the people who need forecasts already know this. Better models (eg, SI, SEIR, Hawkes process based etc) can sometimes pick up epidemic or unusual conditions, but only after the conditions have changed. This is still useful, because there is a (best case) 2 week lag between ground conditions and CDC data being available. Digital surveillance techniques (Flu Trends, Twitter data, etc) all push that data lag back. This is incredibly useful for the people who need forecasts because it gives them lead time. To understand this you need to consider the metrics. The most common metrics for flu forecasting is the "peak week", and "number of people infected at peak". Sometime the total number of people infected in a season is also reported. Temperature-based models do really well on average at both these tasks, but they fail completely at picking the unusual seasons. Google's solution is said to perform worse than that with more false positives. Google flu trends picked the 2009 epidemic season really well, but failed in the 2013 season (when it falsely picked an epidemic). On average that might make it worse than a temperature based model, but that is just bad selection of metrics. It's like reporting average income when your sample has a billionaire: the metric is misleading. If that isn't the perfect example of "bullshit" then I don't know what is. [1] http://www.healthmap.org/flutrends/about/ http://www.healthmap.org/flutrends/about/ [2] http://delphi.midas.cs.cmu.edu/nowcast/about.html http://delphi.midas.cs.cmu.edu/nowcast/about.html https://gcn.com/articles/2016/12/21/cdc-flu-predictions.aspx https://gcn.com/articles/2016/12/21/cdc-flu-predictions.aspx

nl 9y ago

In other words, it isn't good until people would want to give up existing method to get extra benefits or cost savings new one brings. They do. The current state of the art methods used "in production" today all use Flu Trends data from Google[1], other forms of digital data[2], ensemble methods incorporating them all, or human-based "crowdsourced" forecasting[3] One definition of broken for a proposed alternative to the status quo is if the alternative under-performs it. Which is not what happened here. Here's the report into the 2014 CDC's Flu Forecasting competition: https://bmcinfectdis.biomedcentral.com/articles/10.1186/s12879-016-1669-x https://bmcinfectdis.biomedcentral.com/articles/10.1186/s128... Note that there is no mention of the "simple mean temperature" model. That's because it isn't very useful. That model predicts flu increases in winter, and picks up minor variations because of weather patterns. To simplify even further, you can average all the CDC flu data and use that as your prediction and on an average year you'll have a decently performing model. This isn't useful as a forecast, because the people who need forecasts already know this. Better models (eg, SI, SEIR, Hawkes process based etc) can sometimes pick up epidemic or unusual conditions, but only after the conditions have changed. This is still useful, because there is a (best case) 2 week lag between ground conditions and CDC data being available. Digital surveillance techniques (Flu Trends, Twitter data, etc) all push that data lag back. This is incredibly useful for the people who need forecasts because it gives them lead time. To understand this you need to consider the metrics. The most common metrics for flu forecasting is the "peak week", and "number of people infected at peak". Sometime the total number of people infected in a season is also reported. Temperature-based models do really well on average at both these tasks, but they fail completely at picking the unusual seasons. Google's solution is said to perform worse than that with more false positives. Google flu trends picked the 2009 epidemic season really well, but failed in the 2013 season (when it falsely picked an epidemic). On average that might make it worse than a temperature based model, but that is just bad selection of metrics. It's like reporting average income when your sample has a billionaire: the metric is misleading. If that isn't the perfect example of "bullshit" then I don't know what is. [1] http://www.healthmap.org/flutrends/about/ http://www.healthmap.org/flutrends/about/ [2] http://delphi.midas.cs.cmu.edu/nowcast/about.html http://delphi.midas.cs.cmu.edu/nowcast/about.html https://gcn.com/articles/2016/12/21/cdc-flu-predictions.aspx https://gcn.com/articles/2016/12/21/cdc-flu-predictions.aspx