6 ms·

The Parable of Google Flu: Traps in Big Data Analysis [pdf]

68 points by tokenadult 13y ago

nowarninglabel 13y ago
The actual study: http://www.sciencemag.org/content/343/6176/1203 http://www.sciencemag.org/content/343/6176/1203 Gist of it is, Google Trends overestimated flu cases by about double for 100 out of 108 weeks, according to the paper. Edit: There is actually an interesting podcast available on it without a paywall: http://podcasts.aaas.org/science_podcast/SciencePodcast_140314.mp3 http://podcasts.aaas.org/science_podcast/SciencePodcast_1403...
- pyre 13y ago
  > Google Trends overestimated flu cases by about double for 100 out of 108 weeks Well, there's a simple solution to that... Just halve all of the results from the current algorithm!
- deleted 13y ago
  [deleted]
nraynaud 13y ago
I would not stone them, they tried, it doesn't work well, they can fix it or scrap it. At least they tried to use their technology for something positive this time. It's one thousand times better to fail when trying to predict the flu epidemics than successfully using technology to bomb weddings.
- rm999 13y ago
  >It's one thousand times better to fail when trying to predict the flu epidemics than successfully using technology to bomb weddings. That's a pointless and irrelevant comparison. There's a cost to false positives, e.g. creating more vaccinations than necessary; fortunately I don't think the CDC or any other government organizations made any policy decisions based on Google Flu. As a person who works with a lot of data, I think applications like Google Flu have a lot of potential to do a lot of good. But it sounds like Google wasn't properly rigorous in their experiments if Google Flu is such a failure.
gwern 13y ago
Ironically, the original paper (fulltext) fell short of the front page in both submissions: https://news.ycombinator.com/item?id=7405286 https://news.ycombinator.com/item?id=7405286 and https://news.ycombinator.com/item?id=7422496 https://news.ycombinator.com/item?id=7422496 /newest can be so random.
j2kun 13y ago
> When 80-90% of people visiting the doctor for “flu” don’t really have it, you can hardly expect their internet searches to be a reliable source of information. I just attended a talk that graphed the rate of physicians' diagnoses versus the truth, and the conclusion was that about half of physicians drastically overestimate, too (and it's far more than double, closer to five times more!). So when 80-90% of people visiting the doctor for flu don't really have it and those doctors often make type II errors, you can hardly expect their diagnoses to be a reliable source of information.
- nraynaud 13y ago
  flu is like the communist, everybody is afraid of it, everybody believe they have seen one, but upon serious examination we can't find one. I guess the flu epidemic is just a conspiration created to distract us from the fact that nobody walked on the moon or something. On a more serious note, what's exactly the deal with misdiagnosis? If the real thing is another virus, it could also be a virus that we should treat in a similar manner? Or the doctors are loosing precious time often with a wrong first diagnosis in a significant number of cases?
  - Fomite 13y ago
    Diagnosing viral illness is a pain. There are a huge number of winter circulating viruses with roughly the same pattern as the flu, and diagnostic tests for influenza are often very specific but not very sensitive. Meaning if they come up positive, you have it, but if they come up negative...shrug Beyond that, no, it's not likely to be a virus treated in a similar manner. If by manner you mean "drugs". Oseltamivir (aka Tamiflu) only works against influenza, so if you have RSV or some other virus, it won't do you much good. The other way to handle viral diseases is to let them run their course and manage the complications, at which point a strictly accurate diagnosis isn't particularly necessary. TL;DR: It's very hard, fairly expensive, and the payoff per-patient isn't strong.
deleted 13y ago
[deleted]
ffk 13y ago
A few thoughts... Is it possible the CDC estimates are off by half? Two main points come to mind: * How many Type II errors do doctors commit? * How many people with the flu do not visit a doctor? People with good health coverage are probably more likely to visit the doctor preemptively when they are feeling unwell, potentially skewing the results. Also, while google may be off in the exact measurement when compared to the CDC, it looks like the shape of the graph correlates with the CDC data. Overall, this looks approach promising.
- shakethemonkey 13y ago
  Google isn't trying to predict actual flu cases. They're trying to predict the CDC estimates.
- Fomite 13y ago
  The problem with "the shape of the graph" correlating is that when modeling a seasonal disease, this is trivially easy. Public health decisions are made on the magnitude of some of those measurements, not their rough shape, and so being off on the exact measurement is a pretty big problem.
mdisraeli 13y ago
This is interesting and a much needed discussion, however it too completely ignores a key difference: CDC tracks hospitalisation based on lab reports, GFT tracks search terms The assumption, which admittedly Google is making, is that these two are correlated by the same amount each year. However, in periods of economic downturn without free health care, people are going to search more and seek actual medial assistance less. Given the global economy has been shot for the last six years....
- ChuckMcM 13y ago
  > CDC tracks hospitalisation based on lab reports, GFT tracks search terms This is the key difference, and after 5 years it is clear that this is not a valid way to predict the flu, or even monitor it. I think it was a great experiment, and thought it was pretty creative when announced, and now it should either be retired or replaced with a new hypothesis. Because as the Mythbusters would say, this myth is busted.
  - mdisraeli 13y ago
    I disagree, but it depends what you mean by "the flu". If you mean hospitalisations, then yes, you're entirely right. If you mean lost productivity and population health outside of hospital, however, perhaps we need a better means to track this other than indirectly via either current stat.
    - Fomite 13y ago
      The CDC also uses a outpatient physician network to report cases of influenza-like illness in the community.
      - mdisraeli 13y ago
        I didn't know that, thank you! I went digging to try and find exactly what figure Google were using, and exactly how CDC came up with it, but I couldn't manage to pin them down...
  - 13y ago
simonlebo 13y ago
This article argues that the system overestimated the cases (which it did) and therefore did not work. To reproduce the results of traditional systems is not the purpose of systems like GFT. If you look at the graphs you can see that GFT provides an indication of an increase in flu cases before the traditional systems. This is where it has value; it provides an early indication of an emerging problem.
- Fomite 13y ago
  To reproduce the results of traditional systems is exactly the purpose of Google Flu trends. This is evident when you go back and read the original paper. The question has always been "Can we match CDC estimates faster and with different information?" The emerging answer is: "No".
blauwbilgorgel 13y ago
Mean absolute error (MAE) during the out-of-sample period is 0.486 for GFT, 0.311 for lagged CDC, and 0.232 for combined GFT and CDC. A combination of GFT search behavior and actual CDC numbers produced the best model for the authors of this paper. The (media) spin is "GFT as a stand-alone flu tracker fails". Perhaps the article would be of a different tone if the researchers had full access to Google's data. I don't think the mentions of irony, failure, misleading and hubris are deserved nor offer a fair portrayal.
scottfr 13y ago
One issue is that the number of historical data points used in the model is really very very small. The author's point out that the model initially was fit to 1,152 CDC data points. However, these 1,152 data points are highly temporally correlated (the value at time t is a strong predictor of the value at time t+1). As such, they aren't "worth" very much when building a model. Therefore, the effectively independent number of points (what you generally need to build a good model) is in reality much smaller. Looking at the graphs, it appears that for each flu season there is a pretty regular spike of flu incidences. This spike could probably be summarized effectively by four parameters: - Mean - Standard Deviation - Skewness - Kurtosis Looking at it this way, the number of "independent" data points of a flu season is not 365 (one for every day the year), 52 (one for every week), or even 12 (one for every month). Instead it is only 4. Thus the total amount of data they had to build their model was 4*(# years). When you have such small data sets, you can't expect to obtain good results. TL;DR. Time series are very hard to use in predictive models. It is often impossible to generate good predictive models based off them.
- Fomite 13y ago
  The most vexing thing about the research I did on influenza was that my yearly estimates meant despite having a staggering amount of data, my functional N for many experiments was ~ 35, and that was because I reached way into the past.
colin_mccabe 13y ago
This seems like a case of circular reasoning. "Google's results are not as good as the CDC's because Google's results are different than the CDC's." But why do we believe that the CDC estimates are better? I have never gone to the doctor when I had a flu. If it had lasted more than one or two days, or if I had a temperature higher than 102, then I would have gone. But as it was, nobody ever knew that I had the flu except myself and my family. Even if I had wanted to go to the doctor for every time I got a flu or a cold, it takes at least a day or two to schedule a doctor's appointment. By the time the appointment had rolled around I would probably have cancelled, since the flu doesn't usually last very long. This being the US, there are also a lot of uninsured people who can't go to the doctor even if they wanted to. For those people, Google is the only option. The CDC can collect as much data as it wants from the laboratories, but you can't collect what isn't there. When you add in the uninsured people and the people who get better before visiting the doctor, it's not at all surprising that the lab estimates are 2x lower than reality. In fact, I would kind of expect them to be even lower than that.
- scottfr 13y ago
  The Google model is designed to predict the CDC levels, not predict actual flu prevalence (which is unknown). Error in the Google model is by definition the difference from the CDC levels.
  - colin_mccabe 13y ago
    Sure, it's great to compare the Google model with the CDC's. But if there are persistent differences, does it make sense to immediately assume that the Google model is worse at predicting reality than the CDC? Saying that CDC is right "by definition" is not very helpful. [edit: For example, the author of the paper could have compared Google and CDC versus a third data source, like sales of cold medicine. That would have been an interesting comparison, and might have shed some light on which estimates were a better reflection of reality.]
    - Fomite 13y ago
      The CDC's definition is pretty much a gold standard. Things like "sales of cold medicine" and the like fall under the heading of "syndromic surveillance", and believe me, they've been tried for the better part of a decade without yielding much that's impressive. Which is frustrating, because theoretically they should be amazing, but they just aren't, especially when asked to predict rather than back fit.

68 points by tokenadult 13y ago

The actual study: http://www.sciencemag.org/content/343/6176/1203 http://www.sciencemag.org/content/343/6176/1203 Gist of it is, Google Trends overestimated flu cases by about double for 100 out of 108 weeks, according to the paper. Edit: There is actually an interesting podcast available on it without a paywall: http://podcasts.aaas.org/science_podcast/SciencePodcast_140314.mp3 http://podcasts.aaas.org/science_podcast/SciencePodcast_1403...

pyre 13y ago
> Google Trends overestimated flu cases by about double for 100 out of 108 weeks Well, there's a simple solution to that... Just halve all of the results from the current algorithm!
deleted 13y ago
[deleted]

nraynaud 13y ago

I would not stone them, they tried, it doesn't work well, they can fix it or scrap it. At least they tried to use their technology for something positive this time. It's one thousand times better to fail when trying to predict the flu epidemics than successfully using technology to bomb weddings.

rm999 13y ago
>It's one thousand times better to fail when trying to predict the flu epidemics than successfully using technology to bomb weddings. That's a pointless and irrelevant comparison. There's a cost to false positives, e.g. creating more vaccinations than necessary; fortunately I don't think the CDC or any other government organizations made any policy decisions based on Google Flu. As a person who works with a lot of data, I think applications like Google Flu have a lot of potential to do a lot of good. But it sounds like Google wasn't properly rigorous in their experiments if Google Flu is such a failure.

gwern 13y ago

Ironically, the original paper (fulltext) fell short of the front page in both submissions: https://news.ycombinator.com/item?id=7405286 https://news.ycombinator.com/item?id=7405286 and https://news.ycombinator.com/item?id=7422496 https://news.ycombinator.com/item?id=7422496 /newest can be so random.

j2kun 13y ago

> When 80-90% of people visiting the doctor for “flu” don’t really have it, you can hardly expect their internet searches to be a reliable source of information. I just attended a talk that graphed the rate of physicians' diagnoses versus the truth, and the conclusion was that about half of physicians drastically overestimate, too (and it's far more than double, closer to five times more!). So when 80-90% of people visiting the doctor for flu don't really have it and those doctors often make type II errors, you can hardly expect their diagnoses to be a reliable source of information.

nraynaud 13y ago
flu is like the communist, everybody is afraid of it, everybody believe they have seen one, but upon serious examination we can't find one. I guess the flu epidemic is just a conspiration created to distract us from the fact that nobody walked on the moon or something. On a more serious note, what's exactly the deal with misdiagnosis? If the real thing is another virus, it could also be a virus that we should treat in a similar manner? Or the doctors are loosing precious time often with a wrong first diagnosis in a significant number of cases?
- Fomite 13y ago
  Diagnosing viral illness is a pain. There are a huge number of winter circulating viruses with roughly the same pattern as the flu, and diagnostic tests for influenza are often very specific but not very sensitive. Meaning if they come up positive, you have it, but if they come up negative...shrug Beyond that, no, it's not likely to be a virus treated in a similar manner. If by manner you mean "drugs". Oseltamivir (aka Tamiflu) only works against influenza, so if you have RSV or some other virus, it won't do you much good. The other way to handle viral diseases is to let them run their course and manage the complications, at which point a strictly accurate diagnosis isn't particularly necessary. TL;DR: It's very hard, fairly expensive, and the payoff per-patient isn't strong.

deleted 13y ago

[deleted]

ffk 13y ago

A few thoughts... Is it possible the CDC estimates are off by half? Two main points come to mind: * How many Type II errors do doctors commit? * How many people with the flu do not visit a doctor? People with good health coverage are probably more likely to visit the doctor preemptively when they are feeling unwell, potentially skewing the results. Also, while google may be off in the exact measurement when compared to the CDC, it looks like the shape of the graph correlates with the CDC data. Overall, this looks approach promising.

shakethemonkey 13y ago
Google isn't trying to predict actual flu cases. They're trying to predict the CDC estimates.
Fomite 13y ago
The problem with "the shape of the graph" correlating is that when modeling a seasonal disease, this is trivially easy. Public health decisions are made on the magnitude of some of those measurements, not their rough shape, and so being off on the exact measurement is a pretty big problem.

mdisraeli 13y ago

This is interesting and a much needed discussion, however it too completely ignores a key difference: CDC tracks hospitalisation based on lab reports, GFT tracks search terms The assumption, which admittedly Google is making, is that these two are correlated by the same amount each year. However, in periods of economic downturn without free health care, people are going to search more and seek actual medial assistance less. Given the global economy has been shot for the last six years....

ChuckMcM 13y ago
> CDC tracks hospitalisation based on lab reports, GFT tracks search terms This is the key difference, and after 5 years it is clear that this is not a valid way to predict the flu, or even monitor it. I think it was a great experiment, and thought it was pretty creative when announced, and now it should either be retired or replaced with a new hypothesis. Because as the Mythbusters would say, this myth is busted.
- mdisraeli 13y ago
  I disagree, but it depends what you mean by "the flu". If you mean hospitalisations, then yes, you're entirely right. If you mean lost productivity and population health outside of hospital, however, perhaps we need a better means to track this other than indirectly via either current stat.
  - Fomite 13y ago
    The CDC also uses a outpatient physician network to report cases of influenza-like illness in the community.
    - mdisraeli 13y ago
      I didn't know that, thank you! I went digging to try and find exactly what figure Google were using, and exactly how CDC came up with it, but I couldn't manage to pin them down...
- 13y ago

simonlebo 13y ago

This article argues that the system overestimated the cases (which it did) and therefore did not work. To reproduce the results of traditional systems is not the purpose of systems like GFT. If you look at the graphs you can see that GFT provides an indication of an increase in flu cases before the traditional systems. This is where it has value; it provides an early indication of an emerging problem.

Fomite 13y ago
To reproduce the results of traditional systems is exactly the purpose of Google Flu trends. This is evident when you go back and read the original paper. The question has always been "Can we match CDC estimates faster and with different information?" The emerging answer is: "No".

blauwbilgorgel 13y ago

Mean absolute error (MAE) during the out-of-sample period is 0.486 for GFT, 0.311 for lagged CDC, and 0.232 for combined GFT and CDC. A combination of GFT search behavior and actual CDC numbers produced the best model for the authors of this paper. The (media) spin is "GFT as a stand-alone flu tracker fails". Perhaps the article would be of a different tone if the researchers had full access to Google's data. I don't think the mentions of irony, failure, misleading and hubris are deserved nor offer a fair portrayal.

scottfr 13y ago

One issue is that the number of historical data points used in the model is really very very small. The author's point out that the model initially was fit to 1,152 CDC data points. However, these 1,152 data points are highly temporally correlated (the value at time t is a strong predictor of the value at time t+1). As such, they aren't "worth" very much when building a model. Therefore, the effectively independent number of points (what you generally need to build a good model) is in reality much smaller. Looking at the graphs, it appears that for each flu season there is a pretty regular spike of flu incidences. This spike could probably be summarized effectively by four parameters: - Mean - Standard Deviation - Skewness - Kurtosis Looking at it this way, the number of "independent" data points of a flu season is not 365 (one for every day the year), 52 (one for every week), or even 12 (one for every month). Instead it is only 4. Thus the total amount of data they had to build their model was 4*(# years). When you have such small data sets, you can't expect to obtain good results. TL;DR. Time series are very hard to use in predictive models. It is often impossible to generate good predictive models based off them.

Fomite 13y ago
The most vexing thing about the research I did on influenza was that my yearly estimates meant despite having a staggering amount of data, my functional N for many experiments was ~ 35, and that was because I reached way into the past.

colin_mccabe 13y ago

This seems like a case of circular reasoning. "Google's results are not as good as the CDC's because Google's results are different than the CDC's." But why do we believe that the CDC estimates are better? I have never gone to the doctor when I had a flu. If it had lasted more than one or two days, or if I had a temperature higher than 102, then I would have gone. But as it was, nobody ever knew that I had the flu except myself and my family. Even if I had wanted to go to the doctor for every time I got a flu or a cold, it takes at least a day or two to schedule a doctor's appointment. By the time the appointment had rolled around I would probably have cancelled, since the flu doesn't usually last very long. This being the US, there are also a lot of uninsured people who can't go to the doctor even if they wanted to. For those people, Google is the only option. The CDC can collect as much data as it wants from the laboratories, but you can't collect what isn't there. When you add in the uninsured people and the people who get better before visiting the doctor, it's not at all surprising that the lab estimates are 2x lower than reality. In fact, I would kind of expect them to be even lower than that.

scottfr 13y ago
The Google model is designed to predict the CDC levels, not predict actual flu prevalence (which is unknown). Error in the Google model is by definition the difference from the CDC levels.
- colin_mccabe 13y ago
  Sure, it's great to compare the Google model with the CDC's. But if there are persistent differences, does it make sense to immediately assume that the Google model is worse at predicting reality than the CDC? Saying that CDC is right "by definition" is not very helpful. [edit: For example, the author of the paper could have compared Google and CDC versus a third data source, like sales of cold medicine. That would have been an interesting comparison, and might have shed some light on which estimates were a better reflection of reality.]
  - Fomite 13y ago
    The CDC's definition is pretty much a gold standard. Things like "sales of cold medicine" and the like fall under the heading of "syndromic surveillance", and believe me, they've been tried for the better part of a decade without yielding much that's impressive. Which is frustrating, because theoretically they should be amazing, but they just aren't, especially when asked to predict rather than back fit.