5 ms·

The Google model is designed to predict the CDC levels, not predict actual flu prevalence (which is unknown). Error in the Google model is by definition the di

by scottfr 13y ago

The Google model is designed to predict the CDC levels, not predict actual flu prevalence (which is unknown). Error in the Google model is by definition the difference from the CDC levels.

colin_mccabe 13y ago
Sure, it's great to compare the Google model with the CDC's. But if there are persistent differences, does it make sense to immediately assume that the Google model is worse at predicting reality than the CDC? Saying that CDC is right "by definition" is not very helpful. [edit: For example, the author of the paper could have compared Google and CDC versus a third data source, like sales of cold medicine. That would have been an interesting comparison, and might have shed some light on which estimates were a better reflection of reality.]
- Fomite 13y ago
  The CDC's definition is pretty much a gold standard. Things like "sales of cold medicine" and the like fall under the heading of "syndromic surveillance", and believe me, they've been tried for the better part of a decade without yielding much that's impressive. Which is frustrating, because theoretically they should be amazing, but they just aren't, especially when asked to predict rather than back fit.
  - colin_mccabe 13y ago
    I guess I'm missing some context here. Why don't sales of cold medicine impress you? Why are the CDC's numbers a gold standard?
    - Fomite 13y ago
      Things like the sales of cold medicine have been used before, and generally speaking they're pretty poor at detecting anything. They can occasionally work well if you know something happened already, and are looking to see when it started, but as any sort of surveillance system, they haven't panned out despite some very intensive research. The CDC numbers (which are part of ILINet) are pretty decent, and one of the things they do, which hasn't been mentioned here, is they're not trying to find all cases of influenza. They have a network of physicians in all 50 states reporting "Influenza-like Illness". These people then also send samples in for laboratory confirmation of virus type, drug resistance, etc. The numbers are also backwards revised as more data comes in, meaning they improve over the course of the year. It's a very good system for getting an idea of how "bad" the flu season is - the main problem is that it's slow. The ideal, and what people were hoping from Google Flu Trends (and other systems like it, it's not alone) is "ILINet, but in real time".
Fomite 13y ago
This. A failure to match the CDC's confirmed cases is a failure of the Google model, because that's what it's trying to match. It also fails fairly spectacularly on a local level when compared to health department data from some top flight state agencies, like NY.
- colin_mccabe 13y ago
  The CDC's model "fails pretty spectacularly" when it comes to capturing any of the times I've had the flu. I have had the flu several times and the CDC has never been aware of it. As I explained earlier, I never told my doctor that I had the flu or went to a lab. My local health department was never aware of these cases either. It might, however, have appeared on Google flu trends, since since I might have searched for what the impact of a certain level of fever was. It's been a few years so I don't remember if I did or not. And of course, Google has never published which search keywords they use. There are three datasets here: the CDC's, Google's, and reality. Some folks here seem unable to differentiate between the CDC data and reality. But in fact, I really did have the flu, even though CDC didn't think I did. Reality is what matters. Maybe there is reason to believe that CDC's data is closer to reality than Google's. If so, let's hear it.
  - Fomite 13y ago
    The key is, like all population studies, the CDC doesn't have to know about you. It has to know about some people like you, and honestly, if you were well enough to not need to go to the doctor, it's also not really the influenza public health authorities are worried about. The CDC is unapologetic about their data being estimates - but they're very solid estimates, and match more intensive but smaller scale studies pretty well. But we don't need to capture every single flu case - no surveillance system will ever do that, nor need to.
    - colin_mccabe 13y ago
      I just feel like either I am missing something, or there is a lack of rigor here. Clearly, sampling introduces errors and under-reporting introduces errors. What I am looking for is a case (based on data, not hand-waving) that Google's errors are worse than the CDC's, and so far I'm just not seeing it. Maybe I am missing something which is obvious to people in the field. Edit: I see how the CDC's numbers could be a lower bound, but not upper.

by scottfr 13y ago

The Google model is designed to predict the CDC levels, not predict actual flu prevalence (which is unknown). Error in the Google model is by definition the difference from the CDC levels.

colin_mccabe 13y ago

Sure, it's great to compare the Google model with the CDC's. But if there are persistent differences, does it make sense to immediately assume that the Google model is worse at predicting reality than the CDC? Saying that CDC is right "by definition" is not very helpful. [edit: For example, the author of the paper could have compared Google and CDC versus a third data source, like sales of cold medicine. That would have been an interesting comparison, and might have shed some light on which estimates were a better reflection of reality.]

Fomite 13y ago
The CDC's definition is pretty much a gold standard. Things like "sales of cold medicine" and the like fall under the heading of "syndromic surveillance", and believe me, they've been tried for the better part of a decade without yielding much that's impressive. Which is frustrating, because theoretically they should be amazing, but they just aren't, especially when asked to predict rather than back fit.
- colin_mccabe 13y ago
  I guess I'm missing some context here. Why don't sales of cold medicine impress you? Why are the CDC's numbers a gold standard?
  - Fomite 13y ago
    Things like the sales of cold medicine have been used before, and generally speaking they're pretty poor at detecting anything. They can occasionally work well if you know something happened already, and are looking to see when it started, but as any sort of surveillance system, they haven't panned out despite some very intensive research. The CDC numbers (which are part of ILINet) are pretty decent, and one of the things they do, which hasn't been mentioned here, is they're not trying to find all cases of influenza. They have a network of physicians in all 50 states reporting "Influenza-like Illness". These people then also send samples in for laboratory confirmation of virus type, drug resistance, etc. The numbers are also backwards revised as more data comes in, meaning they improve over the course of the year. It's a very good system for getting an idea of how "bad" the flu season is - the main problem is that it's slow. The ideal, and what people were hoping from Google Flu Trends (and other systems like it, it's not alone) is "ILINet, but in real time".

Fomite 13y ago

This. A failure to match the CDC's confirmed cases is a failure of the Google model, because that's what it's trying to match. It also fails fairly spectacularly on a local level when compared to health department data from some top flight state agencies, like NY.

colin_mccabe 13y ago
The CDC's model "fails pretty spectacularly" when it comes to capturing any of the times I've had the flu. I have had the flu several times and the CDC has never been aware of it. As I explained earlier, I never told my doctor that I had the flu or went to a lab. My local health department was never aware of these cases either. It might, however, have appeared on Google flu trends, since since I might have searched for what the impact of a certain level of fever was. It's been a few years so I don't remember if I did or not. And of course, Google has never published which search keywords they use. There are three datasets here: the CDC's, Google's, and reality. Some folks here seem unable to differentiate between the CDC data and reality. But in fact, I really did have the flu, even though CDC didn't think I did. Reality is what matters. Maybe there is reason to believe that CDC's data is closer to reality than Google's. If so, let's hear it.
- Fomite 13y ago
  The key is, like all population studies, the CDC doesn't have to know about you. It has to know about some people like you, and honestly, if you were well enough to not need to go to the doctor, it's also not really the influenza public health authorities are worried about. The CDC is unapologetic about their data being estimates - but they're very solid estimates, and match more intensive but smaller scale studies pretty well. But we don't need to capture every single flu case - no surveillance system will ever do that, nor need to.
  - colin_mccabe 13y ago
    I just feel like either I am missing something, or there is a lack of rigor here. Clearly, sampling introduces errors and under-reporting introduces errors. What I am looking for is a case (based on data, not hand-waving) that Google's errors are worse than the CDC's, and so far I'm just not seeing it. Maybe I am missing something which is obvious to people in the field. Edit: I see how the CDC's numbers could be a lower bound, but not upper.