4 ms·

The Omitted Variable Problem is part of my mental framework that causes me to not believe most epidemiological studies, especially ones that confirm a popular b

by tlb 13y ago

The Omitted Variable Problem is part of my mental framework that causes me to not believe most epidemiological studies, especially ones that confirm a popular belief. The almost universally omitted variable is health-consciousness. Some people are health-conscious and some aren't. People who are health-conscious do a whole bunch of things, some of which help (like exercise, sleep well, eat moderately). They also do things that are widely believed to be good for you, like eating broccoli. So if you do a study, you'll find that people who eat lots of broccoli are healthier. You'll be able to confirm pretty much any widely believed health folk wisdom, unless it's something quite harmful, as long as you omit health-consciousness as a variable.

maxerickson 13y ago
If it was a huge deal, wouldn't epidemiologists be slobbering over the statistical power they would get from using it? That is, if they can control for it, they can make assertions about smaller effects.
- gwern 13y ago
  > If it was a huge deal, wouldn't epidemiologists be slobbering over the statistical power they would get from using it? Why would you want to do that? You can build a nice career on publishing (spurious) associations. As long as relevant variables are omitted, no epidemiologist need ever be unemployed.
  - maxerickson 13y ago
    I'm optimistic enough to believe in a few wild cards trying to do good science. A whole bunch of the rest have to put a little bit of effort into competing with them. It occurred to me a while after my other comment that a belief in the ability of people that test as health conscious to actually do things that improve their health might not be especially justified. I mean, lots of people are quite happy with whatever quackery, maybe it really does all come out in the wash.
  - maxander 13y ago
    That may get you written about in popular-science magazines, but your work will have a low impact factor- this is largely measured in how many of your fellow researchers cite your work, and its crucial to an academic researcher's career. If your colleagues can easily see that your statistical analysis is flawed, they won't waste their own time doing work that builds off of yours.
    - wnewman 13y ago
      It is not necessarily true that just because they are called scientists, real people with real motives, funded by organizations of real people with real motives, operating in a political regime controlled by real people with real motives, will act that way. (By "that way" I mean "your work will have a low impact factor- this is largely measured in how many of your fellow researchers cite your work" and "If your colleagues can easily see that your statistical analysis is flawed, they won't waste their own time doing work that builds off of yours.") See e.g. Chan and Boliver "The Grandparents Effect in Social Mobility", http://asr.sagepub.com/content/78/4/662 http://asr.sagepub.com/content/78/4/662 (HT http://www.arnoldkling.com/blog/a-grandparent-effect/ http://www.arnoldkling.com/blog/a-grandparent-effect/) and many of the works that it cites. There seems to be a sizable mutually-citing "scientific" literature devoted to studying correlation in biological descent while dogmatically not controlling for ordinary DNA-based inheritance of psychological traits from parent to child. (Indeed, not even controlling for DNA-based inheritance of physical traits. Health and height and attractiveness are correlated with income, and are significantly physically heritable.) By not controlling for the properties which are transmitted by sperm meeting egg, your can find all sorts of fascinating possibilities to investigate to explain the correlation. (E.g. Chan and Boliver carefully note that "well-connected grandparents could also use their social contacts to help grandchildren with job searches." Good, good, carefully investigate all the possibilities.)
    - gwern 13y ago
      > but your work will have a low impact factor- this is largely measured in how many of your fellow researchers cite your work, and its crucial to an academic researcher's career. Nope! This is how we might like to to work, but cool associations will get written up forever, long after they have been refuted by randomized experiments. (Heck, papers can outright be retracted and still get cites.) Hahaha - did I say refuted by randomized experiments? Like that ever happens to more than a tiny fraction of epidemiology correlations... No, one's career is quite safe. No one ever lost tenure because the correlations they built their career on turned out to be lame. > If your colleagues can easily see that your statistical analysis is flawed, they won't waste their own time doing work that builds off of yours. No, the point here is that your work can be immaculate and still never reflect causality. How are you going to collect data on every lurking variable? You can't, of course.
- marcosdumay 13y ago
  Controling for hidden variables (the general case of this paradox) is a big deal for all science areas.
- chrisbennet 13y ago
  In his article "Science pseudoscience nutritional epidemiology and meat" http://garytaubes.com/2012/03/science-pseudoscience-nutritional-epidemiology-and-meat/ http://garytaubes.com/2012/03/science-pseudoscience-nutritio... Gary Taubes discusses how epidemiological studies can fail to account for something called the "compliance effect" - the missing variable. It was an eye opener for me.
  - maxerickson 13y ago
    He quotes a textbook chapter in his conclusion. So the idea that you have to at least be careful with your statistics is orthodoxy, not some missing component of the field. (I realize that doesn't make any guarantees about common practices, but it is maybe a little capricious to repudiate the entire field)
rthomas6 13y ago
Surely some studies account for this? If you look for people who do X and people who don't and just analyze their lives, yes this problem is likely to exist. But if you take two randomized samples of a the population and say to group A, "do X," and to group B "don't do X," you have an effective control group. At least I think so. Don't some dietary studies even provide the participants with custom food regimens to try to eliminate extra dietary differences between study participants?
- defen 13y ago
  You're describing a randomized controlled trial and not an epidemiological study.
  - dbecker 13y ago
    You are mistakenly confounding that epidemiological and observational. Epidemiology studies can be randomized or observational.
    - defen 13y ago
      Interesting...what would be an example of a randomized epidemiological study?
- spaux 13y ago
  What you're describing is the "as long as you omit health-consciousness as a variable" part of his comment.
- abraxasz 13y ago
  That's one way to do it (the best way). You can still draw "causal" conclusion on observational studies (where you don't get to explicitly assign people to group A and B, but where those assignment are given to you), but it is much harder and you have to be very careful with what you are doing. If you are interested in that topic I suggest you google "Rubin Causal Framework", or check out some of the papers by Donald Rubin (arguably the pioneer of the field of Causal Inference).
- 13y ago
jasonwatkinspdx 13y ago
Don't throw the baby out with the bathwater. There are ways of calculating whether a hidden variable is more likely to cause the observational data than the independent variables of your model. A simple example would be a wet lawn. We know rain causes a wet lawn, and our observation shows indeed that rain and wet lawns are strongly associated. However, observing a case where a given lawn is wet, and yet there's no associated rain is a clear signal that a latent cause hasn't been accounted for (namely sprinklers). This principle still applies in noisy observational data or probabilistic rather than deterministic causal relations, though you do need a bigger sample to reach the same confidence. We also can look at measures of model fitness. If a variation of the model that hypothesizes a latent causal variable is more likely to generate the observed data than the model without it, we know we've missed something. The general case of this is that we learn the model itself from the observed data. This page from Kevin Murphy is a reasonable survey of the methods: http://www.cs.ubc.ca/~murphyk/Bayes/bayes.html http://www.cs.ubc.ca/~murphyk/Bayes/bayes.html I also recommend his textbook.
B-Con 13y ago
> What Simpson's paradox is not > Every ommitted variable problem Not quite the same thing as the OP (not sure if you were implying it was). Regardless, OVP is still a great thing to have in your mental framework.

by tlb 13y ago

maxerickson 13y ago

If it was a huge deal, wouldn't epidemiologists be slobbering over the statistical power they would get from using it? That is, if they can control for it, they can make assertions about smaller effects.

gwern 13y ago
> If it was a huge deal, wouldn't epidemiologists be slobbering over the statistical power they would get from using it? Why would you want to do that? You can build a nice career on publishing (spurious) associations. As long as relevant variables are omitted, no epidemiologist need ever be unemployed.
- maxerickson 13y ago
  I'm optimistic enough to believe in a few wild cards trying to do good science. A whole bunch of the rest have to put a little bit of effort into competing with them. It occurred to me a while after my other comment that a belief in the ability of people that test as health conscious to actually do things that improve their health might not be especially justified. I mean, lots of people are quite happy with whatever quackery, maybe it really does all come out in the wash.
- maxander 13y ago
  That may get you written about in popular-science magazines, but your work will have a low impact factor- this is largely measured in how many of your fellow researchers cite your work, and its crucial to an academic researcher's career. If your colleagues can easily see that your statistical analysis is flawed, they won't waste their own time doing work that builds off of yours.
  - wnewman 13y ago
    It is not necessarily true that just because they are called scientists, real people with real motives, funded by organizations of real people with real motives, operating in a political regime controlled by real people with real motives, will act that way. (By "that way" I mean "your work will have a low impact factor- this is largely measured in how many of your fellow researchers cite your work" and "If your colleagues can easily see that your statistical analysis is flawed, they won't waste their own time doing work that builds off of yours.") See e.g. Chan and Boliver "The Grandparents Effect in Social Mobility", http://asr.sagepub.com/content/78/4/662 http://asr.sagepub.com/content/78/4/662 (HT http://www.arnoldkling.com/blog/a-grandparent-effect/ http://www.arnoldkling.com/blog/a-grandparent-effect/) and many of the works that it cites. There seems to be a sizable mutually-citing "scientific" literature devoted to studying correlation in biological descent while dogmatically not controlling for ordinary DNA-based inheritance of psychological traits from parent to child. (Indeed, not even controlling for DNA-based inheritance of physical traits. Health and height and attractiveness are correlated with income, and are significantly physically heritable.) By not controlling for the properties which are transmitted by sperm meeting egg, your can find all sorts of fascinating possibilities to investigate to explain the correlation. (E.g. Chan and Boliver carefully note that "well-connected grandparents could also use their social contacts to help grandchildren with job searches." Good, good, carefully investigate all the possibilities.)
  - gwern 13y ago
    > but your work will have a low impact factor- this is largely measured in how many of your fellow researchers cite your work, and its crucial to an academic researcher's career. Nope! This is how we might like to to work, but cool associations will get written up forever, long after they have been refuted by randomized experiments. (Heck, papers can outright be retracted and still get cites.) Hahaha - did I say refuted by randomized experiments? Like that ever happens to more than a tiny fraction of epidemiology correlations... No, one's career is quite safe. No one ever lost tenure because the correlations they built their career on turned out to be lame. > If your colleagues can easily see that your statistical analysis is flawed, they won't waste their own time doing work that builds off of yours. No, the point here is that your work can be immaculate and still never reflect causality. How are you going to collect data on every lurking variable? You can't, of course.
marcosdumay 13y ago
Controling for hidden variables (the general case of this paradox) is a big deal for all science areas.
chrisbennet 13y ago
In his article "Science pseudoscience nutritional epidemiology and meat" http://garytaubes.com/2012/03/science-pseudoscience-nutritional-epidemiology-and-meat/ http://garytaubes.com/2012/03/science-pseudoscience-nutritio... Gary Taubes discusses how epidemiological studies can fail to account for something called the "compliance effect" - the missing variable. It was an eye opener for me.
- maxerickson 13y ago
  He quotes a textbook chapter in his conclusion. So the idea that you have to at least be careful with your statistics is orthodoxy, not some missing component of the field. (I realize that doesn't make any guarantees about common practices, but it is maybe a little capricious to repudiate the entire field)

rthomas6 13y ago

Surely some studies account for this? If you look for people who do X and people who don't and just analyze their lives, yes this problem is likely to exist. But if you take two randomized samples of a the population and say to group A, "do X," and to group B "don't do X," you have an effective control group. At least I think so. Don't some dietary studies even provide the participants with custom food regimens to try to eliminate extra dietary differences between study participants?

defen 13y ago
You're describing a randomized controlled trial and not an epidemiological study.
- dbecker 13y ago
  You are mistakenly confounding that epidemiological and observational. Epidemiology studies can be randomized or observational.
  - defen 13y ago
    Interesting...what would be an example of a randomized epidemiological study?
spaux 13y ago
What you're describing is the "as long as you omit health-consciousness as a variable" part of his comment.
abraxasz 13y ago
That's one way to do it (the best way). You can still draw "causal" conclusion on observational studies (where you don't get to explicitly assign people to group A and B, but where those assignment are given to you), but it is much harder and you have to be very careful with what you are doing. If you are interested in that topic I suggest you google "Rubin Causal Framework", or check out some of the papers by Donald Rubin (arguably the pioneer of the field of Causal Inference).
13y ago

jasonwatkinspdx 13y ago

Don't throw the baby out with the bathwater. There are ways of calculating whether a hidden variable is more likely to cause the observational data than the independent variables of your model. A simple example would be a wet lawn. We know rain causes a wet lawn, and our observation shows indeed that rain and wet lawns are strongly associated. However, observing a case where a given lawn is wet, and yet there's no associated rain is a clear signal that a latent cause hasn't been accounted for (namely sprinklers). This principle still applies in noisy observational data or probabilistic rather than deterministic causal relations, though you do need a bigger sample to reach the same confidence. We also can look at measures of model fitness. If a variation of the model that hypothesizes a latent causal variable is more likely to generate the observed data than the model without it, we know we've missed something. The general case of this is that we learn the model itself from the observed data. This page from Kevin Murphy is a reasonable survey of the methods: http://www.cs.ubc.ca/~murphyk/Bayes/bayes.html http://www.cs.ubc.ca/~murphyk/Bayes/bayes.html I also recommend his textbook.

B-Con 13y ago

> What Simpson's paradox is not > Every ommitted variable problem Not quite the same thing as the OP (not sure if you were implying it was). Regardless, OVP is still a great thing to have in your mental framework.