5 ms·
Statistically Controlling for Confounding Constructs Is Harder Than You Think
- ivan_ah 10y agoI always wondered what exactly people mean when they say “controlling for ...” as if somehow they can tell apart cause A and cause B for the effect C. It turns out if you can measure A and B without noise, the whole "controlling for..." business is legit, but if you only have noisy estimates A' and B' of the actual A and B, then the "controlling for..." technique used by many researchers does not work very well at all. interesting quotes: > The present results demonstrate that, for a very common class of incremental validity arguments, such a strategy runs a high risk of failure. The scope of the problem is considerable: literally hundreds of thousands of studies spanning numerous fields of science have historically relied on measurement-level incremental validity arguments to support strong conclusions about the relationships between theoretical constructs. > We conclude that many previous findings in the social sciences are at high risk of having concluded that two constructs are distinct when they may not in fact be so. Note how polite the authors are. They don't say it straight up that many (most?) multiple regression results are bullshit, they just say "at high risk of being wrong."
- sevenless 10y agoAnother similar story recently was a bug found in analysis software used in many 'brain scan' papers. Then there are a continuous stream of scientific retractions going on (see retractionwatch.com). We do science as follows: data is collected, run through some analytic and/or statistical method, and then the results are published. Then those results get cited by other papers. It's a dependency graph. If some flaw is found in the data or the analysis it would be natural to invalidate this paper and automatically flag all downstream dependencies in the chain of inference. Unfortunately the publishing model does not allow this. I think if we'd developed the internet and software engineering practises before we'd developed scientific publishing, what scientists produce would look less like papers and more like Git repositories. Under this fantasy scenario, we'd have metadata that included statistical methods and software for all analyses. The raw, open data in the repositories could be re-analyzed by anyone else at any time. If there were doubt thrown on a method, or if some data were fraudulent, we could instantly assess the impact on all of current knowledge. I wonder if these recurring problems in curating and re-assessing knowledge will eventually lead to a restructuring of science. Researchers are starting to take the idea of a 'post journal world' seriously (http://blogs.bmj.com/bmj/2016/06/15/richard-smith-what-will-the-post-journal-world-look-like/ http://blogs.bmj.com/bmj/2016/06/15/richard-smith-what-will-...)
- Noseshine 10y agoI think that as long as one remember that a lot of words used in statistics have very different meanings from how one would use them in normal life one should be fine. "Controlling for" is one such term, "explains" is another. Of course, especially once study result make it into mass media all statistical meaning of the words used is completely lost and reinterpreted from the point of view of normal-life usage.
- viraptor 10y agoCould you say how "explains" is different in statistics and normal-life? I know what it means in statistics, but they're actually pretty close, so I'm not sure how it could lead to a non-trivial misunderstanding.
- Noseshine 10y agoStatistical "explains" is just that. It is based purely on the model and numbers. Try this: Create statistical "explanations" for things like Newton's laws, or for everyday explanations like why the uncle didn't go on vacation this year. When you get a statistical "explanation", for example in a linear model, factor X explains 50% of the observed variation, you never know if that is actually the factor - or if it's random, or if it's confounded by an unknown. "explains" in statistics is meant to be used within the context of your model equations and numbers. Even if you get causation by experimenting and controlling the variables and don't just have correlation, who would be satisfied with a statistical "explanation"? You still won't know if there isn't a common factor that you just don't know about yet. Same with statistical significance: It is useless unless someone who actually knows the field checks how/if the model you have relates to the real world. It's not enough to show that everything works fine within your model - you have to know if you have the right model, which requires knowledge outside of statistics. It's just a tool (statistics), by itself it means as much as naked numbers.
- Noseshine 10y ago...or in other words (can't edit what I wrote any more): Statistics can tell you that there is a connection. It can't tell you how (it works).
- adrianN 10y agoUsing faulty statistics only correlates with getting wrong results.
- Houshalter 10y agoThis is really interesting. Trying to summarize the idea of the paper using their example: If people swim more on days when ice cream sales are high, one might conclude that ice cream causes swimming. But if you have information about the temperature, you can fit a statistical model that predicts swimming. The model will learn that temperature predicts swimming perfectly, and the feature of ice cream isn't used at all. So you could say after controlling for temperature, ice cream no longer correlates with swimming. But what if your temperature estimate is noisy or imprecise? For example, if you only have information if the temperature was hotter than 75 degrees, but not exactly what it was. Then this trick no longer works. The temperature variable predicts swimming a lot, but not completely. So ice cream sales are still useful, and so still correlate with swimming after controlling for temperature. I wonder if the way to solve this is using bayes nets. You could test a model where the variables "swimming", "ice cream sales", and "observed temperature", are all caused by some unobserved third variable. This model should fit the data better than alternative models, like ones where "ice cream sales" cause swimming and temperature. But simple linear models can't represent dependencies like this.
- nkurz 10y agoPerhaps you know it, but there's another classic example involving ice cream. It was thought in the 1940's (by the public, by doctors, and by scientists) that ice cream consumption caused children to contract polio. More ice cream is consumed in the summer, there are more polio cases in the summer: https://theglyptodon.wordpress.com/2012/08/21/polio-caused-by-ice-cream/ https://theglyptodon.wordpress.com/2012/08/21/polio-caused-b...
- Noseshine 10y agoMy personal favorite: http://tylervigen.com/view_correlation?id=1597 http://tylervigen.com/view_correlation?id=1597 US spending on science, space, and technology correlates with Suicides by hanging, strangulation and suffocation
- blahi 10y agoThis is called mediation. Only if newly hatched data science machine learning hackers read statistics textbooks instead of coursera moocs...