4 ms·

I always wondered what exactly people mean when they say “controlling for ...” as if somehow they can tell apart cause A and cause B for the effect C. It turns

by ivan_ah 10y ago

I always wondered what exactly people mean when they say “controlling for ...” as if somehow they can tell apart cause A and cause B for the effect C. It turns out if you can measure A and B without noise, the whole "controlling for..." business is legit, but if you only have noisy estimates A' and B' of the actual A and B, then the "controlling for..." technique used by many researchers does not work very well at all. interesting quotes: > The present results demonstrate that, for a very common class of incremental validity arguments, such a strategy runs a high risk of failure. The scope of the problem is considerable: literally hundreds of thousands of studies spanning numerous fields of science have historically relied on measurement-level incremental validity arguments to support strong conclusions about the relationships between theoretical constructs. > We conclude that many previous findings in the social sciences are at high risk of having concluded that two constructs are distinct when they may not in fact be so. Note how polite the authors are. They don't say it straight up that many (most?) multiple regression results are bullshit, they just say "at high risk of being wrong."

sevenless 10y ago
Another similar story recently was a bug found in analysis software used in many 'brain scan' papers. Then there are a continuous stream of scientific retractions going on (see retractionwatch.com). We do science as follows: data is collected, run through some analytic and/or statistical method, and then the results are published. Then those results get cited by other papers. It's a dependency graph. If some flaw is found in the data or the analysis it would be natural to invalidate this paper and automatically flag all downstream dependencies in the chain of inference. Unfortunately the publishing model does not allow this. I think if we'd developed the internet and software engineering practises before we'd developed scientific publishing, what scientists produce would look less like papers and more like Git repositories. Under this fantasy scenario, we'd have metadata that included statistical methods and software for all analyses. The raw, open data in the repositories could be re-analyzed by anyone else at any time. If there were doubt thrown on a method, or if some data were fraudulent, we could instantly assess the impact on all of current knowledge. I wonder if these recurring problems in curating and re-assessing knowledge will eventually lead to a restructuring of science. Researchers are starting to take the idea of a 'post journal world' seriously (http://blogs.bmj.com/bmj/2016/06/15/richard-smith-what-will-the-post-journal-world-look-like/ http://blogs.bmj.com/bmj/2016/06/15/richard-smith-what-will-...)
Noseshine 10y ago
I think that as long as one remember that a lot of words used in statistics have very different meanings from how one would use them in normal life one should be fine. "Controlling for" is one such term, "explains" is another. Of course, especially once study result make it into mass media all statistical meaning of the words used is completely lost and reinterpreted from the point of view of normal-life usage.
- viraptor 10y ago
  Could you say how "explains" is different in statistics and normal-life? I know what it means in statistics, but they're actually pretty close, so I'm not sure how it could lead to a non-trivial misunderstanding.
  - Noseshine 10y ago
    Statistical "explains" is just that. It is based purely on the model and numbers. Try this: Create statistical "explanations" for things like Newton's laws, or for everyday explanations like why the uncle didn't go on vacation this year. When you get a statistical "explanation", for example in a linear model, factor X explains 50% of the observed variation, you never know if that is actually the factor - or if it's random, or if it's confounded by an unknown. "explains" in statistics is meant to be used within the context of your model equations and numbers. Even if you get causation by experimenting and controlling the variables and don't just have correlation, who would be satisfied with a statistical "explanation"? You still won't know if there isn't a common factor that you just don't know about yet. Same with statistical significance: It is useless unless someone who actually knows the field checks how/if the model you have relates to the real world. It's not enough to show that everything works fine within your model - you have to know if you have the right model, which requires knowledge outside of statistics. It's just a tool (statistics), by itself it means as much as naked numbers.
    - Noseshine 10y ago
      ...or in other words (can't edit what I wrote any more): Statistics can tell you that there is a connection. It can't tell you how (it works).
adrianN 10y ago
Using faulty statistics only correlates with getting wrong results.

by ivan_ah 10y ago

sevenless 10y ago

Another similar story recently was a bug found in analysis software used in many 'brain scan' papers. Then there are a continuous stream of scientific retractions going on (see retractionwatch.com). We do science as follows: data is collected, run through some analytic and/or statistical method, and then the results are published. Then those results get cited by other papers. It's a dependency graph. If some flaw is found in the data or the analysis it would be natural to invalidate this paper and automatically flag all downstream dependencies in the chain of inference. Unfortunately the publishing model does not allow this. I think if we'd developed the internet and software engineering practises before we'd developed scientific publishing, what scientists produce would look less like papers and more like Git repositories. Under this fantasy scenario, we'd have metadata that included statistical methods and software for all analyses. The raw, open data in the repositories could be re-analyzed by anyone else at any time. If there were doubt thrown on a method, or if some data were fraudulent, we could instantly assess the impact on all of current knowledge. I wonder if these recurring problems in curating and re-assessing knowledge will eventually lead to a restructuring of science. Researchers are starting to take the idea of a 'post journal world' seriously (http://blogs.bmj.com/bmj/2016/06/15/richard-smith-what-will-the-post-journal-world-look-like/ http://blogs.bmj.com/bmj/2016/06/15/richard-smith-what-will-...)

Noseshine 10y ago

I think that as long as one remember that a lot of words used in statistics have very different meanings from how one would use them in normal life one should be fine. "Controlling for" is one such term, "explains" is another. Of course, especially once study result make it into mass media all statistical meaning of the words used is completely lost and reinterpreted from the point of view of normal-life usage.

viraptor 10y ago
Could you say how "explains" is different in statistics and normal-life? I know what it means in statistics, but they're actually pretty close, so I'm not sure how it could lead to a non-trivial misunderstanding.
- Noseshine 10y ago
  Statistical "explains" is just that. It is based purely on the model and numbers. Try this: Create statistical "explanations" for things like Newton's laws, or for everyday explanations like why the uncle didn't go on vacation this year. When you get a statistical "explanation", for example in a linear model, factor X explains 50% of the observed variation, you never know if that is actually the factor - or if it's random, or if it's confounded by an unknown. "explains" in statistics is meant to be used within the context of your model equations and numbers. Even if you get causation by experimenting and controlling the variables and don't just have correlation, who would be satisfied with a statistical "explanation"? You still won't know if there isn't a common factor that you just don't know about yet. Same with statistical significance: It is useless unless someone who actually knows the field checks how/if the model you have relates to the real world. It's not enough to show that everything works fine within your model - you have to know if you have the right model, which requires knowledge outside of statistics. It's just a tool (statistics), by itself it means as much as naked numbers.
  - Noseshine 10y ago
    ...or in other words (can't edit what I wrote any more): Statistics can tell you that there is a connection. It can't tell you how (it works).

adrianN 10y ago

Using faulty statistics only correlates with getting wrong results.