5 ms·

Why would you blame the study authors for doing a correlation study? I know "CORRELATION IS NOT CAUSATION!!!" is the go-to takedown on the internet, but correla

by slurry 13y ago

Why would you blame the study authors for doing a correlation study? I know "CORRELATION IS NOT CAUSATION!!!" is the go-to takedown on the internet, but correlation studies actually can have significant value - otherwise peer-reviewed journals wouldn't publish them. I'm also skeptical of Politifact's competence to adjudicate public health scholarship.

aaronbrethorst 13y ago
I'm also skeptical of Politifact's competence. FTFY
aamar 13y ago
I agree that correlation studies can be valuable, but I see many correlation studies as (intentionally or not) exploiting a propensity (arguably a bug) in human reasoning that conflates it with causation. In this situation, we have thorough documentation of multiple people clearly making the error. I'm actually not sure the study authors should be blamed. But: given how politicized this question is, they could have reasonably anticipated the misuse of their results, and thus could have written their results in such a way as to avoid this. Or they could have publicly corrected non-experts who cited them for causation. Or: they could have controlled for factors that would make mortality and insurance-status independent. This last option is difficult & requires complex judgement calls (see [1] for a reasonable attempt), but even if you feel the other two aren't required of academics, this last one very much may be. (Separately, I agree that Politifact should not be trusted automatically, but these seem like reasonable analyses, and I did not quickly find anything better.) [1] http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2739025/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2739025/
Fomite 13y ago
Not only do they have significant value, but they're the only type of evidence you'll ever have for something like this - it's impossible, and almost certainly unethical - to run a randomized trial of keeping people from having health insurance. Also, while "Correlation is not causation" is, as you mentioned, a tired canard on the internet, people often seem to forget that all things that do have a causal relationship with have some form of association. Association doesn't prove causation, but it's a damned fine first step, and miles above "guessing", which is what happens without evidence.
- ekianjo 13y ago
  Sorry, but no. 100 times no. Correlation is very often linked to a third factor or multiple factors which are not visible, nor measured in observational studies. Besides, let's not disregard the fact that correlation still has some good chance to be pure luck. Even correlation with 95% confidence statistical significance can be a random result in a non-nil number of times. So, no, you never prove anything nor imply anything at all with correlation. You're still guessing.
  - Fomite 13y ago
    Find me where I said proof. There are all kinds of things that we cannot prove, because it is either impossible or wildly unethical to conduct a randomized study. For those things, you can make a determined effort to control for as many "third factors" - the technical term for them is confounders - and that gives you a level of evidence which is well above guessing. Since I can't reply to your comment, my responses here: > "You didn't say proof but you said it's better than guessing, and I don't agree with you at all." It is better than guessing. You're welcome to disagree, but a well conducted observational study is considerably firmer evidence than pulling it from your posterior. > "What if there is a correlation between Vegetarian-lifestyle and Serial-killers ? Does it tell you that it's better than guessing ? Do you even question if the association/correlation makes remote sense ? Is there any underlying mechanism of action that would remotely explain rationally why this correlation could be linked to any real causation phenomenon ?" All you've done is describe a really bad study. You can have really bad RCTs as well, by the way. Of course you question whether or not an observed association has a clear biological or social mechanism. And you attempt to control for other variables that might influence the link between your exposure and your outcome. You run followup studies in different populations to try to understand if the result is a widespread phenomena, or a fleeting bit of statistical noise. Basically, you do your job well. Which is why I used phrases like "a good first step". Your example is about as useful as "Programming is useless because once I coded something poorly and corrupted my data".
    - ekianjo 13y ago
      You didn't say proof but you said it's better than guessing, and I don't agree with you at all. What if there is a correlation between Vegetarian-lifestyle and Serial-killers ? Does it tell you that it's better than guessing ? Do you even question if the association/correlation makes remote sense ? Is there any underlying mechanism of action that would remotely explain rationally why this correlation could be linked to any real causation phenomenon ? Correlation is useless, and there's a ton of observational studies out there finding correlations every single day for which we have no rational explanation at all. Observational studies are full of variations in the way they are designed, the way they are reported and the subjects of the studies, it's rather a miracle if you actually detect a hint of causation based on the garbage noise that you get.
mattfenwick 13y ago
> correlation studies actually can have significant value - otherwise peer-reviewed journals wouldn't publish them This is an appeal to authority. And unfortunately, the evidence is accumulating that there are problems with that authority. The success of peer-review depends on the quality of the peers and of the review. If those reviewing don't understand how to evaluate a correlation study, or do understand but don't take the time to properly evaluate it, then garbage will slip through. It turns out, lots of garbage is produced and sent to the reviewers, as was noted in a recent Nature feature: http://www.nature.com/news/scientific-method-statistical-errors-1.14700 http://www.nature.com/news/scientific-method-statistical-err... see also: http://www.nature.com/news/weak-statistical-standards-implicated-in-scientific-irreproducibility-1.14131 http://www.nature.com/news/weak-statistical-standards-implic... http://www.nature.com/nature/focus/reproducibility/index.html http://www.nature.com/nature/focus/reproducibility/index.htm... --------- tl,dr; lots of crappy correlation studies are published in peer-reviewed journals. These studies later turn out to be irreproducible.

by slurry 13y ago

aaronbrethorst 13y ago

I'm also skeptical of Politifact's competence. FTFY

aamar 13y ago

I agree that correlation studies can be valuable, but I see many correlation studies as (intentionally or not) exploiting a propensity (arguably a bug) in human reasoning that conflates it with causation. In this situation, we have thorough documentation of multiple people clearly making the error. I'm actually not sure the study authors should be blamed. But: given how politicized this question is, they could have reasonably anticipated the misuse of their results, and thus could have written their results in such a way as to avoid this. Or they could have publicly corrected non-experts who cited them for causation. Or: they could have controlled for factors that would make mortality and insurance-status independent. This last option is difficult & requires complex judgement calls (see [1] for a reasonable attempt), but even if you feel the other two aren't required of academics, this last one very much may be. (Separately, I agree that Politifact should not be trusted automatically, but these seem like reasonable analyses, and I did not quickly find anything better.) [1] http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2739025/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2739025/

Fomite 13y ago

Not only do they have significant value, but they're the only type of evidence you'll ever have for something like this - it's impossible, and almost certainly unethical - to run a randomized trial of keeping people from having health insurance. Also, while "Correlation is not causation" is, as you mentioned, a tired canard on the internet, people often seem to forget that all things that do have a causal relationship with have some form of association. Association doesn't prove causation, but it's a damned fine first step, and miles above "guessing", which is what happens without evidence.

ekianjo 13y ago
Sorry, but no. 100 times no. Correlation is very often linked to a third factor or multiple factors which are not visible, nor measured in observational studies. Besides, let's not disregard the fact that correlation still has some good chance to be pure luck. Even correlation with 95% confidence statistical significance can be a random result in a non-nil number of times. So, no, you never prove anything nor imply anything at all with correlation. You're still guessing.
- Fomite 13y ago
  Find me where I said proof. There are all kinds of things that we cannot prove, because it is either impossible or wildly unethical to conduct a randomized study. For those things, you can make a determined effort to control for as many "third factors" - the technical term for them is confounders - and that gives you a level of evidence which is well above guessing. Since I can't reply to your comment, my responses here: > "You didn't say proof but you said it's better than guessing, and I don't agree with you at all." It is better than guessing. You're welcome to disagree, but a well conducted observational study is considerably firmer evidence than pulling it from your posterior. > "What if there is a correlation between Vegetarian-lifestyle and Serial-killers ? Does it tell you that it's better than guessing ? Do you even question if the association/correlation makes remote sense ? Is there any underlying mechanism of action that would remotely explain rationally why this correlation could be linked to any real causation phenomenon ?" All you've done is describe a really bad study. You can have really bad RCTs as well, by the way. Of course you question whether or not an observed association has a clear biological or social mechanism. And you attempt to control for other variables that might influence the link between your exposure and your outcome. You run followup studies in different populations to try to understand if the result is a widespread phenomena, or a fleeting bit of statistical noise. Basically, you do your job well. Which is why I used phrases like "a good first step". Your example is about as useful as "Programming is useless because once I coded something poorly and corrupted my data".
  - ekianjo 13y ago
    You didn't say proof but you said it's better than guessing, and I don't agree with you at all. What if there is a correlation between Vegetarian-lifestyle and Serial-killers ? Does it tell you that it's better than guessing ? Do you even question if the association/correlation makes remote sense ? Is there any underlying mechanism of action that would remotely explain rationally why this correlation could be linked to any real causation phenomenon ? Correlation is useless, and there's a ton of observational studies out there finding correlations every single day for which we have no rational explanation at all. Observational studies are full of variations in the way they are designed, the way they are reported and the subjects of the studies, it's rather a miracle if you actually detect a hint of causation based on the garbage noise that you get.

mattfenwick 13y ago

> correlation studies actually can have significant value - otherwise peer-reviewed journals wouldn't publish them This is an appeal to authority. And unfortunately, the evidence is accumulating that there are problems with that authority. The success of peer-review depends on the quality of the peers and of the review. If those reviewing don't understand how to evaluate a correlation study, or do understand but don't take the time to properly evaluate it, then garbage will slip through. It turns out, lots of garbage is produced and sent to the reviewers, as was noted in a recent Nature feature: http://www.nature.com/news/scientific-method-statistical-errors-1.14700 http://www.nature.com/news/scientific-method-statistical-err... see also: http://www.nature.com/news/weak-statistical-standards-implicated-in-scientific-irreproducibility-1.14131 http://www.nature.com/news/weak-statistical-standards-implic... http://www.nature.com/nature/focus/reproducibility/index.html http://www.nature.com/nature/focus/reproducibility/index.htm... --------- tl,dr; lots of crappy correlation studies are published in peer-reviewed journals. These studies later turn out to be irreproducible.