11 ms·

The scientific study has become a flawed manual for living

41 points by tokenadult 13y ago

lutusp 13y ago
A quote: "So while it's interesting to hear that women are less likely to die of heart disease if they have a partner, no one's going to go out to find one in order to have a marginally better chance of making it to old age." The above sentence shows that the author -- and most readers -- miss the single most important thing about cohort studies -- which is that they can't distinguish causes from effects. Contrary to everyday understanding, the statement "women are less likely to die of heart disease if they have a partner" does not mean that choosing to have a partner results in a reduction of a woman's chance to die of heart disease, and it does not mean that seeking out a partner will reduce one's changes to die of heart disease. It only means that having a partner, and reduced chance of dying of heart disease, are correlated. The above very common confusion is the source of any number of meaningless "breakthroughs" in science publishing, and fuels any number of public controversies with little or no scientific substance.
- dllthomas 13y ago
  It does mean that the particular causal link in question is marginally more likely than if the correlation had not been observed. Not enough to base action on, though. It certainly implies (how strongly depends on the actual data) that there is some causal link (A -> B, B -> A, or A <- C -> B), because something is reducing the entropy between the two systems. That again, though, doesn't really tell us enough to be useful.
  - Retric 13y ago
    It can be a lot more complex than that. Suppose healthy people live longer and are more attractive and Attractive people are more likely to be married. So A <- D <- C -> B. Sometimes A between .5 and .8 -> B but A less than .5 or more than .8 -> !B. So now you can have study's that show more A -> B and others that less A -> B. (Think nutrition)
    - dllthomas 13y ago
      My arrows weren't meant to be direct causal links. It's still true that there are potential situations I didn't touch (A <- B and B -> A, &c), but those can be reduced to multiple instances of the above. Edited to add: "Sometimes A between .5 and .8 -> B but A less than .5 or more than .8 -> !B. So now you can have study's that show more A -> B and others that less A -> B. (Think nutrition)" My arrows also weren't meant to be implication, just a causal relationship. A -> B and !A -> B would be the same thing. I should probably have been more explicit about all these points.
      - lutusp 13y ago
        > My arrows also weren't meant to be implication, just a causal relationship. But in the typical case of a measured correlation, no causal relationship can be implied based on the data alone, regardless of the direction. This is especially true in examples of data mining, where many correlations are located that arise purely from chance rather than design.
        dllthomas 13y ago
        No particular causal relationship, right. Existence of a causal relationship can be.
        Retric 13y ago
        He is referring to the issue where data-mining creates a lot of meaningless correlations. http://xkcd.com/882/ http://xkcd.com/882/ aka many correlations are located that arise purely from chance rather than design A single study is generally close to worthless.
        dllthomas 13y ago
        Yes, the correlation has to be sufficiently unlikely to have arisen by chance. Even in that case, though, our estimate of green jellybeans having some causal relationship with acne should increase.
  - lutusp 13y ago
    > It does mean that the particular causal link in question is marginally more likely than if the correlation had not been observed. Not really -- it fails to consider the possibility that B caused A rather than the reverse, or that an unevaluated cause C caused both A and B. This is why I hate hearing the common phrase, "In a recent study, (something) has been linked to (something)". The words aren't false, but they're misleading to an untrained person. > That again, though, doesn't really tell us enough to be useful. That's certainly true.
    - dllthomas 13y ago
      "Not really -- it fails to consider the possibility that B caused A rather than the reverse, or that an unevaluated cause C caused both A and B." Yes really! Just because things are uncertain doesn't mean they're equally uncertain as you learn new facts! "A causes B" doesn't become more likely relative to "B causes A" when you learn there is a correlation, but both become more likely relative to "there is no causal relationship between A and B whatsoever" and thus more likely overall. Probabilities sum to 100%. We learned that one possibility is less likely - that probability has to go somewhere.
      - lutusp 13y ago
        > Just because things are uncertain doesn't mean they're equally uncertain as you learn new facts! But that's the foundation on which science is built -- the null hypothesis -- the default precept that a correlation has no meaning without evidence supporting one. The idea that unexplained correlations are assumed to arise from chance, not an uncovered cause. Until there's evidence that cannot be explained any other way, two apparently connected things are presumed to be connected by chance. > ... but both become more likely relative to "there is no causal relationship between A and B whatsoever" You're starting out making an assumption that has no basis. In science, the default assumption about a correlation is that it has no meaning whatsoever. > Probabilities sum to 100%. Yes, they do, but the initial probability is 100% that the correlation arises from chance. This is why the scientists at the LHC refused to speculate about the existence of the Higgs until they had acquired a p factor of 5σ. Details: http://blogs.scientificamerican.com/observations/2012/07/17/five-sigmawhats-that/ http://blogs.scientificamerican.com/observations/2012/07/17/... "Scientists use p-values to test the likelihood of hypotheses. In an experiment comparing some phenomenon A to phenomenon B, researchers construct two hypotheses: that “A and B are not correlated,” which is known as the null hypothesis, and that “A and B are correlated,” which is known as the research hypothesis." "The researchers then assume the null hypothesis (because it’s the most conservative supposition, intellectually) and calculate the probability of obtaining data as extreme or more extreme than what they observed, given that there is no relationship between A and B. This calculation, which yields the p-value, can be based on any of several different statistical tests. If the p-value is low, for example 0.01, this means that there is only a small chance (one percent for p=0.01) that the data would have been observed by chance without the correlation. Usually there is a pre-established threshold in a field of study for rejecting the null hypothesis and claiming that A and B are correlated. Values of p=0.05 and p=0.01 are very common in many scientific disciplines."
  - pavelrub 13y ago
    It means nothing. Causation can exist without correlation.
    - dllthomas 13y ago
      "Causation can exist without correlation." True, but I'm pretty sure I didn't rely on an assumption that it couldn't. Too much correlation without any thing causing that correlation is unlikely. How unlikely depends on just how much correlation. When something is way more unlikely given X than Y, that is evidence for X over Y.
      - pavelrub 13y ago
        No. Correlation doesn't make causation more likely. This is something you have made up.
        dllthomas 13y ago
        It's not something I've made up. It is a simple combination of Bayes' Theorem and thermodynamics. A strong correlation is a lower entropy system. The stronger the correlation, the more likely it is that there is some explanation for that correlation, even if we haven't found it yet.
        pavelrub 13y ago
        This is hand waving based on dubious analogies. You keep repeating the same thing without providing any justification, proof or sense behind it, despite being given counterexamples and counterarguments. Please show me a mathematical derivation of this "simple" result, based on whatever statistical model you would like. We are not talking about the likelihood of an "explanation", we are talking about the likelihood of x causing y.
  - snowwrestler 13y ago
    Only if you are testing a specific hypothesis. If you simply observe an apparent correlation for the first time, a causal relationship is no more likely than before your observation. It's simply the basis for forming a hypothesis, which is the first step needed to even look for causation. Most hypotheses do not pass testing. This implies that most observed correlations turn out to not be causal.
    - dllthomas 13y ago
      "Most observed correlations turn out not to be causal" is perfectly well consistent with "causation is more likely having observed a correlation than having failed to observe a correlation, or even not yet having observed".
      - snowwrestler 13y ago
        If you look for a specific correlation and fail to find it, yes. If you simply observe an unexpected correlation for the first time, no. You have no idea if it unique, remarkable, repeatable, etc.
- penguindev 13y ago
  This. If it's not a double-blind random crossover trial, it probably means nada. Thus, most nutritional, health, and economics 'science' is trash. Unfortunately, it also means it's very hard to prove things are safe or unsafe. Phracking, pesticides, GMO, ... all get a free ride for too long, IMHO, because they can say 'well nobody has proved they're unsafe'.
  - sp332 13y ago
    It's not trash. A single-blind trial is much better than nothing. Any variables you can remove will improve the probability of the result being right.
    - gnaritas 13y ago
      Not much better than nothing; results are useful after others have replicated them, until then one should assume the trial was flawed.
      - dllthomas 13y ago
        "one should assume the trial was flawed." Only to the extent that similar, pre-replication studies are typically flawed. I don't actually know what extent that is.
        gnaritas 13y ago
        Agreed.
        lutusp 13y ago
        >> "one should assume the trial was flawed." > Only to the extent that similar, pre-replication studies are typically flawed. I don't actually know what extent that is. There's a logical error in this approach. If one study has some probability p to be systematically flawed, can an evaluation of that study's outcome be modified by the knowledge that ten other studies, each with the same probability of flaw, exist? Well, no, not really -- they must all be assumed to be flawed, regardless of how many studies exist. If this were not the case, we could cancel out the effect of the systematic flaws in each study by performing a lot of flawed studies. (When this idea is put in practice, often it results in the round-filing of all the studies that failed to meet the researchers' expectations.) Notice that I'm addressing the question of how we might evaluate the outcome of one study, given the existence of other similar studies. A meta-analysis of all the studies at once uses different rules.
- jpmattia 13y ago
  > The above sentence shows that the author -- and most readers -- miss the single most important thing about cohort studies -- which is that they can't distinguish causes from effects. Even that sentence needs work: They can't distinguish between correlations and causes. And having spent too much time with the medical community lately, I'm not sure they're completely straight on the concepts either.
  - marcosdumay 13y ago
    > They can't distinguish between correlations and causes. Now I'm lost. As far as I understand, those studies only find correlations. What am I missing?
    - mjt0229 13y ago
      You're not missing anything. The people who are taking actions based on the correlations (i.e., treating them as causes) are probably missing something though.
  - lutusp 13y ago
    > Even that sentence needs work: They can't distinguish between correlations and causes. Fair enough, that's correct and important. My original didn't encompass the issue of tertiary causes, i.e. causes not accounted for in the correlation.
- zaptheimpaler 13y ago
  Theres another serious problem with blindly throwing in data and looking at what correlations come out. If you have a 100 variables and test for 100*99 correlations, you will very likely find one. Looking for patterns without a specific hypothesis to test (as a lot of pop-science studies do) is a dangerous approach.
  - dllthomas 13y ago
    http://xkcd.com/882 http://xkcd.com/882
jcromartie 13y ago
It's rational to be irrational, because being irrational feels good, which is what makes life worth living. At least this is true for the average person. There are individuals who get a lot out of a more methodical evidence-driven life.
- wwweston 13y ago
  I wonder how many such individuals there are -- maybe we could find out what the correlation is between methodical evidence-driven people and people who feel good!
hkon 13y ago
... And discussion on the internet.
squozzer 13y ago
Maybe worse, studies have become a bludgeon to justify government policy.
0xdeadbeefbabe 13y ago
The misunderstood scientific study is still a flawed manual for living.
nazgulnarsil 13y ago
We need services that curate how much you should actually pay attention to a given study. Science journalists have proven themselves inadequate to the task because they are after eyeballs. Large, well designed studies, with applicable samples and large effect sizes? Yes. Small studies with poor design on rats, or elderly Finnish twins with diabetes? No. Why can't I find a list of some of the largest and most robust systematic reviews and meta anlyses ever done?
- biofox 13y ago
  "Why can't I find a list of some of the largest and most robust systematic reviews and meta anlyses ever done?" The Cochrane Collaboration is the closest thing I know of: http://www.cochrane.org/ http://www.cochrane.org/ http://en.wikipedia.org/wiki/Cochrane_Collaboration http://en.wikipedia.org/wiki/Cochrane_Collaboration
  - nazgulnarsil 13y ago
    Yes, and I use them extensively. But they do not publish a guide to the studies they think are most worth paying attention to due to their quality and size.

41 points by tokenadult 13y ago

A quote: "So while it's interesting to hear that women are less likely to die of heart disease if they have a partner, no one's going to go out to find one in order to have a marginally better chance of making it to old age." The above sentence shows that the author -- and most readers -- miss the single most important thing about cohort studies -- which is that they can't distinguish causes from effects. Contrary to everyday understanding, the statement "women are less likely to die of heart disease if they have a partner" does not mean that choosing to have a partner results in a reduction of a woman's chance to die of heart disease, and it does not mean that seeking out a partner will reduce one's changes to die of heart disease. It only means that having a partner, and reduced chance of dying of heart disease, are correlated. The above very common confusion is the source of any number of meaningless "breakthroughs" in science publishing, and fuels any number of public controversies with little or no scientific substance.

dllthomas 13y ago
It does mean that the particular causal link in question is marginally more likely than if the correlation had not been observed. Not enough to base action on, though. It certainly implies (how strongly depends on the actual data) that there is some causal link (A -> B, B -> A, or A <- C -> B), because something is reducing the entropy between the two systems. That again, though, doesn't really tell us enough to be useful.
- Retric 13y ago
  It can be a lot more complex than that. Suppose healthy people live longer and are more attractive and Attractive people are more likely to be married. So A <- D <- C -> B. Sometimes A between .5 and .8 -> B but A less than .5 or more than .8 -> !B. So now you can have study's that show more A -> B and others that less A -> B. (Think nutrition)
  - dllthomas 13y ago
    My arrows weren't meant to be direct causal links. It's still true that there are potential situations I didn't touch (A <- B and B -> A, &c), but those can be reduced to multiple instances of the above. Edited to add: "Sometimes A between .5 and .8 -> B but A less than .5 or more than .8 -> !B. So now you can have study's that show more A -> B and others that less A -> B. (Think nutrition)" My arrows also weren't meant to be implication, just a causal relationship. A -> B and !A -> B would be the same thing. I should probably have been more explicit about all these points.
    - lutusp 13y ago
      > My arrows also weren't meant to be implication, just a causal relationship. But in the typical case of a measured correlation, no causal relationship can be implied based on the data alone, regardless of the direction. This is especially true in examples of data mining, where many correlations are located that arise purely from chance rather than design.
      - dllthomas 13y ago
        No particular causal relationship, right. Existence of a causal relationship can be.
        Retric 13y ago
        He is referring to the issue where data-mining creates a lot of meaningless correlations. http://xkcd.com/882/ http://xkcd.com/882/ aka many correlations are located that arise purely from chance rather than design A single study is generally close to worthless.
        dllthomas 13y ago
        Yes, the correlation has to be sufficiently unlikely to have arisen by chance. Even in that case, though, our estimate of green jellybeans having some causal relationship with acne should increase.
- lutusp 13y ago
  > It does mean that the particular causal link in question is marginally more likely than if the correlation had not been observed. Not really -- it fails to consider the possibility that B caused A rather than the reverse, or that an unevaluated cause C caused both A and B. This is why I hate hearing the common phrase, "In a recent study, (something) has been linked to (something)". The words aren't false, but they're misleading to an untrained person. > That again, though, doesn't really tell us enough to be useful. That's certainly true.
  - dllthomas 13y ago
    "Not really -- it fails to consider the possibility that B caused A rather than the reverse, or that an unevaluated cause C caused both A and B." Yes really! Just because things are uncertain doesn't mean they're equally uncertain as you learn new facts! "A causes B" doesn't become more likely relative to "B causes A" when you learn there is a correlation, but both become more likely relative to "there is no causal relationship between A and B whatsoever" and thus more likely overall. Probabilities sum to 100%. We learned that one possibility is less likely - that probability has to go somewhere.
    - lutusp 13y ago
      > Just because things are uncertain doesn't mean they're equally uncertain as you learn new facts! But that's the foundation on which science is built -- the null hypothesis -- the default precept that a correlation has no meaning without evidence supporting one. The idea that unexplained correlations are assumed to arise from chance, not an uncovered cause. Until there's evidence that cannot be explained any other way, two apparently connected things are presumed to be connected by chance. > ... but both become more likely relative to "there is no causal relationship between A and B whatsoever" You're starting out making an assumption that has no basis. In science, the default assumption about a correlation is that it has no meaning whatsoever. > Probabilities sum to 100%. Yes, they do, but the initial probability is 100% that the correlation arises from chance. This is why the scientists at the LHC refused to speculate about the existence of the Higgs until they had acquired a p factor of 5σ. Details: http://blogs.scientificamerican.com/observations/2012/07/17/five-sigmawhats-that/ http://blogs.scientificamerican.com/observations/2012/07/17/... "Scientists use p-values to test the likelihood of hypotheses. In an experiment comparing some phenomenon A to phenomenon B, researchers construct two hypotheses: that “A and B are not correlated,” which is known as the null hypothesis, and that “A and B are correlated,” which is known as the research hypothesis." "The researchers then assume the null hypothesis (because it’s the most conservative supposition, intellectually) and calculate the probability of obtaining data as extreme or more extreme than what they observed, given that there is no relationship between A and B. This calculation, which yields the p-value, can be based on any of several different statistical tests. If the p-value is low, for example 0.01, this means that there is only a small chance (one percent for p=0.01) that the data would have been observed by chance without the correlation. Usually there is a pre-established threshold in a field of study for rejecting the null hypothesis and claiming that A and B are correlated. Values of p=0.05 and p=0.01 are very common in many scientific disciplines."
- pavelrub 13y ago
  It means nothing. Causation can exist without correlation.
  - dllthomas 13y ago
    "Causation can exist without correlation." True, but I'm pretty sure I didn't rely on an assumption that it couldn't. Too much correlation without any thing causing that correlation is unlikely. How unlikely depends on just how much correlation. When something is way more unlikely given X than Y, that is evidence for X over Y.
    - pavelrub 13y ago
      No. Correlation doesn't make causation more likely. This is something you have made up.
      - dllthomas 13y ago
        It's not something I've made up. It is a simple combination of Bayes' Theorem and thermodynamics. A strong correlation is a lower entropy system. The stronger the correlation, the more likely it is that there is some explanation for that correlation, even if we haven't found it yet.
        pavelrub 13y ago
        This is hand waving based on dubious analogies. You keep repeating the same thing without providing any justification, proof or sense behind it, despite being given counterexamples and counterarguments. Please show me a mathematical derivation of this "simple" result, based on whatever statistical model you would like. We are not talking about the likelihood of an "explanation", we are talking about the likelihood of x causing y.
- snowwrestler 13y ago
  Only if you are testing a specific hypothesis. If you simply observe an apparent correlation for the first time, a causal relationship is no more likely than before your observation. It's simply the basis for forming a hypothesis, which is the first step needed to even look for causation. Most hypotheses do not pass testing. This implies that most observed correlations turn out to not be causal.
  - dllthomas 13y ago
    "Most observed correlations turn out not to be causal" is perfectly well consistent with "causation is more likely having observed a correlation than having failed to observe a correlation, or even not yet having observed".
    - snowwrestler 13y ago
      If you look for a specific correlation and fail to find it, yes. If you simply observe an unexpected correlation for the first time, no. You have no idea if it unique, remarkable, repeatable, etc.
penguindev 13y ago
This. If it's not a double-blind random crossover trial, it probably means nada. Thus, most nutritional, health, and economics 'science' is trash. Unfortunately, it also means it's very hard to prove things are safe or unsafe. Phracking, pesticides, GMO, ... all get a free ride for too long, IMHO, because they can say 'well nobody has proved they're unsafe'.
- sp332 13y ago
  It's not trash. A single-blind trial is much better than nothing. Any variables you can remove will improve the probability of the result being right.
  - gnaritas 13y ago
    Not much better than nothing; results are useful after others have replicated them, until then one should assume the trial was flawed.
    - dllthomas 13y ago
      "one should assume the trial was flawed." Only to the extent that similar, pre-replication studies are typically flawed. I don't actually know what extent that is.
      - gnaritas 13y ago
        Agreed.
      - lutusp 13y ago
        >> "one should assume the trial was flawed." > Only to the extent that similar, pre-replication studies are typically flawed. I don't actually know what extent that is. There's a logical error in this approach. If one study has some probability p to be systematically flawed, can an evaluation of that study's outcome be modified by the knowledge that ten other studies, each with the same probability of flaw, exist? Well, no, not really -- they must all be assumed to be flawed, regardless of how many studies exist. If this were not the case, we could cancel out the effect of the systematic flaws in each study by performing a lot of flawed studies. (When this idea is put in practice, often it results in the round-filing of all the studies that failed to meet the researchers' expectations.) Notice that I'm addressing the question of how we might evaluate the outcome of one study, given the existence of other similar studies. A meta-analysis of all the studies at once uses different rules.
jpmattia 13y ago
> The above sentence shows that the author -- and most readers -- miss the single most important thing about cohort studies -- which is that they can't distinguish causes from effects. Even that sentence needs work: They can't distinguish between correlations and causes. And having spent too much time with the medical community lately, I'm not sure they're completely straight on the concepts either.
- marcosdumay 13y ago
  > They can't distinguish between correlations and causes. Now I'm lost. As far as I understand, those studies only find correlations. What am I missing?
  - mjt0229 13y ago
    You're not missing anything. The people who are taking actions based on the correlations (i.e., treating them as causes) are probably missing something though.
- lutusp 13y ago
  > Even that sentence needs work: They can't distinguish between correlations and causes. Fair enough, that's correct and important. My original didn't encompass the issue of tertiary causes, i.e. causes not accounted for in the correlation.
zaptheimpaler 13y ago
Theres another serious problem with blindly throwing in data and looking at what correlations come out. If you have a 100 variables and test for 100*99 correlations, you will very likely find one. Looking for patterns without a specific hypothesis to test (as a lot of pop-science studies do) is a dangerous approach.
- dllthomas 13y ago
  http://xkcd.com/882 http://xkcd.com/882

jcromartie 13y ago

It's rational to be irrational, because being irrational feels good, which is what makes life worth living. At least this is true for the average person. There are individuals who get a lot out of a more methodical evidence-driven life.

wwweston 13y ago
I wonder how many such individuals there are -- maybe we could find out what the correlation is between methodical evidence-driven people and people who feel good!

hkon 13y ago

... And discussion on the internet.

squozzer 13y ago

Maybe worse, studies have become a bludgeon to justify government policy.

0xdeadbeefbabe 13y ago

The misunderstood scientific study is still a flawed manual for living.

nazgulnarsil 13y ago

We need services that curate how much you should actually pay attention to a given study. Science journalists have proven themselves inadequate to the task because they are after eyeballs. Large, well designed studies, with applicable samples and large effect sizes? Yes. Small studies with poor design on rats, or elderly Finnish twins with diabetes? No. Why can't I find a list of some of the largest and most robust systematic reviews and meta anlyses ever done?

biofox 13y ago
"Why can't I find a list of some of the largest and most robust systematic reviews and meta anlyses ever done?" The Cochrane Collaboration is the closest thing I know of: http://www.cochrane.org/ http://www.cochrane.org/ http://en.wikipedia.org/wiki/Cochrane_Collaboration http://en.wikipedia.org/wiki/Cochrane_Collaboration
- nazgulnarsil 13y ago
  Yes, and I use them extensively. But they do not publish a guide to the studies they think are most worth paying attention to due to their quality and size.