6 ms·

I long for the day when these correlation-digging studies with absurd claims will stop. https://xkcd.com/882/ https://xkcd.com/882/

by yxitcti 11y ago

adenadel 11y ago
You should take a look at the journal article. They comment on how they corrected for multiple testing "In addition to MANOVA, the Bonferroni procedure was used to protect against multiple comparisons." The Bonferroni correction is one of the most conservative ways to correct for multiple testing. https://en.wikipedia.org/wiki/Bonferroni_correction https://en.wikipedia.org/wiki/Bonferroni_correction
- Gatsky 11y ago
  The study is still 'correlation-digging'. Bonferroni won't fix the problem that these kinds of studies will throw up statistically significant associations that provide no insight into anything meaningful, due to confounders, biases, poor design etc. There is no equivalent xkcd however.
  - adenadel 11y ago
    I don't think that anyone would take the results of this article as a statement of absolute fact. In the discussion they mentioned some more (better controlled) studies investigating the role of cocoa flavanols in "cognitive tasks". Doing any longitudinal study with humans suffer from confounders, biases, and poor design because we can't do controlled, long-term research on humans. You just have to take these limitations into account when you are interpreting the results and use any findings as areas for possible further investigation.
    - Gatsky 11y ago
      Yes, that's fine, I was pointing out that correcting for multiple hypothesis testing does not deal with the issue cited (albeit rather abruptly) by yxitcti.
  - Adabenflaben 11y ago
    There's more to it than just the things you mentioned, and I'd argue that it's an even bigger problem. The fact studies are adaptively chosen and performed based on the results of past studies on the same (or related) data leads to increased instances of false discovery. This is something that is directly addressed in some very recent recent work (not uncoincidentally making reference to this exact XKCD comic): http://googleresearch.blogspot.com/2015/08/the-reusable-holdout-preserving.html http://googleresearch.blogspot.com/2015/08/the-reusable-hold...
    - Gatsky 11y ago
      Freedman's paradox refers to adaptive data analysis within one dataset. The reusable holdout is designed to permit adaptive analysis on the same dataset without inflating the false discovery rate. The problem doesn't apply if you are collecting new data and testing the hypothesis, that's the ideal scenario.
      - Adabenflaben 11y ago
        Yeah, Freedman's paradox is definitely part of what I'm alluding to. The other aspect is that for widely studied phenomena, some groups are likely to come to wrong conclusions just due to statistical error. E.g., if 100 groups independently investigating whether "Eating chocolate is associated with improved brain function" (and we'll assume it's false, just to make my point), a few are likely going to come to the conclusion that it's true. Even worse, some studies with "expected" results may not be published (either by choice of the investigator or by the reviewers); instead, the "interesting" or "exciting" results will get published.
    - autokad 11y ago
      very true. more specifically, you generally find what you are looking for. much like trying to google if you have some sort of rare disease. people should pay attention to how they feel 30 minutes after eating a particular food more than any research study that has ever been published.
elchief 11y ago
Can we just have this one? It's chocolate, man.
- chris_wot 11y ago
  Sure, eat your heart out. Just try not to eat your heart out.
vixen99 11y ago
Correlation-digging studies absurd or otherwise have led to an awful lot of peer-reviewed literature detailing a raft of <<molecular mechanisms>> by which flavonoids may be implicated in improved brain function. To pick one at random: "Flavonoids exert a multiplicity of neuroprotective actions within the brain, including a potential to protect neurons against injury induced by neurotoxins, an ability to suppress neuroinflammation, and the potential to promote memory, learning and cognitive function. These effects appear to be underpinned by two common processes . . ." http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2593006/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2593006/ Most of us enjoy eating the stuff so it's no surprise that chocolate as a flavonoid-containing substance gets special attention.
noobie 11y ago
Can someone please explain to me the difference between p-value and the confidence interval?

I long for the day when these correlation-digging studies with absurd claims will stop. https://xkcd.com/882/ https://xkcd.com/882/

by yxitcti 11y ago

I long for the day when these correlation-digging studies with absurd claims will stop. https://xkcd.com/882/ https://xkcd.com/882/

adenadel 11y ago

You should take a look at the journal article. They comment on how they corrected for multiple testing "In addition to MANOVA, the Bonferroni procedure was used to protect against multiple comparisons." The Bonferroni correction is one of the most conservative ways to correct for multiple testing. https://en.wikipedia.org/wiki/Bonferroni_correction https://en.wikipedia.org/wiki/Bonferroni_correction

Gatsky 11y ago
The study is still 'correlation-digging'. Bonferroni won't fix the problem that these kinds of studies will throw up statistically significant associations that provide no insight into anything meaningful, due to confounders, biases, poor design etc. There is no equivalent xkcd however.
- adenadel 11y ago
  I don't think that anyone would take the results of this article as a statement of absolute fact. In the discussion they mentioned some more (better controlled) studies investigating the role of cocoa flavanols in "cognitive tasks". Doing any longitudinal study with humans suffer from confounders, biases, and poor design because we can't do controlled, long-term research on humans. You just have to take these limitations into account when you are interpreting the results and use any findings as areas for possible further investigation.
  - Gatsky 11y ago
    Yes, that's fine, I was pointing out that correcting for multiple hypothesis testing does not deal with the issue cited (albeit rather abruptly) by yxitcti.
- Adabenflaben 11y ago
  There's more to it than just the things you mentioned, and I'd argue that it's an even bigger problem. The fact studies are adaptively chosen and performed based on the results of past studies on the same (or related) data leads to increased instances of false discovery. This is something that is directly addressed in some very recent recent work (not uncoincidentally making reference to this exact XKCD comic): http://googleresearch.blogspot.com/2015/08/the-reusable-holdout-preserving.html http://googleresearch.blogspot.com/2015/08/the-reusable-hold...
  - Gatsky 11y ago
    Freedman's paradox refers to adaptive data analysis within one dataset. The reusable holdout is designed to permit adaptive analysis on the same dataset without inflating the false discovery rate. The problem doesn't apply if you are collecting new data and testing the hypothesis, that's the ideal scenario.
    - Adabenflaben 11y ago
      Yeah, Freedman's paradox is definitely part of what I'm alluding to. The other aspect is that for widely studied phenomena, some groups are likely to come to wrong conclusions just due to statistical error. E.g., if 100 groups independently investigating whether "Eating chocolate is associated with improved brain function" (and we'll assume it's false, just to make my point), a few are likely going to come to the conclusion that it's true. Even worse, some studies with "expected" results may not be published (either by choice of the investigator or by the reviewers); instead, the "interesting" or "exciting" results will get published.
  - autokad 11y ago
    very true. more specifically, you generally find what you are looking for. much like trying to google if you have some sort of rare disease. people should pay attention to how they feel 30 minutes after eating a particular food more than any research study that has ever been published.

elchief 11y ago

Can we just have this one? It's chocolate, man.

chris_wot 11y ago
Sure, eat your heart out. Just try not to eat your heart out.

vixen99 11y ago

Correlation-digging studies absurd or otherwise have led to an awful lot of peer-reviewed literature detailing a raft of <<molecular mechanisms>> by which flavonoids may be implicated in improved brain function. To pick one at random: "Flavonoids exert a multiplicity of neuroprotective actions within the brain, including a potential to protect neurons against injury induced by neurotoxins, an ability to suppress neuroinflammation, and the potential to promote memory, learning and cognitive function. These effects appear to be underpinned by two common processes . . ." http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2593006/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2593006/ Most of us enjoy eating the stuff so it's no surprise that chocolate as a flavonoid-containing substance gets special attention.

noobie 11y ago

Can someone please explain to me the difference between p-value and the confidence interval?