4 ms·

There's more to it than just the things you mentioned, and I'd argue that it's an even bigger problem. The fact studies are adaptively chosen and performed base

There's more to it than just the things you mentioned, and I'd argue that it's an even bigger problem. The fact studies are adaptively chosen and performed based on the results of past studies on the same (or related) data leads to increased instances of false discovery. This is something that is directly addressed in some very recent recent work (not uncoincidentally making reference to this exact XKCD comic): http://googleresearch.blogspot.com/2015/08/the-reusable-holdout-preserving.html http://googleresearch.blogspot.com/2015/08/the-reusable-hold...

Gatsky 11y ago
Freedman's paradox refers to adaptive data analysis within one dataset. The reusable holdout is designed to permit adaptive analysis on the same dataset without inflating the false discovery rate. The problem doesn't apply if you are collecting new data and testing the hypothesis, that's the ideal scenario.
- Adabenflaben 11y ago
  Yeah, Freedman's paradox is definitely part of what I'm alluding to. The other aspect is that for widely studied phenomena, some groups are likely to come to wrong conclusions just due to statistical error. E.g., if 100 groups independently investigating whether "Eating chocolate is associated with improved brain function" (and we'll assume it's false, just to make my point), a few are likely going to come to the conclusion that it's true. Even worse, some studies with "expected" results may not be published (either by choice of the investigator or by the reviewers); instead, the "interesting" or "exciting" results will get published.
autokad 11y ago
very true. more specifically, you generally find what you are looking for. much like trying to google if you have some sort of rare disease. people should pay attention to how they feel 30 minutes after eating a particular food more than any research study that has ever been published.

Gatsky 11y ago

Freedman's paradox refers to adaptive data analysis within one dataset. The reusable holdout is designed to permit adaptive analysis on the same dataset without inflating the false discovery rate. The problem doesn't apply if you are collecting new data and testing the hypothesis, that's the ideal scenario.

Adabenflaben 11y ago
Yeah, Freedman's paradox is definitely part of what I'm alluding to. The other aspect is that for widely studied phenomena, some groups are likely to come to wrong conclusions just due to statistical error. E.g., if 100 groups independently investigating whether "Eating chocolate is associated with improved brain function" (and we'll assume it's false, just to make my point), a few are likely going to come to the conclusion that it's true. Even worse, some studies with "expected" results may not be published (either by choice of the investigator or by the reviewers); instead, the "interesting" or "exciting" results will get published.

autokad 11y ago

very true. more specifically, you generally find what you are looking for. much like trying to google if you have some sort of rare disease. people should pay attention to how they feel 30 minutes after eating a particular food more than any research study that has ever been published.