3 ms·
Freedman's paradox refers to adaptive data analysis within one dataset. The reusable holdout is designed to permit adaptive analysis on the same dataset without
by Gatsky 11y ago
Freedman's paradox refers to adaptive data analysis within one dataset. The reusable holdout is designed to permit adaptive analysis on the same dataset without inflating the false discovery rate. The problem doesn't apply if you are collecting new data and testing the hypothesis, that's the ideal scenario.
- Adabenflaben 11y agoYeah, Freedman's paradox is definitely part of what I'm alluding to. The other aspect is that for widely studied phenomena, some groups are likely to come to wrong conclusions just due to statistical error. E.g., if 100 groups independently investigating whether "Eating chocolate is associated with improved brain function" (and we'll assume it's false, just to make my point), a few are likely going to come to the conclusion that it's true. Even worse, some studies with "expected" results may not be published (either by choice of the investigator or by the reviewers); instead, the "interesting" or "exciting" results will get published.