4 ms·

I am not a trained statistician, so can someone help me understand why this is even publishable? I read the abstract and found they quote their key statistic a

I am not a trained statistician, so can someone help me understand why this is even publishable? I read the abstract and found they quote their key statistic at P = 0.01. The "NHANES III" study that their data comes from has over 1000 variables. Can't you just cherry-pick a handful of "significant" variables at the P = 0.01 level, when there are 1000 variables to choose from?

nonbel 10y ago
The entire idea of using a default null hypothesis and calculating a p-value in order to learn something is fatally flawed to begin with. If p-values are to be of any use, the hypothesis you are trying to reject needs to be predicted by your theory/model, so set that as the null hypothesis. You know... how science works. To start you off: http://library.mpib-berlin.mpg.de/ft/gg/GG_Mindless_2004.pdf http://library.mpib-berlin.mpg.de/ft/gg/GG_Mindless_2004.pdf http://www.fisme.science.uu.nl/staff/christianb/downloads/meehl1967.pdf http://www.fisme.science.uu.nl/staff/christianb/downloads/me... http://andrewgelman.com/2016/09/30/why-the-garden-of-forking-paths-criticism-of-p-values-is-not-like-a-famous-borscht-belt-comedy-bit/ http://andrewgelman.com/2016/09/30/why-the-garden-of-forking...
martingoodson 10y ago
Yes. They could also cherry pick an outcome variable (eg diastolic blood pressure etc). Given that they haven't applied any multiple testing correction (eg bonferroni or fdr) it's basically impossible to NOT find a significant association in this dataset.