2 ms·
Even more troubling, they completely screwed up the analysis, making a mistake that even undergraduates know to avoid. They measured 12 factors (as shown in ta
by leereeves 5y ago
Even more troubling, they completely screwed up the analysis, making a mistake that even undergraduates know to avoid.
They measured 12 factors (as shown in table 1) across 4 treatment groups (male/female in two generations separately). That's 48 variables, so with the p<0.05 significance level they used, they would, on average, see 2.4 "statistically significant" results just by random chance.
If I'm counting correctly, they found 2 "statistically significant" results, exactly as expected by random chance.
In other words, their results are statistically indistinguishable from chance, exactly like the famous "jelly beans cause acne" thought experiment.
https://www.explainxkcd.com/wiki/index.php/882:_Significant https://www.explainxkcd.com/wiki/index.php/882:_Significant
Edit: Who's downvoting this without a response? Is there a mistake or is it simply inconvenient for the people who badly want to see an effect?
- 1e-9 5y agoYes, it’s not valid to reject a null hypothesis involving multiple tests without using a multiple testing correction. The simplest conservative approach would be to use a Bonferroni correction and check whether any of the M tests has a p-value below alpha/M. Clearly, these results are not that significant. The study suggests a couple potential effects. To credibly evaluate them, the study needs to be repeated with more mice and/or fewer tests along with appropriate statistical analysis.