4 ms·

I have a question. Would I be on firm statistical footing if I started looking for effects after the study, as long as I choose a p-value such that (1-((1-p)^n)

I have a question. Would I be on firm statistical footing if I started looking for effects after the study, as long as I choose a p-value such that (1-((1-p)^n) < 0.05? In other words, I run a study to determine if jellybeans cause acne[1]. The result is inconclusive (p < .05). Now--after the results are collected--I wish to check the correlation between color and acne. There are 20 colors. Would it be statistically sound to "correct" for my cherrypicking by setting p = 0.0025? That would result in 1 - (0.9975^20) ≈ 0.049 [1] https://xkcd.com/882/ https://xkcd.com/882/

conistonwater 11y ago
> Would I be on firm statistical footing if I started looking for effects after the study, as long as I choose a p-value such that (1-((1-p)^n) < 0.05? No. For one thing, how do you actually know n? If in your analysis you can make c independent binary choices, n would be on the order of 2^c. For most reasonable sequence of choices, n would be impractically large. And if you think you know n, how would you convince everyone else that your value of n is reasonable/trustworthy? The xkcd case (which is solved by applying the Bonferroni correction) is special because there's only one test run in 20 experiments, so the correction is more straightforward. There is a good paper about this garden of forking paths by Gelman and Loken [1]. [1] (pdf) http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/p_...
bunderbunder 11y ago
It sounds like what you're describing is roughly the same as the Bonferroni correction: https://en.wikipedia.org/wiki/Bonferroni_correction https://en.wikipedia.org/wiki/Bonferroni_correction It's no panacea. When you use it you reduce your statistical power - in other words, your likelihood of detecting genuine effects goes down.

conistonwater 11y ago

> Would I be on firm statistical footing if I started looking for effects after the study, as long as I choose a p-value such that (1-((1-p)^n) < 0.05? No. For one thing, how do you actually know n? If in your analysis you can make c independent binary choices, n would be on the order of 2^c. For most reasonable sequence of choices, n would be impractically large. And if you think you know n, how would you convince everyone else that your value of n is reasonable/trustworthy? The xkcd case (which is solved by applying the Bonferroni correction) is special because there's only one test run in 20 experiments, so the correction is more straightforward. There is a good paper about this garden of forking paths by Gelman and Loken [1]. [1] (pdf) http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/p_...

bunderbunder 11y ago

It sounds like what you're describing is roughly the same as the Bonferroni correction: https://en.wikipedia.org/wiki/Bonferroni_correction https://en.wikipedia.org/wiki/Bonferroni_correction It's no panacea. When you use it you reduce your statistical power - in other words, your likelihood of detecting genuine effects goes down.