4 ms·
Related but different for people who haven't done a lot of stats: your significance goes down with the number of hypotheses. If you've got 20 scenarios and loo
by rfrey 15y ago
Related but different for people who haven't done a lot of stats: your significance goes down with the number of hypotheses. If you've got 20 scenarios and looking for a 5% significance, one of those will be significant purely by chance.
You can correct for the "multiple hypotheses" problem by using a significance equal to (1 - (0.95)^(1/n)), substituting whatever significance you want for the 0.95 and using n=number of hypotheses. http://en.wikipedia.org/wiki/Bonferroni_correction http://en.wikipedia.org/wiki/Bonferroni_correction
The table for that correction is equally frightening:
#hypoth req.sig
=================
1 0.05
2 0.025
3 0.017
5 0.01
10 0.005
- gcp 15y agoBonferroni correction is an extremely conservative correction, which loses significance very quickly. Depending on the relation between the hypothesises, and the relative magnitude of the significance levels, much better methods are available, such as bootstrap step-downs. But in general, it's good to make people aware how quickly you are doing multiple comparisons and how it invalidates the significance levels completely. Corresponding xkcd, which really explains it brilliantly: http://xkcd.com/882/ http://xkcd.com/882/
- laughinghan 15y agoIs it just me, or does that table look like req.sig = (intended significance)/(#hypoth) ? Which is kind of a simpler formula. Edit: Following your link, the table you listed is indeed of the Bonferroni correction, and the formula is as I stated. The formula you stated is actually of the Sidak correction, which "is often confused with the Bonferroni correction", according to your link.
- timr 15y agoThe Dunn-Sidak correction is preferred over the Bonferroni correction -- it's much less conservative, and will allow the finding of significance in situations that Bonferroni would miss. Another thing: depending on how your problem is structured, it might be a bit confusing to think of these as corrections for the number of hypotheses. I like to think of them in terms of the number of planned/unplanned comparisons that are being performed: you can do an experiment with a single stated hypothesis, yet still need to use these corrections if you perform "unplanned" comparisons using the data (a.k.a "data mining", or "a data fishing expedition") later on. This is a really great summary of planned vs. unplanned comparisons and why they matter: http://udel.edu/~mcdonald/statanovaplanned.html http://udel.edu/~mcdonald/statanovaplanned.html