3 ms·
I agree on your point, if we are sufficiently creative we can get many extreme results. For example, I made an addition to the first paragraph of my previous co
by kgwgk 11y ago
I agree on your point, if we are sufficiently creative we can get many extreme results. For example, I made an addition to the first paragraph of my previous comment, that you might have missed, giving an example where the probability of the null hypothesis being true when p=0.05 is zero (or arbitrarily small, if we replace the discrete probability lump under the alternative hypothesis by a continuous distribution which is concentrated enough). I also added a comment on the second paragraph, by the way.
One minor comment on your example. If H0:fair coin and H1:two-headed and the statistic is the number of heads h, I cannot reject (at the 0.05 level) the null when n (the number of flips) is small even if I'm only getting heads. For one flip, p[h=1|H0]=0.5. For two flips p[h=2|H0]=0.25. For n>5 you will of course reject the null hypothesis for every case where H1 is true (and for ~5% of the cases where H0 is true). There will be no false negatives. But I guess you have noticed that this doesn't help with the false discovery rate in this example: when H1 is true the p-value will be very small (1/2^n) so if the observed p-value is ~0.05 (or any other value larger than 1/2^n) then it's for sure a false positive (because there will be at least one occurrence of tails).
Ok, enough time wasted on this subject :-)
- marvy 11y ago> so if the observed p-value is ~0.05 (or any other value larger than 1/2^n) then it's for sure a false positive (because there will be at least one occurrence of tails). Good point! I didn't think of that. > Ok, enough time wasted on this subject :-) Even better point!