3 ms·

Ok, so the pretty result is mostly arbitrary. Fair enough. Re: false negatives... you seem to be living in a world of bell curves, or at least a mostly contin

by marvy 11y ago

Ok, so the pretty result is mostly arbitrary. Fair enough. Re: false negatives... you seem to be living in a world of bell curves, or at least a mostly continuous world. I can easily make (very contrived) experiments where false negatives just don't happen. For instance: I have two coins. One is a perfectly fair coin. The other is a two-headed coin. You see me flip one of them. The null hypothesis is that I flipped the fair coin. A false negative means deciding the coin is fair but it's really not. This will never happen, because you will only decide that if it lands tails, and then it must be fair. (If I only flip it once, the false positive rate is something like 1/3, not 0.) But this is probably much too contrived for your taste, and maybe even for mine. But it's almost 5am, and I must go to sleep now, or else it will get bright soon, and I never will. I now appreciate the value of the 20min procrastination setting.

kgwgk 11y ago
I agree on your point, if we are sufficiently creative we can get many extreme results. For example, I made an addition to the first paragraph of my previous comment, that you might have missed, giving an example where the probability of the null hypothesis being true when p=0.05 is zero (or arbitrarily small, if we replace the discrete probability lump under the alternative hypothesis by a continuous distribution which is concentrated enough). I also added a comment on the second paragraph, by the way. One minor comment on your example. If H0:fair coin and H1:two-headed and the statistic is the number of heads h, I cannot reject (at the 0.05 level) the null when n (the number of flips) is small even if I'm only getting heads. For one flip, p[h=1|H0]=0.5. For two flips p[h=2|H0]=0.25. For n>5 you will of course reject the null hypothesis for every case where H1 is true (and for ~5% of the cases where H0 is true). There will be no false negatives. But I guess you have noticed that this doesn't help with the false discovery rate in this example: when H1 is true the p-value will be very small (1/2^n) so if the observed p-value is ~0.05 (or any other value larger than 1/2^n) then it's for sure a false positive (because there will be at least one occurrence of tails). Ok, enough time wasted on this subject :-)
- marvy 11y ago
  > so if the observed p-value is ~0.05 (or any other value larger than 1/2^n) then it's for sure a false positive (because there will be at least one occurrence of tails). Good point! I didn't think of that. > Ok, enough time wasted on this subject :-) Even better point!

by marvy 11y ago

kgwgk 11y ago

I agree on your point, if we are sufficiently creative we can get many extreme results. For example, I made an addition to the first paragraph of my previous comment, that you might have missed, giving an example where the probability of the null hypothesis being true when p=0.05 is zero (or arbitrarily small, if we replace the discrete probability lump under the alternative hypothesis by a continuous distribution which is concentrated enough). I also added a comment on the second paragraph, by the way. One minor comment on your example. If H0:fair coin and H1:two-headed and the statistic is the number of heads h, I cannot reject (at the 0.05 level) the null when n (the number of flips) is small even if I'm only getting heads. For one flip, p[h=1|H0]=0.5. For two flips p[h=2|H0]=0.25. For n>5 you will of course reject the null hypothesis for every case where H1 is true (and for ~5% of the cases where H0 is true). There will be no false negatives. But I guess you have noticed that this doesn't help with the false discovery rate in this example: when H1 is true the p-value will be very small (1/2^n) so if the observed p-value is ~0.05 (or any other value larger than 1/2^n) then it's for sure a false positive (because there will be at least one occurrence of tails). Ok, enough time wasted on this subject :-)

marvy 11y ago
> so if the observed p-value is ~0.05 (or any other value larger than 1/2^n) then it's for sure a false positive (because there will be at least one occurrence of tails). Good point! I didn't think of that. > Ok, enough time wasted on this subject :-) Even better point!