4 ms·

I'm probably commenting too late to get my question answered, but here goes: the article has a pretty picture where they show how likely your p-values will misl

by marvy 11y ago

I'm probably commenting too late to get my question answered, but here goes: the article has a pretty picture where they show how likely your p-values will mislead you depending on how likely the null hypothesis is. For instance, they say if you think that the null hypothesis has a 50% probability of being right, and you get p=5%, then there's still a 29% chance the null hypothesis is true. But according to my calculations, the right number should be 1/21 = 4.8%. What am I missing here? Or are they wrong? My calculations are below: Curious George has 200 fascinating phenomena he wishes to investigate. In reality, 100 of those are real, and the other hundred are mere coincidences. The experiments for the 100 real phenomena all show that "yes, this is for real". (I'm assuming no false negatives.) Most of the 100 experiments that test bogus phenomena show that "this is bogus", but 5 of them achieve a significance of p=5%, as expected. George then runs of to tell the Man in the Yellow Hat about his 105 amazing discoveries. If Yellow Hat Man knows that half of the phenomena that capture George's attention are bogus, he knows that 5/105 = 1/21 = 4.8% of George's discoveries are likely bogus, even though he doesn't know which ones.

kgwgk 11y ago
Assume that you're sampling from a normal distribution with known standard deviation sigma (1 for simplicity) and unknown mean mu. To test if the mean is larger than (the null hypothesis) mu=0 you can check if the observed value is larger than 1.64 sigma (for the 95% confidence test). So if your observation is larger than 1.64 you reject the null hypothesis. Your calculation would be correct only if the assumption "no false negatives" is approximately valid. This is the case when the true value is large in terms of sigma (say mu=6). Then for the 100 cases with mu=0 you'll reject the null 5 times on average, and for each one of the 100 cases with mu=100 you will reject the null (unless you're unlucky: there will be a false negative around once in 150000 trials). But you're conditioning on p<0.05, not on p~0.05. It's easy to see that it's much easier to get p=0.05 if mu=0 (this is a 1.64 sigma event) than if mu=6 (it's a 4.36 sigma event). If mu=0, you will get on average 1 (out of 100) observation with 0.04<p<0.05 (i.e. 1.64<x<1.75). The probability of obtaining an observation on that range when mu=6 is very small (0.0004 out of 100). Almost 100% of the "discoveries" with p~0.05 will be false (when mu=6 you will get p-values around 1e-9). When the true value of mu gets closer to 0, you cannot ignore the false negatives. For example if mu=0.1 the rejection rate will be quite similar to the mu=0 case (the probability of getting 0.4<p<0.5 is 1.2% and 1% respectively) and almost 50% of the "discoveries" with p~0.05 will be false. Somewhere between the two extreme cases, there is a lower bound for this "false discovery rate". See http://faculty.washington.edu/jonno/SISG-2011/lectures/sellkeetal01.pdf http://faculty.washington.edu/jonno/SISG-2011/lectures/sellk... and in particular figure 2.
- marvy 11y ago
  First of all, thank you for responding, I expected that no one would. Second of all, you point out that "no false negatives" is not always realistic. Fair enough. That means that 4.8% need not be the right answer. But at least it's clear how I got it. I still have no clue how on earth they got 29%, rather than 28% or 31%. Am I missing something?
  - kgwgk 11y ago
    1/(1-1/(e p log(p)))) This is formula (3) in the linked paper. I agree that the derivation is not obvious, but probably you should get the same result using the calculation I described for different values of mu and looking for the minimum.
    - marvy 11y ago
      thanks!
- marvy 11y ago
  I think I'm still missing something here. In particular, I'm still not getting 29% as a lower bound. I'm getting around 20%. If we compare mu=0 with mu=1.64, the probability density at x=1.64 is roughly 0.1 and 0.4, respectively, so the lower bound should be .1/(.1+.4)=1/5. No? Unless they were assuming something other than "two normal distributions with the same variance"?
  - kgwgk 11y ago
    You're absolutely right! My example was mainly for illustration, I was not sure that it would give exactly the same lower bound (but I was indeed surprised that it's below the 29% in those papers, which I thought a "hard" bound). It seems the bound that you are calculating (that I have reproduced, R code below) was already published more than 50 years ago for this specific case of a normal distribution. See slide 10 in http://www.biostat.uzh.ch/teaching/master/previous/seminarbayes/LeonhardHeld_slides.pdf http://www.biostat.uzh.ch/teaching/master/previous/seminarba... I have not really read the paper of Sellke et al. entirely, but it seems that the "calibration" they propose is more general but it makes some assumptions about the distribution of the p-value and it's therefore approximative. p=0.05 null=0.5 c0=qnorm(1-p) x=seq(0,5,0.01) y=100*dnorm(c0)/(dnorm(c0)+dnorm(x-c0)) calib=100/(1-1/(exp(1)*p*log(p))) actual=min(y) plot(x,y,type="l",ylim=c(0,100),ylab="%null",xlab="mu1",bty="l",xaxs="i",yaxs="i") title(paste(null*100,"% null p = ",p,sep="")) legend("topleft",c(paste("Sellke, Bayarri, Berger (2001) =",format(calib,digits=3)), paste("Edwards, Lindman, Savage (1963) =",format(actual,digits=3))), lwd=2,col=c("red","blue"),bty="n") abline(h=calib,col="red") abline(h=actual,col="blue") grid()

by marvy 11y ago

kgwgk 11y ago

Assume that you're sampling from a normal distribution with known standard deviation sigma (1 for simplicity) and unknown mean mu. To test if the mean is larger than (the null hypothesis) mu=0 you can check if the observed value is larger than 1.64 sigma (for the 95% confidence test). So if your observation is larger than 1.64 you reject the null hypothesis. Your calculation would be correct only if the assumption "no false negatives" is approximately valid. This is the case when the true value is large in terms of sigma (say mu=6). Then for the 100 cases with mu=0 you'll reject the null 5 times on average, and for each one of the 100 cases with mu=100 you will reject the null (unless you're unlucky: there will be a false negative around once in 150000 trials). But you're conditioning on p<0.05, not on p~0.05. It's easy to see that it's much easier to get p=0.05 if mu=0 (this is a 1.64 sigma event) than if mu=6 (it's a 4.36 sigma event). If mu=0, you will get on average 1 (out of 100) observation with 0.04<p<0.05 (i.e. 1.64<x<1.75). The probability of obtaining an observation on that range when mu=6 is very small (0.0004 out of 100). Almost 100% of the "discoveries" with p~0.05 will be false (when mu=6 you will get p-values around 1e-9). When the true value of mu gets closer to 0, you cannot ignore the false negatives. For example if mu=0.1 the rejection rate will be quite similar to the mu=0 case (the probability of getting 0.4<p<0.5 is 1.2% and 1% respectively) and almost 50% of the "discoveries" with p~0.05 will be false. Somewhere between the two extreme cases, there is a lower bound for this "false discovery rate". See http://faculty.washington.edu/jonno/SISG-2011/lectures/sellkeetal01.pdf http://faculty.washington.edu/jonno/SISG-2011/lectures/sellk... and in particular figure 2.

marvy 11y ago
First of all, thank you for responding, I expected that no one would. Second of all, you point out that "no false negatives" is not always realistic. Fair enough. That means that 4.8% need not be the right answer. But at least it's clear how I got it. I still have no clue how on earth they got 29%, rather than 28% or 31%. Am I missing something?
- kgwgk 11y ago
  1/(1-1/(e p log(p)))) This is formula (3) in the linked paper. I agree that the derivation is not obvious, but probably you should get the same result using the calculation I described for different values of mu and looking for the minimum.
  - marvy 11y ago
    thanks!
marvy 11y ago
I think I'm still missing something here. In particular, I'm still not getting 29% as a lower bound. I'm getting around 20%. If we compare mu=0 with mu=1.64, the probability density at x=1.64 is roughly 0.1 and 0.4, respectively, so the lower bound should be .1/(.1+.4)=1/5. No? Unless they were assuming something other than "two normal distributions with the same variance"?
- kgwgk 11y ago
  You're absolutely right! My example was mainly for illustration, I was not sure that it would give exactly the same lower bound (but I was indeed surprised that it's below the 29% in those papers, which I thought a "hard" bound). It seems the bound that you are calculating (that I have reproduced, R code below) was already published more than 50 years ago for this specific case of a normal distribution. See slide 10 in http://www.biostat.uzh.ch/teaching/master/previous/seminarbayes/LeonhardHeld_slides.pdf http://www.biostat.uzh.ch/teaching/master/previous/seminarba... I have not really read the paper of Sellke et al. entirely, but it seems that the "calibration" they propose is more general but it makes some assumptions about the distribution of the p-value and it's therefore approximative. p=0.05 null=0.5 c0=qnorm(1-p) x=seq(0,5,0.01) y=100*dnorm(c0)/(dnorm(c0)+dnorm(x-c0)) calib=100/(1-1/(exp(1)*p*log(p))) actual=min(y) plot(x,y,type="l",ylim=c(0,100),ylab="%null",xlab="mu1",bty="l",xaxs="i",yaxs="i") title(paste(null*100,"% null p = ",p,sep="")) legend("topleft",c(paste("Sellke, Bayarri, Berger (2001) =",format(calib,digits=3)), paste("Edwards, Lindman, Savage (1963) =",format(actual,digits=3))), lwd=2,col=c("red","blue"),bty="n") abline(h=calib,col="red") abline(h=actual,col="blue") grid()