4 ms·
> P-values are quite intuitive. Do you agree now that if the null hypothesis is true the p-value is uniformly distributed between 0 and 1? Or do you still thin
by kgwgk 11y ago
> P-values are quite intuitive.
Do you agree now that if the null hypothesis is true the p-value is uniformly distributed between 0 and 1? Or do you still think that "if that were true, p-value would be entirely useless"?
In any case, you were completely wrong about p-values a few months ago; maybe they are not so intuitive after all.
https://news.ycombinator.com/item?id=10156510 https://news.ycombinator.com/item?id=10156510
- jsprogrammer 11y agoWhere was I wrong? >Do you agree now that if the null hypothesis is true the p-value is uniformly distributed between 0 and 1? If the null hypothesis is true, the p-value will converge to 1. (This makes complete intuitive sense as well, if you hypothesis is true, every observation you make should/will agree with it, making the ratio of agreeable_observations:total_observations = 1.) You haven't shown anything where a true null hypothesis uniformly generates p-values between 0 and 1. Perhaps for a single observation you can only get a 0 or 1 value, and, perhaps for a over/under the average test (where every experiment will give 0 or 1) your sequence of p-values for each observation would be uniformly distributed, but p-values are generally computed as a normalized sum over many observations, in which case the value should converge to 1 if the null hypothesis is true. One unintuitive aspect is that you might expect p-value to converge to 0 if the null hypothesis is false, however p-value is undefined when the null hypothesis is false.
- kgwgk 11y agoOk, I see you still have your very own definition of "p-value". I just thought it was useful to make clear to the audience that your "p-values" are not the same "p-values" being discussed here. Edit: Probably you don't care, but for the record: "Since the value of x that defines the left tail or right tail event is a random variable, this makes the p-value a function of x and a random variable in itself defined uniformly over [0,1] interval, assuming x is continuous." [ https://en.wikipedia.org/wiki/P-value https://en.wikipedia.org/wiki/P-value ] "In statistics, when a p-value is used as a test statistic for a simple null hypothesis, and the distribution of the test statistic is continuous, then the p-value is uniformly distributed between 0 and 1 if the null hypothesis is true." [ https://en.wikipedia.org/wiki/Uniform_distribution_(continuous) https://en.wikipedia.org/wiki/Uniform_distribution_(continuo... ]
- jsprogrammer 11y agoMaybe you should pull out your full derivation. Random Wikipedia text doesn't count (particularly when it is just bald assertion, as in this case). Most null hypotheses hypothesize a normally distributed measurement error. For any given distribution (continuous uniform), of course, the p-value means something different. That is what I have been arguing this whole time.
- kgwgk 11y agoYou complained I didn't show anything, so there you have something. Do you have anything, even if it is a bald random Wikipedia assertion, supporting your position? (Rhetorical question, I couldn't care less.) Here is a full derivation from a random blog, which I guess doesn't count either: https://shihho.wordpress.com/2012/11/27/pvalue_distribution/ https://shihho.wordpress.com/2012/11/27/pvalue_distribution/ Another one: https://joyeuserrance.wordpress.com/2011/04/22/proof-that-p-values-under-the-null-are-uniformly-distributed/ https://joyeuserrance.wordpress.com/2011/04/22/proof-that-p-...
- jsprogrammer 11y agoYou seem awfully invested in a discussion that you don't care about. >https://shihho.wordpress.com/2012/11/27/pvalue_distribution/ https://shihho.wordpress.com/2012/11/27/pvalue_distribution/ This treatment commits the same error as the other poster (evanpw; I believe nonbel commits the same). The image at the bottom is only showing p-values for single observations. That is near useless (as the chart shows). An actual experiment requires repeated observation and analysis, not one-off observations. npval <- pnorm( rnorm(nSim, mean, std.dev), mean, std.dev ) hist(npval, breaks=40, xlab=xlabel, main=title, col='gray') R is probably the worst choice of language here, but my understanding of `pnorm` is that, when passed a list of numbers as the first argument (as `rnorm` returns), it will return a list of p-values (assuming normal distribution with given parameters), where the index in the list corresponds to the p-value, taking into account only that single observation (ie. it ignores all other observations in the list [ie. not how p-value hypothesis testing works {at least, in theory; real world practices may differ}]). Your second link appears to be the same argument (at least, they appear to use the same equations). The charts are a nice visualization of how you can get a uniform distribution from a normal one (or vice versa) though.
- nonbel 11y ago>"You haven't shown anything where a true null hypothesis uniformly generates p-values between 0 and 1." Try this in R. It calculates p-values for testing the hypothesis that two groups (n=10) come from the same distribution (normal with mean=0, sd=1). It does 100,000 such comparisons, then makes a histogram. This results in approximately a uniform(0,1) distribution of p-values: hist(replicate(10^5,t.test(rnorm(10),rnorm(10),var.equal=T)$p.value)) If you increase to 10^6, 10^7, etc it will converge upon the uniform distribution. That is what people mean. Edit: Or something like this. Keep adding to the sample, the p-value will random walk between 0 and 1. The mean will converge onto 0.5: n=10; Nsim=10000 a=rnorm(n); b=rnorm(n) p=matrix(nrow=Nsim,ncol=2) for(i in 1:Nsim){ p[i,1]=length(a) p[i,2]=t.test(a,b,var.equal=T)$p.value a=c(a,rnorm(n)); b=c(b,rnorm(n)) plot(p[,1],p[,2],ylim=c(0,1), ylab="pvalue", xlab="sample size") }