4 ms·

.

mehrdadn 7y ago
> It means that if the null hypothesis were true, you expect your p-values to be a random variable contained inside a nice bell shaped normal. 0.005 is dead center so it's very likely but O.2 which seems very unprobable is actually only 1std further, it's well inside the bell curve. I'm completely lost here. How is 0.005 "dead center"? Are you assuming p = 0 is the center? Are there negative p-values I'm not seeing that somehow balance the positive ones? How can a random variable that's strictly between 0 and 1 even follow a bell curve?
- firebacon 7y ago
  Check out these links [0] [1] or google for "p value distribution" or "p curve" [0] http://www-ist.massey.ac.nz/dstirlin/CAST/CAST/HtestPValue/testPValue3.html http://www-ist.massey.ac.nz/dstirlin/CAST/CAST/HtestPValue/t... [1] https://en.wikipedia.org/wiki/P-value#Distribution https://en.wikipedia.org/wiki/P-value#Distribution
  - mehrdadn 7y ago
    I'm not following those links either. How is p uniformly distributed under H0? If you assume H0 then obtaining a p-value near 0 is going to be damn impossible. Whereas obtaining one similarly close to 0.5 is going to be ridiculously more likely. Am I severely lacking sleep and going crazy or something? Maybe I should check back in like half a day to see what people have said, I feel like I must be completely confused right now because literally nothing I've read so far makes sense to me.
    - dash2 7y ago
      They're right. If the null hypothesis is true, then the probability of getting any p value is equal. Put it another way, the p value is the probability of getting the observed data (or more extreme) under the null. So, under the null, 10% of the time you will get data with a p value of 10% or less; 20% of the time, you will get data with a p value of 20% or less; and so on. And that's the uniform distribution! Here's an R example to play with: pvals <- replicate(10000, { x <- rnorm(100) y <- rnorm(100) t.test(x, y)$p.value }) plot(density(pvals)) That will plot you a nice uniform line on [0, 1]. (NB: I have no idea why OP talked about p values following a normal distribution. That doesn't make sense to me, and I think the post has been deleted.)
      - mehrdadn 7y ago
        So I don't have (or know) R, but I do have access to Mathematica, and this is most definitely not giving me a uniform distribution (how could it?!): << HypothesisTesting` With[{n = 10000000, dist = NormalDistribution[]}, Histogram[Last[NormalPValue[RandomVariate[dist, n] - RandomVariate[dist, n]]], 500]] Why is your x variable though? If H0 is true shouldn't your x be fixed? I think if you remove the subtraction though then you do get a uniform distribution -- in which case I see what the claim is, yeah. Wasn't really clear to me earlier but indeed, getting p = 5% means you have a 5% chance of getting observations that extreme, so I guess it is uniformly distributed!
        dash2 7y ago
        You'd get the same result whether you test the hypothesis y > x (with both random variables) or y > 0. Under the null, the p-value is uniform on [0, 1], whatever that null is. It follows from the definition of a p value. (Disclaimer: I am not a real statistician....)
    - kgwgk 7y ago
      > How is p uniformly distributed under H0? That's the very definition of a p-value! The mapping of data to p-values is chosen to have a uniform distribution of p-values when the data is distributed according to the null hypothesis. That's the property that makes p-values interesting.
kmm 7y ago
At least for continuous statistics, the p-value is uniformly distributed when the null hypothesis is true.
mruts 7y ago
I don’t think you understand what a p-value is. The p-value is a percentage output of testing whether a given normal distribution actually has a non-zero mean. It is phrased in terms of the null hypothesis. So a P < 0.05 means that there is less than a 5% chance that the null hypothesis is true. Conversely, this means that there is a 95% chance that what you’re testing is true.
- firebacon 7y ago
  > So a P < 0.05 means that there is less than a 5% chance that the null hypothesis is true. Conversely, this means that there is a 95% chance that what you’re testing is true. See https://en.wikipedia.org/wiki/P-value#Basic%20concepts https://en.wikipedia.org/wiki/P-value#Basic%20concepts
- lonelappde 7y ago
  mruts' (and most people who studied a little statistics) misunderstanding of the basic definition is exactly what author and others have been campaigning against for years.
- sweeneyrod 7y ago
  You don't understand what a p-value is. > a P < 0.05 means that there is less than a 5% chance that the null hypothesis is true. In other words, P(H0 | X) where H0 is the null hypothesis being true and X is the data observed. But that is not what a p-value is, they actually represent P(X | H0).

> It means that if the null hypothesis were true, you expect your p-values to be a random variable contained inside a nice bell shaped normal. 0.005 is dead center so it's very likely but O.2 which seems very unprobable is actually only 1std further, it's well inside the bell curve. I'm completely lost here. How is 0.005 "dead center"? Are you assuming p = 0 is the center? Are there negative p-values I'm not seeing that somehow balance the positive ones? How can a random variable that's strictly between 0 and 1 even follow a bell curve?

firebacon 7y ago
Check out these links [0] [1] or google for "p value distribution" or "p curve" [0] http://www-ist.massey.ac.nz/dstirlin/CAST/CAST/HtestPValue/testPValue3.html http://www-ist.massey.ac.nz/dstirlin/CAST/CAST/HtestPValue/t... [1] https://en.wikipedia.org/wiki/P-value#Distribution https://en.wikipedia.org/wiki/P-value#Distribution
- mehrdadn 7y ago
  I'm not following those links either. How is p uniformly distributed under H0? If you assume H0 then obtaining a p-value near 0 is going to be damn impossible. Whereas obtaining one similarly close to 0.5 is going to be ridiculously more likely. Am I severely lacking sleep and going crazy or something? Maybe I should check back in like half a day to see what people have said, I feel like I must be completely confused right now because literally nothing I've read so far makes sense to me.
  - dash2 7y ago
    They're right. If the null hypothesis is true, then the probability of getting any p value is equal. Put it another way, the p value is the probability of getting the observed data (or more extreme) under the null. So, under the null, 10% of the time you will get data with a p value of 10% or less; 20% of the time, you will get data with a p value of 20% or less; and so on. And that's the uniform distribution! Here's an R example to play with: pvals <- replicate(10000, { x <- rnorm(100) y <- rnorm(100) t.test(x, y)$p.value }) plot(density(pvals)) That will plot you a nice uniform line on [0, 1]. (NB: I have no idea why OP talked about p values following a normal distribution. That doesn't make sense to me, and I think the post has been deleted.)
    - mehrdadn 7y ago
      So I don't have (or know) R, but I do have access to Mathematica, and this is most definitely not giving me a uniform distribution (how could it?!): << HypothesisTesting` With[{n = 10000000, dist = NormalDistribution[]}, Histogram[Last[NormalPValue[RandomVariate[dist, n] - RandomVariate[dist, n]]], 500]] Why is your x variable though? If H0 is true shouldn't your x be fixed? I think if you remove the subtraction though then you do get a uniform distribution -- in which case I see what the claim is, yeah. Wasn't really clear to me earlier but indeed, getting p = 5% means you have a 5% chance of getting observations that extreme, so I guess it is uniformly distributed!
      - dash2 7y ago
        You'd get the same result whether you test the hypothesis y > x (with both random variables) or y > 0. Under the null, the p-value is uniform on [0, 1], whatever that null is. It follows from the definition of a p value. (Disclaimer: I am not a real statistician....)
  - kgwgk 7y ago
    > How is p uniformly distributed under H0? That's the very definition of a p-value! The mapping of data to p-values is chosen to have a uniform distribution of p-values when the data is distributed according to the null hypothesis. That's the property that makes p-values interesting.

kmm 7y ago

At least for continuous statistics, the p-value is uniformly distributed when the null hypothesis is true.

mruts 7y ago

I don’t think you understand what a p-value is. The p-value is a percentage output of testing whether a given normal distribution actually has a non-zero mean. It is phrased in terms of the null hypothesis. So a P < 0.05 means that there is less than a 5% chance that the null hypothesis is true. Conversely, this means that there is a 95% chance that what you’re testing is true.

firebacon 7y ago
> So a P < 0.05 means that there is less than a 5% chance that the null hypothesis is true. Conversely, this means that there is a 95% chance that what you’re testing is true. See https://en.wikipedia.org/wiki/P-value#Basic%20concepts https://en.wikipedia.org/wiki/P-value#Basic%20concepts
lonelappde 7y ago
mruts' (and most people who studied a little statistics) misunderstanding of the basic definition is exactly what author and others have been campaigning against for years.
sweeneyrod 7y ago
You don't understand what a p-value is. > a P < 0.05 means that there is less than a 5% chance that the null hypothesis is true. In other words, P(H0 | X) where H0 is the null hypothesis being true and X is the data observed. But that is not what a p-value is, they actually represent P(X | H0).