6 ms·
> To say it again: it is completely consistent with the null hypothesis to see p-values of 0.2 and 0.005 from two replications of the same damn experiment. I d
by mehrdadn 7y ago
> To say it again: it is completely consistent with the null hypothesis to see p-values of 0.2 and 0.005 from two replications of the same damn experiment.
I don't really follow this. Could someone clarify what is meant here? At what point would this author say something is not consistent with the null hypothesis?
- pacbard 7y agoHe is coming at that conclusion from a Bayesian point of view to statistics. He is seeing the p-value as a random variable that can take values from 0 to 1 and follows some distribution. Under these hypotheses, observing a p-value of 0.20 and 0.005 is completely reasonable even if unlikely. Those are just two draws from a random variable. Edit. Under Bayesian statistics testing the null hypothesis is a moot point as it becomes possible to directly model the distribution of the possible effects. Thinking of it as being able to look at a picture of something (the p-value) vs looking at a movie of it (the distribution of the effects).
- pfortuny 7y agoWhat he says is (I gather) worse: those events are only separated be 1.1std deviations, which is little.
- mehrdadn 7y agoI think that's what he's saying too, but what is that supposed to show? Is he arguing against some claim that every interval of 1 standard deviation is equally significant? Did anybody make this claim? So far as I know, nobody considers (say) a 6-sigma effect to be 6 times stronger than a 1-sigma effect...
- kgwgk 7y agoI don’t get it either. It’s a trivial consequence of having a threshold: if we say two cities are “far” when they are at least 1000 miles away then Washington D.C. is not far from Jacksonville while Boston is far from Jacksonville, even though Boston is not far from Washington.
- nerdponx 7y agoThat's not quite the problem. Let's assume you've already decided in advance what "far" means. Without moving either city from its current location, the same experiment can give you "very far" and "very close" in identical replications.
- kgwgk 7y agoI’m commenting on pfortuny’s (correct) interpretation “those events are only separated be 1.1std deviations, which is little” of the original post by Gelman (the difference between a significant result and a non-significant result may not be significant = a city which is far from Jacksonville and a city which is not far from Jacksonville may not be far from each other).
- mehrdadn 7y agoI still don't get it. What p-value observations would not be "reasonable" here? It seems to me he's saying that anything between 0 and 1 is completely reasonable, which is a completely pointless statement as I see it.
- lonelappde 7y agoHis point is that the whole point of the field of statistics is that cherry picking results or getting lucky once is not proof of anything, unlike nonstatistical mathematics where one example is enough to justify a claim.
- JustFinishedBSG 7y agoThat's not even a bayesian point of view, even in a frequentist setting the p-value is a conditional probability, conditioned on the dataset.
- kgwgk 7y ago> He is seeing the p-value as a random variable that can take values from 0 to 1 and follows some distribution. That is the frequentist point of view!
- JustFinishedBSG 7y ago.
- mehrdadn 7y ago> It means that if the null hypothesis were true, you expect your p-values to be a random variable contained inside a nice bell shaped normal. 0.005 is dead center so it's very likely but O.2 which seems very unprobable is actually only 1std further, it's well inside the bell curve. I'm completely lost here. How is 0.005 "dead center"? Are you assuming p = 0 is the center? Are there negative p-values I'm not seeing that somehow balance the positive ones? How can a random variable that's strictly between 0 and 1 even follow a bell curve?
- firebacon 7y agoCheck out these links [0] [1] or google for "p value distribution" or "p curve" [0] http://www-ist.massey.ac.nz/dstirlin/CAST/CAST/HtestPValue/testPValue3.html http://www-ist.massey.ac.nz/dstirlin/CAST/CAST/HtestPValue/t... [1] https://en.wikipedia.org/wiki/P-value#Distribution https://en.wikipedia.org/wiki/P-value#Distribution
- mehrdadn 7y agoI'm not following those links either. How is p uniformly distributed under H0? If you assume H0 then obtaining a p-value near 0 is going to be damn impossible. Whereas obtaining one similarly close to 0.5 is going to be ridiculously more likely. Am I severely lacking sleep and going crazy or something? Maybe I should check back in like half a day to see what people have said, I feel like I must be completely confused right now because literally nothing I've read so far makes sense to me.
- dash2 7y agoThey're right. If the null hypothesis is true, then the probability of getting any p value is equal. Put it another way, the p value is the probability of getting the observed data (or more extreme) under the null. So, under the null, 10% of the time you will get data with a p value of 10% or less; 20% of the time, you will get data with a p value of 20% or less; and so on. And that's the uniform distribution! Here's an R example to play with: pvals <- replicate(10000, { x <- rnorm(100) y <- rnorm(100) t.test(x, y)$p.value }) plot(density(pvals)) That will plot you a nice uniform line on [0, 1]. (NB: I have no idea why OP talked about p values following a normal distribution. That doesn't make sense to me, and I think the post has been deleted.)
- krona 7y agoWell, if there is no effect (the effect-size is zero), two different experimenters will likely see two different p-values regardless of how large an experiment either of them runs.
- mehrdadn 7y agoSo by this logic "it is completely consistent with the null hypothesis to see p-values of 0.00000001 and 0.99999999 from two replications of the same damn experiment"? At which point, what is even the point of this statement?
- lonelappde 7y agoThe point is that you need to replicate experiments and not cherry pick the one with the highest p-value.
- penagwin 7y agoMy understanding (Someone please correct me if I'm wrong) is that while the values can range from limx->0 and limx->1 , the actual distribution will be skewed in cases where it is statistically significant. E.g. If you repeat the experiment 100x you could find a p=.999999 but more likely they'll be close to .05 (or whatever alpha you choose)
- kgwgk 7y agoIf the null hypothesis is true the p-value is distributed uniformly in [0 1] (at least ignoring discrete data, composite hypothesis and other cases which are not easy). The distribution will be different when then true hypothesis is not true (but you may also get non-significant results even if the null hypothesis is not true). I’m not sure if that’s what you mean by “the actual distribution will be skewed in cases where it is statistically significant.”
- penagwin 7y agoSorry I was definitely kinda vague - I meant when the alternative hypothesis is true the distribution should be skewed towards 0.
- paulddraper 7y agoI have no idea. For a identical normal populations, repeating an experiment produces p<=0.2 20% of the time, and produces p<=0.005 0.5% of the time. A coin comes up on the same side 3 times in a row vs 8 times in a row...I have no idea why we should shrug and consider the plausibility of coin bias in these two cases about the same. --- EDIT: I he means that in the case of an actual difference in the populations, p=0.2 and p=0.005 are both pretty likely outcomes. When the populations are the same, p=0.2 and p=0.005 are quite different happenings. This is because p-value methods doesn't worry very much about type II errors.
- probabils 7y agoNasim Taleb wrote extensively on this. > P-values are shown to be extremely skewed and volatile, regardless of the sample size n, and vary greatly across repetitions of exactly same protocols under identical stochastic copies of the phenomenon; such volatility makes the minimum p value diverge significantly from the "true" one. Setting the power is shown to offer little remedy unless sample size is increased markedly or the p-value is lowered by at least one order of magnitude. https://arxiv.org/abs/1603.07532 https://arxiv.org/abs/1603.07532 Video: https://www.youtube.com/watch?v=8qrfSh07rT0 https://www.youtube.com/watch?v=8qrfSh07rT0
- mehrdadn 7y agoAre you sure this answers my question? Note that I was neither asking why p-hacking is bad nor even why p-values are bad.
- probabils 7y agoYes, it answers your question. What you call "p-value" is a sample from the "p-value distribution" of your experiment. Taleb shows you can sample a p-value of 0.05 when the actual "true" p-value is 0.12.
- mehrdadn 7y agoI guess I have to sleep on this because at a quick glance I can't really make sense of how it answers my question. I do find it ironic though that this is so difficult to explain that I apparently have to read a paper to understand it... I would've thought the blog post was trying to explain things in simple terms...
- probabils 7y agoWell, if this stuff was easy to understand, we wouldn't be having a crisis because of it, would we? Probability is highly non-intuitive (kind of like quantum mechanics), so most people (including scientists) don't understand it and just memorize "protocols" and "formulas" and "p-value stuff". https://www.tandfonline.com/doi/full/10.1080/00031305.2018.1518264 https://www.tandfonline.com/doi/full/10.1080/00031305.2018.1...
- bonoboTP 7y agoThe distribution of the p-value, given that the null is true, is uniform between 0 and 1. IF the null is true, you're equally likely to get a p-value of 0.01 and 0.87.
- bart_spoon 7y agoIts a meta observation of how we view p-values. He's saying that the same logic behind p-values and declaring something "statistically significant" when p < X (often 0.05), falls apart when applied to differences between two observed p-values. Essentially, you would say there is no "statistically significant" difference between an experiment that resulted in a p-value of 0.2 and one that resulted in a p-value of 0.05. And the argument will hold no matter what threshold you choose for rejecting the null hypothesis. You can choose to reject if p > X, and for any X, there will be values greater than X that, applying this meta-logic, are not statistically different from X. > At what point would this author say something is not consistent with the null hypothesis? Gelman's argument, I presume, is against the idea of significance testing as a whole. Declaring something "statistically significant" is in itself a very problematic thing, as it distills the entire phenomenon, the uncertainty surrounding the experiment, and the uncertainty surrounding the researcher's decisions to a single, binary conclusion. Gelman is a Bayesian (perhaps the most famous modern Bayesian), and the Bayesian philosophy is to focus on producing a posterior distribution of the phenomenon being studied. I presume the alternative to significance and null hypothesis testing that he was suggest would be something closer to a model where people are reporting their priors/data/posteriors, and the discussion focuses around the implications and replication of those.
- madhadron 7y agoSay you run an experiment. You get some data. From that data you calculate a p-value. The data has randomness, so the p-value is a random variable and has a distribution. By chance you can get results that look significant. By chance you can get results that look insignificant.