12 ms·

Not Even Scientists Can Easily Explain P-values

data_hope 11y ago
In my experience, people in science who are not statisticians have not completely understood the p value. Typically people are clueless on p-Values and the statistical power and how they relate.
- MollyR 11y ago
  This is scarily true. When my lab (bioinformatics) would work/collaborate with research biologists and their labs, the research biologists lack of statistics would become very clear. Especially when we did data mining on their qpcr experiments, we constantly have to explain things like bonferroni corrections. Edit: This is just my experience at one university with 6-7 biological research labs. I regret using the generalization, apologies.
peatmoss 11y ago
Lest anyone skimming the headline misses it, the point here isn't that these scientists couldn't correctly explain a p-value. The scientists here all could. It's that they couldn't explain it in a way that made intuitive sense to anyone. So, rather than being a dig at scientists, I think the points here are: 1. p-values are intrinsically none-too-intuitive. 2. What p-values tell us isn't necessarily that interesting.
- Alex3917 11y ago
  > What p-values tell us isn't necessarily that interesting. The problem is that the concept sounds more important than it is if you don't understand it. This is a bad failure state. The issue is that the probability of some phenomena happening by chance is mostly orthogonal to the probability of that phenomena being true or whatever, but this seems outside the ability of most people to grasp.
  - peatmoss 11y ago
    I can also say that, when I was starting stats, the term p-value seemed to hint tantalizingly at "probability." I cannot be the only person seduced into erroneously thinking that a p-value had something to do with the probability that something could be untrue.
    - lmm 11y ago
      Well a p-value is a probability, so I don't think that's exactly misleading.
- jsprogrammer 11y ago
  > 1. p-values are intrinsically none-too-intuitive P-values are quite intuitive. They are the ratio of observations made which agree with the null hypothesis to the total number of observations made. Edit: If you are going to downmod, at least point out the error you perceive.
  - nickpsecurity 11y ago
    That was as intuitive as the seemless connection between the Magic Bullet Theory and Ballistics.
    - jsprogrammer 11y ago
      It comes directly from the calculation of p-values through probability theory.
      - omginternets 11y ago
        I'm reminded of a joke about how Haskell empowers you to apply the infinite power of abstract mathematics using the intuitive simplicity of abstract mathematics. I think the resistance you're encountering stems from the fact that you're appealing to formality rather than intuition.
        senekerim 11y ago
        Also from the fact that he doesn't know what he's talking about.
        jsprogrammer 11y ago
        You talking about me?
        jsprogrammer 11y ago
        Counting positive observations vs. total observations is a formality rather than intuition? The formality is the axioms of probability. The intuition is that you make a theory and measure how many predictions it gets right.
- cousin_it 11y ago
  Especially when publication bias comes into play: https://xkcd.com/882/ https://xkcd.com/882/
  - KingMob 11y ago
    Sadly true. On the one hand, the scientists ignored the multiple comparisons problem, and should have applied something like the Bonferroni or False Discovery Rate corrections. OTOH, they got publications and publicity for ignoring it.
- vlehto 11y ago
  >2. What p-values tell us isn't necessarily that interesting. High p-value is the first thing to look at a study. If it's high, then you can simply assume the findings are unreliable. But if it's low, it doesn't yet mean anything. The p-values allow you to discard worst mumbojumbo quickly. That might not be interesting, but it's valuable. The thing to remember is that "we don't know" =/ "untrue".
- blahblah3 11y ago
  I would argue that the p-value way of reasoning (probability of something happening given a hypothesis) corresponds to an intuitive way many people reason about probability. For example, suppose the police are investigating the robbery of a home. They find a stranger's (call him person S) fingerprints all over the house. If S claimed he was innocent, a questioner might ask: "If you're really innocent, then why were your fingerprints all over the house?" This is another way of stating that P(Data | innocent) is small (similar to the p-value concept). Since we often don't have a strong prior reason for believing a given individual is innocent or guilty, the p-value heuristic can be reasonable. The formal connection to P(innocent | Data) is given by Bayes' theorem
- btilly 11y ago
  There is a third point. Which is that scientists consistently agree with clearly incorrect statements about p-values. That means that knowing the correct definition is no guarantee that you're actually using it correctly. (Odds are you aren't.)
- vectorjohn 11y ago
  I've come to think that if someone can't explain something well, they probably don't understand it as well as they think. I've heard people that fancy themselves experts on this or that software concept, and when trying to explain it use all the technical jargon they possibly can. It comes off as just regurgitating things they've heard before. In other words, I suspect the scientists that couldn't explain p-values in an intuitive way, didn't themselves fully grasp the subtleties of p-values.
graffitici 11y ago
From what I understand, p-values are typically used by frequentist method for verifying that the "prediction" seems correct. On the other hand, the Bayesian approach is to split the data into training and test sets, and verify how much of it holds. Does this make sense? If the p-values are not very good at conveying "confidence in the prediction," is this another argument in favor of a more Bayesian approach to statistics? Any thoughts would be appreciated!
- gbrown 11y ago
  Well, neither your description of p-values, nor your understanding of Bayesian inference is correct, so... no?
  - TeMPOraL 11y ago
    It would be helpful to tell how the commenter is wrong, and what is the correct description.
    - gbrown 11y ago
      It would also be helpful if they'd read the article.
      - TeMPOraL 11y ago
        I read the article and it's almost completely orthogonal to the questions asked by 'graffitici. There's nothing suggesting that they didn't read the article, so now I assert that you have three things to explain - where they're wrong, what's the right answer to their question and why on Earth did you think they didn't read the article?
        gbrown 11y ago
        From the article: "We want to know if results are right, but a p-value doesn’t measure that. It can’t tell you the magnitude of an effect, the strength of the evidence or the probability that the finding was the result of chance." From the first sentence of the parent comment: "From what I understand, p-values are typically used by frequentist method for verifying that the "prediction" seems correct." If my reply seemed glib, it was because I thought the question was so far off base, that graffitici almost certainly hadn't read the article. To relate to the topic under discussion, the distribution of possible comments produced by people who have carefully read the article is extremely unlikely to result in a post as extremely misinformed as the one above. If there was a simple statistical misunderstanding, perhaps that would be worth clearing up. Instead, the parent comment used some statistical words, but was very nearly incoherent. Perhaps I shouldn't have said anything if I wasn't willing to write a protracted essay about the differences between frequentist and Bayesian inference.
goodside 11y ago
My go-to phrase: "Even if there were no real difference, we would see a result like this X% of the time anyway."
- llasram 11y ago
  I agree that makes intuitive sense when applied to explicitly constructed experiments we could attempt to exactly repeat, but the intuition begins to break down in more complicated circumstances, such as retrospective or longitutinal studies. Saying something would happen "X% of the time anyway" becomes a philosophical issue.
  - omginternets 11y ago
    Yes, well, we could talk about frequentist vs bayesian statistics, but that's hardly the point of the article. That is what a p-value means, feasibility aside. There are also more intuitive ways to think about it, "false-positive rate" being one.
- omginternets 11y ago
  Mine is remarkably similar, though less general: "if p=.05, then running a bogus experiment 20 times ensures you'll publish." Suddenly, the abysmal state of nutritional epidemiology makes sense. How many labs do you suspect are running "red meat = death" studies?
  - TeMPOraL 11y ago
    One is reminded of an XKCD classic - https://xkcd.com/882/ https://xkcd.com/882/. Sadly, it only goes to point out the sad state of contemporary science. Especially in terms of nutrition, the only way to stay sane is to stick to the good old street-fighting Bayes - if popularly consumed product X was leading to cancer in any way relevant to your life, you'd see people dropping like flies all over the place.
    - omginternets 11y ago
jowiar 11y ago
That it's called a "p-value" is rather indicative of it's nebulous nature. For something that is used in a rather applied context, giving it a one-letter name that has zero context is a bit of a cop-out.
- jsprogrammer 11y ago
  I am sure it comes from probability theory, where it is common to use the notation P(x) to mean, "the probability of observing `x`".
- GFK_of_xmaspast 11y ago
  That's how basically everything in mathematics works tho.
elbigbad 11y ago
P-Values can be pretty unintuitive to people unfamiliar, but can readily be explained I think. I do think that this book did an excellent job of explaining it: What Is a P-Value Anyway? http://www.pearsonhighered.com/vickers/ http://www.pearsonhighered.com/vickers/ Excerpt (starts at p.57 in the link above): [I]f you do nothing else, please try to remember the following sentence: “the p-value is the probability that the data would be at least as extreme as those observed, if the null hypothesis were true.” Though I’d prefer that you also understood it—about which, teeth brushing. I have three young children. In the evening, before we get to bedtime stories (bedtime stories being a nice way to end the day), we have to persuade them all to bathe, use the toilet, clean their teeth, change into pajamas, get their clothes ready for the next day and then actually get into bed (the persuading part being a nice way to go crazy). My five-year-old can often be found sitting on his bed, fully dressed, claiming to have clean teeth. The give-away is the bone dry toothbrush: he says that he has brushed his teeth, I tell him that he couldn’t have. My reasoning here goes like this: the toothbrush is dry; it is unlikely that the toothbrush would be dry if my son had cleaned his teeth; therefore he hasn’t cleaned his teeth. Or using statistician-speak: here are the data (a dry toothbrush); here is a hypothesis (my son has cleaned his teeth); the data would be unusual if the hypothesis were true, therefore we should reject the hypothesis. [...] So here is what to parrot when we run into each other at a bar and I still haven’t managed to work out any new party tricks: “The p-value is the probability that the data would be at least as extreme as those observed, if the null hypothesis were true.” When I recover from shock, you can explain it to me in terms of a toothbrush (“The probability of the toothbrush being dry if you’ve just cleaned your teeth”).
- nonbel 11y ago
  >"My reasoning here goes like this: the toothbrush is dry; it is unlikely that the toothbrush would be dry if my son had cleaned his teeth; therefore he hasn’t cleaned his teeth." I don't think this works so well because you are jumping from a qualitative statement (dry or not dry) to a quantitative ("at least as extreme").
- sbov 11y ago
  This is way too much for the layman and full of mathematical language. What is null hypothesis? What is "at least as extreme as"? What is probability? Nevermind that the null hypothesis forces us to think in double negatives, which isn't the most intuitive. It also isn't clear to me how you would apply your example given how most p-values are used. I read an article that says exercising makes you richer with a 95% confidence. So, as a layman, if I attempt to apply your example, it leads me directly into a common p-value mistake: the probability of me becoming richer if I exercise. 95%, right? Wrong!
jroitgrund 11y ago
The coin example makes perfect sense to me and I had no idea what a p-value was before I read this article. Here's an intuitive explanation: a p-value is the probability of getting your experimental results given that your hypothesis is wrong.
- bweitzman 11y ago
  I think the coin example makes sense but is not a good example in this case. You will never be able to determine the validity of a coin just by flipping it. Any sequence of flips is technically valid but possibly just very rare for a fair coin. The p-value won't tell you if the coin is fair or not, but it can tell you the probability that the coin is fair.
  - skybrian 11y ago
    If the sequence is rare enough then we do, in practice, conclude that the coin is unfair. If you can't make that conclusion then science doesn't work; no observations will convince you.
    - bweitzman 11y ago
      Of course. If you flip the coin 100 times and get all heads than you are safe to call the coin unfair, even though you would expect to see that result (1/2)^100 and thus would be wrong once in a while. I think that's sort of the point that the article is making actually. That high probability does not imply truth. There are other non probabilistic ways to verify that coin is unfair, for example by looking at the density throughout the coin.
      - TeMPOraL 11y ago
        Observations do not imply truth. 0 and 1 are not probabilities, and in real life you can't prove something using observations in a logical sense. Real world runs on probabilities, not boolean logic. Even if you look at the density of the metal throughout the coin, there's still a chance I've altered your device to report the coin is fair. Or a passing microsingularity decided to play games with the scanning beam. Or you're just imagining the whole thing. That's not to say one should despair that the world is unknowable. One only has to get used to the fact that, in practice, "true" just means "extremely, extremely likely".
blahblah3 11y ago
The point of p-values is to limit false positive rates. If you only accept results with p-value <= 0.05, your false positive rate is at most 5%. Of course, if you had apriori knowledge that everything was a "null result" , then 100% of the results you accept would be false positives. By Bayes' theorem, P(H|D) = P(D|H) * P(H) / P(D) . Therefore, given a fixed "prior probability" of a hypothesis P(H), a lower p-value (P(D|H)) implies a lower probability of the null being true. I think this relationship leads to a lot of confusion when thinking of p-values.
- jcranmer 11y ago
  You are incorrect. False positive rates in practice appear to be about 20-50%, even with a p-value of 0.05.
  - blahblah3 11y ago
    I was describing the theory behind the p-value. In practice, people slice and dice the dataset in multiple ways and try several specifications which increases the false positive rate (see the multiple comparisons problem)
  - nonbel 11y ago
    The best estimate right now is 20-50% replication rate for fields that rely on p less than 0.05 to judge results. So I think you meant 50-80% false positive rate, although these are not equivalent (ie is the null hypothesis of no difference ever really exactly true...): >"Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; [...] In cell biology, two industrial laboratories reported success replicating the original results of landmark findings in only 11 and 25% of the attempted cases" http://www.ncbi.nlm.nih.gov/pubmed/26315443 http://www.ncbi.nlm.nih.gov/pubmed/26315443 Edit: For comparison, Dr. Oz was accused of fraud and called in front of congress for only being wrong half the time: https://www.washingtonpost.com/news/morning-mix/wp/2014/12/19/half-of-dr-ozs-medical-advice-is-baseless-or-wrong-study-says/ https://www.washingtonpost.com/news/morning-mix/wp/2014/12/1...
nickpsecurity 11y ago
My takeaway is to not depend on or use them wherever possible. If I do use them, I get some heuristics from a survey of a bunch of statistics experts claiming to know what it is. And I'll be sure to keep a minimal, survey size of 30. ;)
jforman 11y ago
Having been both a scientist and, now, a product manager for a medical device, I think science would benefit from the work around verification vs. validation: https://en.wikipedia.org/wiki/Verification_and_validation https://en.wikipedia.org/wiki/Verification_and_validation A p-value is a verification tool and nothing more, and yet scientists all too often take the "hunt for the p-value" as the only goal. Rather than fish through data for a p-value that ends up defining the scientific narrative in a paper, p-values need to be integrated into a much larger validation process driven by pre-defined scientific goals. It's no wonder that so many results are found to be specious given today's scientific culture.
Jach 11y ago
P-values are only hard to explain when you try to explain in English. Stick to the math, where it's just P(X >= x | Ho) and once you understand the math it's immediately clear in a way an English sentence (or paragraph trying to equate to the math without math) isn't. My go-to presentation on this is: http://www.biostat.jhsph.edu/~cfrangak/cominte/goodmanvalues.pdf http://www.biostat.jhsph.edu/~cfrangak/cominte/goodmanvalues...
omginternets 11y ago
From my limited teaching experience, renaming "p-value" to "probability of a false positive" makes it all very simple.
- nonbel 11y ago
  This is a very common, but dangerously wrong, interpretation of p-values. Others in this thread have expressed the same misconception, see the paper linked here: https://news.ycombinator.com/item?id=10628860 https://news.ycombinator.com/item?id=10628860
  - omginternets 11y ago
    Duly noted, thanks for the reference!
TallGuyShort 11y ago
This is an excellent talk I attended a while ago that explains it very well from a programmer's perspective: https://www.youtube.com/watch?v=5Dnw46eC-0o&list=PL055Epbe6d5Y8_iZPo7pH3hOnAtchMCJt&index=8 https://www.youtube.com/watch?v=5Dnw46eC-0o&list=PL055Epbe6d...
cafebeen 11y ago
This is a surprisingly meta article... It claims there's a general lack of statistical understanding among experts, and then supports itself by making sweeping generalizations from a tiny sample size!
nonbel 11y ago
>"I’ve come to think that the most fundamental problem with p-values is that no one can really say what they are." Not surprising. The correct interpretation of p-values was not discovered until ~2 years ago. If you search around the internet you will find out that, according to the discoverer, he couldn't find a stats journal to publish it. The number of people who know the correct definition must be vanishingly small. Thank god for arxiv, or I would have never understood p-values: http://arxiv.org/abs/1311.0081 http://arxiv.org/abs/1311.0081 tldr: Calculating the p-value + sample size is a lossless compression algorithm. The thing being compressed is a likelihood function (way of describing effect size).
- kgwgk 11y ago
  I think you overestimate the importance of that paper. You can get "a" likelihood from the p-value in the same way that a sufficient statistic determines "the" likelihood function. In the best case, if the sufficient statistic is one-dimensional and you are doing a one-tailed test, both likelihood functions are the same (because the mapping from the sufficient statistic to the p-value is bijective). In general you're losing information (there will be multiple values of the sufficient statistic mapped to the same p-value).
leephillips 11y ago
I'm surprised nobody has brought up the backlash against routine use of the p-value among scientists and journals. At least one journal has actually banned papers that report p-values: http://www.sciencebasedmedicine.org/psychology-journal-bans-significance-testing/ http://www.sciencebasedmedicine.org/psychology-journal-bans-...
- jsprogrammer 11y ago
  Leave it to psychology to ban pretty much the only scientific tool we have...
btilly 11y ago
My belief is that p-values are popular exactly BECAUSE they can be misunderstood so easily as the answer to the question people really want answered. Namely, "What are the odds that this is the right answer?" I many years of experience in helping people understand A/B testing. And I have found that no matter how many times you give a correct explanation, people will have this exact misunderstanding. Repeatedly. So if you're A/B testing, here is something to consider. If you successfully set up a testing culture where you decide at a 95% confidence level, and are running one A/B test per week, you will make an average of 2-3 mistakes per year. In the long run, mistakes are unavoidable. You will make mistakes. The only question is how often you make mistakes, and how bad those mistakes are. P-values manage to bound how often you make mistakes with the trick of most of the time saying you couldn't figure out an answer. THIS IS USELESS FOR A BUSINESS. A business needs to decide what text to use, even if they're not confident. Your A/B tests should always produce a usable business answer. The question of interest is how often you make bad mistakes and how bad they are. And p-values are useless for that.
nartz 11y ago
If i flipped a regular coin 1000 times, and happened to get 900 heads in a row, you may think 'hey, something is wrong with this coin' - however, could you be sure something is up? How would you go about telling whether something is really up, or if this is just a fluke chance this one time? The motivation for the p-value is to tell you what the probability of a fluke chance this would be. The way to do better is to run the experiment over and over, flipping this coin 1000 times again, and again. However, since science experiments are extremely expensive, instead we only run them one time, and try to 'make sure this wasnt some fluke chance'.
timvdalen 11y ago
Is the video broken for anyone else?
graycat 11y ago
How about: The p-value is, in a statistical hypothesis test, the probability of Type I error, that is the probability of rejecting the null hypothesis when it is true. Or, suppose we have some data and want to use it to estimate the value of some number b. It can be that if we can assume something about b, e.g., that b = 0, then we can calculate the probability distribution of our estimate of b. This assumption about b is the null hypothesis. Intuitively it is a hypothesis that there is no effect, that what we thought might have happened didn't, was a null effect. We can make two mistakes. We can reject the null hypothesis when it is true -- this is called Type I error. Or we can accept the null hypothesis when it is false -- this is called Type II error. With our null hypothesis, we get and look at the distribution of our estimate of b and see where our actual estimate is in that distribution. The p-value is the probability, from the distribution of our estimate, of getting an estimate as far or farther from our null hypothesis value for b as we did. So, if the p-value is really small, say, 1%, then we can reject the null hypothesis, that is, say that it is false, and be wrong only 1% of the time. E.g., if our null hypothesis is that b = 0 and our estimate of b is 10 and from the distribution of our estimate a value of our estimate being as far as 10 from b = 0 is 1%, then we reject that b = 0 and conclude that it b is not zero and are wrong only 1% of the time. If the probability of our estimate being greater than or equal to 10 is 1% and we reject the null hypothesis, then we conclude that b > 0. This is all just hypothesis testing in statistics 101. Will also want to know about the power of a test, the t-test, the F ratio, the chi-squared test, and resampling and distribution-free tests. There is more detail in https://en.wikipedia.org/wiki/Type_I_and_type_II_errors https://en.wikipedia.org/wiki/Type_I_and_type_II_errors
- nonbel 11y ago
  >"How about: The p-value is, in a statistical hypothesis test, the probability of Type I error, that is the probability of rejecting the null hypothesis when it is true." No, see here: "Of critical importance, as Goodman (1993) has pointed out, is the extensive failure to recognize the incompatibility of Fisher’s evidential p value with the Type I error rate, α, of Neyman–Pearson statistical orthodoxy." P Values are not Error Probabilities http://www.uv.es/sestio/TechRep/tr14-03.pdf http://www.uv.es/sestio/TechRep/tr14-03.pdf
myNXTact 11y ago
Could someone grade the explanation I gave my non-technical boss last week? In the context of a regression analysis, I said, "P-values indicate the chance that the apparent effect of the variable is from random fluctuations in the data instead of the variable itself."
- analog31 11y ago
  While we're at it, let's try this one too: The p-value, if correctly computed, is the probability that data collected under different conditions actually come from the same distribution. My definition sidesteps the word "effect."
  - myNXTact 11y ago
    My two cents is that once you start talking about "distributions" you add a layer of obfuscation that makes it a non-intuitive explanation.
    - analog31 11y ago
      Thanks. That's a good point. I wonder if there's a better term, or way of explaining it.
- tedsanders 11y ago
  I would grade this explanation poorly, because it is wrong. It's impossible to know the odds of an apparent effect coming from random data fluctuations or a real effect. The p-value cannot tell you these odds. The p-value only tells you one half of that---the odds that random fluctuations would cause an apparent effect.
jdeisenberg 11y ago
My attempt to describe it for psych research methods students - in visual novel form: http://evc-cit.info/psych018/hyptest/index.html http://evc-cit.info/psych018/hyptest/index.html
pvaldes 11y ago
I'm sure that there are lots of scientists that can easily explain p-values. "Some scientists can not easily explain p-values" should be a more accurate, and much better, title.
lovboat 11y ago
In order to use p-values properly, that is to make decision, you should explain what are you going to do with the information that the p-value provides. Rejecting the null hypothesis when the p-value < 0.05 is a sensible thing, but making a strong decision when the p-value is < 0.0001 doesn't make sense in many circumstances. You should have a scheme for the decision before having the p-value.

In my experience, people in science who are not statisticians have not completely understood the p value. Typically people are clueless on p-Values and the statistical power and how they relate.

MollyR 11y ago
This is scarily true. When my lab (bioinformatics) would work/collaborate with research biologists and their labs, the research biologists lack of statistics would become very clear. Especially when we did data mining on their qpcr experiments, we constantly have to explain things like bonferroni corrections. Edit: This is just my experience at one university with 6-7 biological research labs. I regret using the generalization, apologies.

peatmoss 11y ago

Lest anyone skimming the headline misses it, the point here isn't that these scientists couldn't correctly explain a p-value. The scientists here all could. It's that they couldn't explain it in a way that made intuitive sense to anyone. So, rather than being a dig at scientists, I think the points here are: 1. p-values are intrinsically none-too-intuitive. 2. What p-values tell us isn't necessarily that interesting.

Alex3917 11y ago
> What p-values tell us isn't necessarily that interesting. The problem is that the concept sounds more important than it is if you don't understand it. This is a bad failure state. The issue is that the probability of some phenomena happening by chance is mostly orthogonal to the probability of that phenomena being true or whatever, but this seems outside the ability of most people to grasp.
- peatmoss 11y ago
  I can also say that, when I was starting stats, the term p-value seemed to hint tantalizingly at "probability." I cannot be the only person seduced into erroneously thinking that a p-value had something to do with the probability that something could be untrue.
  - lmm 11y ago
    Well a p-value is a probability, so I don't think that's exactly misleading.
jsprogrammer 11y ago
> 1. p-values are intrinsically none-too-intuitive P-values are quite intuitive. They are the ratio of observations made which agree with the null hypothesis to the total number of observations made. Edit: If you are going to downmod, at least point out the error you perceive.
- nickpsecurity 11y ago
  That was as intuitive as the seemless connection between the Magic Bullet Theory and Ballistics.
  - jsprogrammer 11y ago
    It comes directly from the calculation of p-values through probability theory.
    - omginternets 11y ago
      I'm reminded of a joke about how Haskell empowers you to apply the infinite power of abstract mathematics using the intuitive simplicity of abstract mathematics. I think the resistance you're encountering stems from the fact that you're appealing to formality rather than intuition.
      - senekerim 11y ago
        Also from the fact that he doesn't know what he's talking about.
        jsprogrammer 11y ago
        You talking about me?
      - jsprogrammer 11y ago
        Counting positive observations vs. total observations is a formality rather than intuition? The formality is the axioms of probability. The intuition is that you make a theory and measure how many predictions it gets right.
cousin_it 11y ago
Especially when publication bias comes into play: https://xkcd.com/882/ https://xkcd.com/882/
- KingMob 11y ago
  Sadly true. On the one hand, the scientists ignored the multiple comparisons problem, and should have applied something like the Bonferroni or False Discovery Rate corrections. OTOH, they got publications and publicity for ignoring it.
vlehto 11y ago
>2. What p-values tell us isn't necessarily that interesting. High p-value is the first thing to look at a study. If it's high, then you can simply assume the findings are unreliable. But if it's low, it doesn't yet mean anything. The p-values allow you to discard worst mumbojumbo quickly. That might not be interesting, but it's valuable. The thing to remember is that "we don't know" =/ "untrue".
blahblah3 11y ago
I would argue that the p-value way of reasoning (probability of something happening given a hypothesis) corresponds to an intuitive way many people reason about probability. For example, suppose the police are investigating the robbery of a home. They find a stranger's (call him person S) fingerprints all over the house. If S claimed he was innocent, a questioner might ask: "If you're really innocent, then why were your fingerprints all over the house?" This is another way of stating that P(Data | innocent) is small (similar to the p-value concept). Since we often don't have a strong prior reason for believing a given individual is innocent or guilty, the p-value heuristic can be reasonable. The formal connection to P(innocent | Data) is given by Bayes' theorem
btilly 11y ago
There is a third point. Which is that scientists consistently agree with clearly incorrect statements about p-values. That means that knowing the correct definition is no guarantee that you're actually using it correctly. (Odds are you aren't.)
vectorjohn 11y ago
I've come to think that if someone can't explain something well, they probably don't understand it as well as they think. I've heard people that fancy themselves experts on this or that software concept, and when trying to explain it use all the technical jargon they possibly can. It comes off as just regurgitating things they've heard before. In other words, I suspect the scientists that couldn't explain p-values in an intuitive way, didn't themselves fully grasp the subtleties of p-values.

graffitici 11y ago

From what I understand, p-values are typically used by frequentist method for verifying that the "prediction" seems correct. On the other hand, the Bayesian approach is to split the data into training and test sets, and verify how much of it holds. Does this make sense? If the p-values are not very good at conveying "confidence in the prediction," is this another argument in favor of a more Bayesian approach to statistics? Any thoughts would be appreciated!

gbrown 11y ago
Well, neither your description of p-values, nor your understanding of Bayesian inference is correct, so... no?
- TeMPOraL 11y ago
  It would be helpful to tell how the commenter is wrong, and what is the correct description.
  - gbrown 11y ago
    It would also be helpful if they'd read the article.
    - TeMPOraL 11y ago
      I read the article and it's almost completely orthogonal to the questions asked by 'graffitici. There's nothing suggesting that they didn't read the article, so now I assert that you have three things to explain - where they're wrong, what's the right answer to their question and why on Earth did you think they didn't read the article?
      - gbrown 11y ago
        From the article: "We want to know if results are right, but a p-value doesn’t measure that. It can’t tell you the magnitude of an effect, the strength of the evidence or the probability that the finding was the result of chance." From the first sentence of the parent comment: "From what I understand, p-values are typically used by frequentist method for verifying that the "prediction" seems correct." If my reply seemed glib, it was because I thought the question was so far off base, that graffitici almost certainly hadn't read the article. To relate to the topic under discussion, the distribution of possible comments produced by people who have carefully read the article is extremely unlikely to result in a post as extremely misinformed as the one above. If there was a simple statistical misunderstanding, perhaps that would be worth clearing up. Instead, the parent comment used some statistical words, but was very nearly incoherent. Perhaps I shouldn't have said anything if I wasn't willing to write a protracted essay about the differences between frequentist and Bayesian inference.

goodside 11y ago

My go-to phrase: "Even if there were no real difference, we would see a result like this X% of the time anyway."

llasram 11y ago
I agree that makes intuitive sense when applied to explicitly constructed experiments we could attempt to exactly repeat, but the intuition begins to break down in more complicated circumstances, such as retrospective or longitutinal studies. Saying something would happen "X% of the time anyway" becomes a philosophical issue.
- omginternets 11y ago
  Yes, well, we could talk about frequentist vs bayesian statistics, but that's hardly the point of the article. That is what a p-value means, feasibility aside. There are also more intuitive ways to think about it, "false-positive rate" being one.
omginternets 11y ago
Mine is remarkably similar, though less general: "if p=.05, then running a bogus experiment 20 times ensures you'll publish." Suddenly, the abysmal state of nutritional epidemiology makes sense. How many labs do you suspect are running "red meat = death" studies?
- TeMPOraL 11y ago
  One is reminded of an XKCD classic - https://xkcd.com/882/ https://xkcd.com/882/. Sadly, it only goes to point out the sad state of contemporary science. Especially in terms of nutrition, the only way to stay sane is to stick to the good old street-fighting Bayes - if popularly consumed product X was leading to cancer in any way relevant to your life, you'd see people dropping like flies all over the place.
  - omginternets 11y ago

jowiar 11y ago

That it's called a "p-value" is rather indicative of it's nebulous nature. For something that is used in a rather applied context, giving it a one-letter name that has zero context is a bit of a cop-out.

jsprogrammer 11y ago
I am sure it comes from probability theory, where it is common to use the notation P(x) to mean, "the probability of observing `x`".
GFK_of_xmaspast 11y ago
That's how basically everything in mathematics works tho.

elbigbad 11y ago

P-Values can be pretty unintuitive to people unfamiliar, but can readily be explained I think. I do think that this book did an excellent job of explaining it: What Is a P-Value Anyway? http://www.pearsonhighered.com/vickers/ http://www.pearsonhighered.com/vickers/ Excerpt (starts at p.57 in the link above): [I]f you do nothing else, please try to remember the following sentence: “the p-value is the probability that the data would be at least as extreme as those observed, if the null hypothesis were true.” Though I’d prefer that you also understood it—about which, teeth brushing. I have three young children. In the evening, before we get to bedtime stories (bedtime stories being a nice way to end the day), we have to persuade them all to bathe, use the toilet, clean their teeth, change into pajamas, get their clothes ready for the next day and then actually get into bed (the persuading part being a nice way to go crazy). My five-year-old can often be found sitting on his bed, fully dressed, claiming to have clean teeth. The give-away is the bone dry toothbrush: he says that he has brushed his teeth, I tell him that he couldn’t have. My reasoning here goes like this: the toothbrush is dry; it is unlikely that the toothbrush would be dry if my son had cleaned his teeth; therefore he hasn’t cleaned his teeth. Or using statistician-speak: here are the data (a dry toothbrush); here is a hypothesis (my son has cleaned his teeth); the data would be unusual if the hypothesis were true, therefore we should reject the hypothesis. [...] So here is what to parrot when we run into each other at a bar and I still haven’t managed to work out any new party tricks: “The p-value is the probability that the data would be at least as extreme as those observed, if the null hypothesis were true.” When I recover from shock, you can explain it to me in terms of a toothbrush (“The probability of the toothbrush being dry if you’ve just cleaned your teeth”).

nonbel 11y ago
>"My reasoning here goes like this: the toothbrush is dry; it is unlikely that the toothbrush would be dry if my son had cleaned his teeth; therefore he hasn’t cleaned his teeth." I don't think this works so well because you are jumping from a qualitative statement (dry or not dry) to a quantitative ("at least as extreme").
sbov 11y ago
This is way too much for the layman and full of mathematical language. What is null hypothesis? What is "at least as extreme as"? What is probability? Nevermind that the null hypothesis forces us to think in double negatives, which isn't the most intuitive. It also isn't clear to me how you would apply your example given how most p-values are used. I read an article that says exercising makes you richer with a 95% confidence. So, as a layman, if I attempt to apply your example, it leads me directly into a common p-value mistake: the probability of me becoming richer if I exercise. 95%, right? Wrong!

jroitgrund 11y ago

The coin example makes perfect sense to me and I had no idea what a p-value was before I read this article. Here's an intuitive explanation: a p-value is the probability of getting your experimental results given that your hypothesis is wrong.

bweitzman 11y ago
I think the coin example makes sense but is not a good example in this case. You will never be able to determine the validity of a coin just by flipping it. Any sequence of flips is technically valid but possibly just very rare for a fair coin. The p-value won't tell you if the coin is fair or not, but it can tell you the probability that the coin is fair.
- skybrian 11y ago
  If the sequence is rare enough then we do, in practice, conclude that the coin is unfair. If you can't make that conclusion then science doesn't work; no observations will convince you.
  - bweitzman 11y ago
    Of course. If you flip the coin 100 times and get all heads than you are safe to call the coin unfair, even though you would expect to see that result (1/2)^100 and thus would be wrong once in a while. I think that's sort of the point that the article is making actually. That high probability does not imply truth. There are other non probabilistic ways to verify that coin is unfair, for example by looking at the density throughout the coin.
    - TeMPOraL 11y ago
      Observations do not imply truth. 0 and 1 are not probabilities, and in real life you can't prove something using observations in a logical sense. Real world runs on probabilities, not boolean logic. Even if you look at the density of the metal throughout the coin, there's still a chance I've altered your device to report the coin is fair. Or a passing microsingularity decided to play games with the scanning beam. Or you're just imagining the whole thing. That's not to say one should despair that the world is unknowable. One only has to get used to the fact that, in practice, "true" just means "extremely, extremely likely".

blahblah3 11y ago

The point of p-values is to limit false positive rates. If you only accept results with p-value <= 0.05, your false positive rate is at most 5%. Of course, if you had apriori knowledge that everything was a "null result" , then 100% of the results you accept would be false positives. By Bayes' theorem, P(H|D) = P(D|H) * P(H) / P(D) . Therefore, given a fixed "prior probability" of a hypothesis P(H), a lower p-value (P(D|H)) implies a lower probability of the null being true. I think this relationship leads to a lot of confusion when thinking of p-values.

jcranmer 11y ago
You are incorrect. False positive rates in practice appear to be about 20-50%, even with a p-value of 0.05.
- blahblah3 11y ago
  I was describing the theory behind the p-value. In practice, people slice and dice the dataset in multiple ways and try several specifications which increases the false positive rate (see the multiple comparisons problem)
- nonbel 11y ago
  The best estimate right now is 20-50% replication rate for fields that rely on p less than 0.05 to judge results. So I think you meant 50-80% false positive rate, although these are not equivalent (ie is the null hypothesis of no difference ever really exactly true...): >"Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; [...] In cell biology, two industrial laboratories reported success replicating the original results of landmark findings in only 11 and 25% of the attempted cases" http://www.ncbi.nlm.nih.gov/pubmed/26315443 http://www.ncbi.nlm.nih.gov/pubmed/26315443 Edit: For comparison, Dr. Oz was accused of fraud and called in front of congress for only being wrong half the time: https://www.washingtonpost.com/news/morning-mix/wp/2014/12/19/half-of-dr-ozs-medical-advice-is-baseless-or-wrong-study-says/ https://www.washingtonpost.com/news/morning-mix/wp/2014/12/1...

nickpsecurity 11y ago

My takeaway is to not depend on or use them wherever possible. If I do use them, I get some heuristics from a survey of a bunch of statistics experts claiming to know what it is. And I'll be sure to keep a minimal, survey size of 30. ;)

jforman 11y ago

Having been both a scientist and, now, a product manager for a medical device, I think science would benefit from the work around verification vs. validation: https://en.wikipedia.org/wiki/Verification_and_validation https://en.wikipedia.org/wiki/Verification_and_validation A p-value is a verification tool and nothing more, and yet scientists all too often take the "hunt for the p-value" as the only goal. Rather than fish through data for a p-value that ends up defining the scientific narrative in a paper, p-values need to be integrated into a much larger validation process driven by pre-defined scientific goals. It's no wonder that so many results are found to be specious given today's scientific culture.

Jach 11y ago

P-values are only hard to explain when you try to explain in English. Stick to the math, where it's just P(X >= x | Ho) and once you understand the math it's immediately clear in a way an English sentence (or paragraph trying to equate to the math without math) isn't. My go-to presentation on this is: http://www.biostat.jhsph.edu/~cfrangak/cominte/goodmanvalues.pdf http://www.biostat.jhsph.edu/~cfrangak/cominte/goodmanvalues...

omginternets 11y ago

From my limited teaching experience, renaming "p-value" to "probability of a false positive" makes it all very simple.

nonbel 11y ago
This is a very common, but dangerously wrong, interpretation of p-values. Others in this thread have expressed the same misconception, see the paper linked here: https://news.ycombinator.com/item?id=10628860 https://news.ycombinator.com/item?id=10628860
- omginternets 11y ago
  Duly noted, thanks for the reference!

TallGuyShort 11y ago

This is an excellent talk I attended a while ago that explains it very well from a programmer's perspective: https://www.youtube.com/watch?v=5Dnw46eC-0o&list=PL055Epbe6d5Y8_iZPo7pH3hOnAtchMCJt&index=8 https://www.youtube.com/watch?v=5Dnw46eC-0o&list=PL055Epbe6d...

cafebeen 11y ago

This is a surprisingly meta article... It claims there's a general lack of statistical understanding among experts, and then supports itself by making sweeping generalizations from a tiny sample size!

nonbel 11y ago

>"I’ve come to think that the most fundamental problem with p-values is that no one can really say what they are." Not surprising. The correct interpretation of p-values was not discovered until ~2 years ago. If you search around the internet you will find out that, according to the discoverer, he couldn't find a stats journal to publish it. The number of people who know the correct definition must be vanishingly small. Thank god for arxiv, or I would have never understood p-values: http://arxiv.org/abs/1311.0081 http://arxiv.org/abs/1311.0081 tldr: Calculating the p-value + sample size is a lossless compression algorithm. The thing being compressed is a likelihood function (way of describing effect size).

kgwgk 11y ago
I think you overestimate the importance of that paper. You can get "a" likelihood from the p-value in the same way that a sufficient statistic determines "the" likelihood function. In the best case, if the sufficient statistic is one-dimensional and you are doing a one-tailed test, both likelihood functions are the same (because the mapping from the sufficient statistic to the p-value is bijective). In general you're losing information (there will be multiple values of the sufficient statistic mapped to the same p-value).

leephillips 11y ago

I'm surprised nobody has brought up the backlash against routine use of the p-value among scientists and journals. At least one journal has actually banned papers that report p-values: http://www.sciencebasedmedicine.org/psychology-journal-bans-significance-testing/ http://www.sciencebasedmedicine.org/psychology-journal-bans-...

jsprogrammer 11y ago
Leave it to psychology to ban pretty much the only scientific tool we have...

btilly 11y ago

My belief is that p-values are popular exactly BECAUSE they can be misunderstood so easily as the answer to the question people really want answered. Namely, "What are the odds that this is the right answer?" I many years of experience in helping people understand A/B testing. And I have found that no matter how many times you give a correct explanation, people will have this exact misunderstanding. Repeatedly. So if you're A/B testing, here is something to consider. If you successfully set up a testing culture where you decide at a 95% confidence level, and are running one A/B test per week, you will make an average of 2-3 mistakes per year. In the long run, mistakes are unavoidable. You will make mistakes. The only question is how often you make mistakes, and how bad those mistakes are. P-values manage to bound how often you make mistakes with the trick of most of the time saying you couldn't figure out an answer. THIS IS USELESS FOR A BUSINESS. A business needs to decide what text to use, even if they're not confident. Your A/B tests should always produce a usable business answer. The question of interest is how often you make bad mistakes and how bad they are. And p-values are useless for that.

nartz 11y ago

If i flipped a regular coin 1000 times, and happened to get 900 heads in a row, you may think 'hey, something is wrong with this coin' - however, could you be sure something is up? How would you go about telling whether something is really up, or if this is just a fluke chance this one time? The motivation for the p-value is to tell you what the probability of a fluke chance this would be. The way to do better is to run the experiment over and over, flipping this coin 1000 times again, and again. However, since science experiments are extremely expensive, instead we only run them one time, and try to 'make sure this wasnt some fluke chance'.

timvdalen 11y ago

Is the video broken for anyone else?

graycat 11y ago

How about: The p-value is, in a statistical hypothesis test, the probability of Type I error, that is the probability of rejecting the null hypothesis when it is true. Or, suppose we have some data and want to use it to estimate the value of some number b. It can be that if we can assume something about b, e.g., that b = 0, then we can calculate the probability distribution of our estimate of b. This assumption about b is the null hypothesis. Intuitively it is a hypothesis that there is no effect, that what we thought might have happened didn't, was a null effect. We can make two mistakes. We can reject the null hypothesis when it is true -- this is called Type I error. Or we can accept the null hypothesis when it is false -- this is called Type II error. With our null hypothesis, we get and look at the distribution of our estimate of b and see where our actual estimate is in that distribution. The p-value is the probability, from the distribution of our estimate, of getting an estimate as far or farther from our null hypothesis value for b as we did. So, if the p-value is really small, say, 1%, then we can reject the null hypothesis, that is, say that it is false, and be wrong only 1% of the time. E.g., if our null hypothesis is that b = 0 and our estimate of b is 10 and from the distribution of our estimate a value of our estimate being as far as 10 from b = 0 is 1%, then we reject that b = 0 and conclude that it b is not zero and are wrong only 1% of the time. If the probability of our estimate being greater than or equal to 10 is 1% and we reject the null hypothesis, then we conclude that b > 0. This is all just hypothesis testing in statistics 101. Will also want to know about the power of a test, the t-test, the F ratio, the chi-squared test, and resampling and distribution-free tests. There is more detail in https://en.wikipedia.org/wiki/Type_I_and_type_II_errors https://en.wikipedia.org/wiki/Type_I_and_type_II_errors

nonbel 11y ago
>"How about: The p-value is, in a statistical hypothesis test, the probability of Type I error, that is the probability of rejecting the null hypothesis when it is true." No, see here: "Of critical importance, as Goodman (1993) has pointed out, is the extensive failure to recognize the incompatibility of Fisher’s evidential p value with the Type I error rate, α, of Neyman–Pearson statistical orthodoxy." P Values are not Error Probabilities http://www.uv.es/sestio/TechRep/tr14-03.pdf http://www.uv.es/sestio/TechRep/tr14-03.pdf

myNXTact 11y ago

Could someone grade the explanation I gave my non-technical boss last week? In the context of a regression analysis, I said, "P-values indicate the chance that the apparent effect of the variable is from random fluctuations in the data instead of the variable itself."

analog31 11y ago
While we're at it, let's try this one too: The p-value, if correctly computed, is the probability that data collected under different conditions actually come from the same distribution. My definition sidesteps the word "effect."
- myNXTact 11y ago
  My two cents is that once you start talking about "distributions" you add a layer of obfuscation that makes it a non-intuitive explanation.
  - analog31 11y ago
    Thanks. That's a good point. I wonder if there's a better term, or way of explaining it.
tedsanders 11y ago
I would grade this explanation poorly, because it is wrong. It's impossible to know the odds of an apparent effect coming from random data fluctuations or a real effect. The p-value cannot tell you these odds. The p-value only tells you one half of that---the odds that random fluctuations would cause an apparent effect.

jdeisenberg 11y ago

My attempt to describe it for psych research methods students - in visual novel form: http://evc-cit.info/psych018/hyptest/index.html http://evc-cit.info/psych018/hyptest/index.html

pvaldes 11y ago

I'm sure that there are lots of scientists that can easily explain p-values. "Some scientists can not easily explain p-values" should be a more accurate, and much better, title.

lovboat 11y ago

In order to use p-values properly, that is to make decision, you should explain what are you going to do with the information that the p-value provides. Rejecting the null hypothesis when the p-value < 0.05 is a sensible thing, but making a strong decision when the p-value is < 0.0001 doesn't make sense in many circumstances. You should have a scheme for the decision before having the p-value.