32 ms·

Why I've lost faith in p values

354 points by anacleto 8y ago

rossdavidh 8y ago
The core issue is that p-values are cheaper to get than replicating the study, but replicating the study is the only reliable way to see if it's true or not. Sometimes the expensive/time-consuming way, is the only good way.
- lisper 8y ago
  Replication by itself is not enough. You need pre-registration too. Otherwise you can p-hack the replications.
  - rossdavidh 8y ago
    True, but the article was making a point about "even if we set aside p-hacking for a moment", so I was going with that. But I agree, pre-registration is important as well.
  - o2348diuu 8y ago
    Not really disagreeing with you necessarily, but you can hack pre-registration too, in a way that becomes more tenable with smaller studies, which benefit less from pre-registration anyway. The tables are misleading too: of course p(h0=T|p<.05) is going to be about 0.50 if your power is 0.50 and p(h1) = 0.10. That is maybe the bigger problem than p-values per se. Also, he fails to compute the other cell, which is p(h0=T|p>.05), which ends up being about 0.95. The problem in that table is have no power to detect anything. In real life, you don't know p(h0)--that's the point of doing the study. So Bayesianism doesn't really help you. The real problem is lack of power.
  - tomrod 8y ago
    Pre-registration?
    - lisper 8y ago
      http://www.apa.org/science/about/psa/2015/08/pre-registration.aspx http://www.apa.org/science/about/psa/2015/08/pre-registratio...
nonbel 8y ago
tldr: The author got a PhD in 1993[1] and is just now figuring out that p-values are not false positive rates [1]http://mindbrain.ucdavis.edu/people/sjluck http://mindbrain.ucdavis.edu/people/sjluck I was lucky and figured it out before getting a degree. Its got to be hard for people in this position to look back on their previous work where the most fundamental aspect of interpreting the results was incorrect. He gets it right that statistics are good for estimation, but there is a part two. You need to come up with a theory that makes a prediction to compare to these estimates, and then test that. Ie, your prediction about the distribution of the results is the "null hypothesis". I think p-values are probably ok for that.
- SubiculumCode 8y ago
  I've met Steve Luck multiple times over the last decade. He is a very rigorous and insightful scientist, with expertise in a wide range of methods (especially eeg) and psychological phenomena. His wife Prof. Lisa Oaks may be even more impressive in these regards, but that is an aside. The point I would like to make though is that there are lots of aspects to research and stats is just one part to master of many. It should not be up to a neuroscientist to make posts like this. The stats community should be actively pushing the scientific community to alternatives that match interpretative intuition with reality of the statistical metric.
  - nonbel 8y ago
    I went through the same thing. I was taught the same half-assed BS statistics with all the wrong interpretations and was surrounded by people just following along with that. Still, it was a point of duty/honor for me to figure out what these statistical results meant. How is interpreting results not a key part of a scientists job? I guess if you don't want to bother learning to interpret results then you can do science by either being a lab tech or theorist.
deleted 8y ago
[deleted]
thaumasiotes 8y ago
Sounds like someone who never understood statistics, still doesn't, and doesn't want to. A particularly glaring issue is this offhand comment: > this is a statement about what happens when the null hypothesis is actually true. In real research, we don't know whether the null hypothesis is actually true. If we knew that, we wouldn't need any statistics! In real research, we have a p value, and we want to know whether we should accept or reject the null hypothesis. That isn't a question that any statistical approach will help you with. There's a reason we talk in terms of "rejecting" or "failing to reject" a hypothesis. We don't do statistical tests to accept hypotheses, only to reject them. The concept of accepting one hypothesis based on a comparison between it and one other hypothesis is ludicrous on its face, suffering exactly the problems associated with Pascal's wager.
- nonbel 8y ago
  >"That isn't a question that any statistical approach will help you with. There's a reason we talk in terms of "rejecting" or "failing to reject" a hypothesis. We don't do statistical tests to accept hypotheses, only to reject them." In (Neyman-Pearson's) hypothesis testing you definitely can accept an hypothesis. This is not true for (Fisher's) significance testing though. This paper is probably fine to explain it: https://onlinelibrary.wiley.com/doi/pdf/10.1002/0470013192.bsa438 https://onlinelibrary.wiley.com/doi/pdf/10.1002/0470013192.b...
- s17n 8y ago
  He clearly meant "fail to reject" where he said "accept", you're quibbling about semantics. The whole post is about how p-values (and indeed, known statistical techniques) don't actually help you decide whether or not you should reject the null.
  - thaumasiotes 8y ago
    The post is about a confusion between the question that p-values address ("how likely is this data to have come from the null hypothesis?") and another very different question ("given several findings that were unlikely to have come from various different hypotheses, how many are likely to be spurious?").
wrp 8y ago
One of the best articles covering this issues is Meehl[1][2]. You can find discussion in various places like Gelman[3] and Reinhart[4]. [1] Meehl, Paul E (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66(1), 195–244. [2] http://meehl.umn.edu/files/144whysummariespdf http://meehl.umn.edu/files/144whysummariespdf [3] http://andrewgelman.com/2015/03/23/paul-meehl-continues-boss/ http://andrewgelman.com/2015/03/23/paul-meehl-continues-boss... [4] https://www.refsmmat.com/notebooks/meehl.html https://www.refsmmat.com/notebooks/meehl.html
deleted 8y ago
[deleted]
tprice7 8y ago
'The fundamental problem is that p values don't mean what we "need" them to mean, that is p(null | significant effect).' From Bayes' theorem, this more useful probability is given by p * x, where x = p(null) / p(significant effect). Maybe we could just lower the accepted threshold for statistical significance by several orders of magnitude so that, for statistically significant p, p * x is still small even for careful (i.e. big) estimates of x (e.g. maybe a Fermi approximation of the total number of experiments ever performed in the field in question). This doesn't necessarily imply impractically big sample sizes, although obviously this depends on the specifics (I believe the p value for a given value of the t-statistic decays exponentially with sample size).
- kkylin 8y ago
  This was basically the suggestion here: https://www.nature.com/articles/s41562-017-0189-z.epdf https://www.nature.com/articles/s41562-017-0189-z.epdf Also previous HN discussion: https://news.ycombinator.com/item?id=15192610 https://news.ycombinator.com/item?id=15192610
- nonbel 8y ago
  I don't follow your argument. You've got two premises: 1) You are saying that people are committing the transposing the conditional fallacy: p(H0|data) != p(data|H0): - OK 2) You say to use Bayes theorem to get the value we want: - OK, but actually a better formulation is p(H_0|data) = p(H_0)*p(data|H_0)/[p(H_0)*p(data|H_0) + p(H_1)*p(data|H_1) + ... + p(H_n)*p(data|H_n)] You probably don't need to add up all the way to hypothesis n since the terms eventually become negligible and can be dropped from the denominator. The point is that you have to compare how likely the result would be under other hypotheses, not just H_0. 3) You propose lowering the threshold for "significance" - How does this follow from the premises? Lets say you get a very low value for p(H_0)p(data|H_0), this can still be much higher than p(H_1)p(data|H_1), etc so it is still the best choice. Ie, you can get a low p-value given H_0 but if there is no better model out there you should still keep H_0.
  - edwinksl 8y ago
    Minor point, but you are missing p(H_0)*p(data|H_0) in the denominator in bullet point 2.
piotrkaminski 8y ago
> instead of asking whether an effect is null or not, we should ask how big the effect is likely to be given the data. However, at the end of the day, editors need to make an all-or-none decision about whether to publish a paper Yet another way in which the traditional publishing structure actively harms science.
- Bromskloss 8y ago
  Do you have an alternative way of publishing in mind?
  - piotrkaminski 8y ago
    My criticism is not necessarily constructive. ;) But it's not too hard to imagine something along the lines of arXiv combined with a rating/commenting system not unlike HN itself, combined with a Facebook-ish algorithm for surfacing articles relevant to each reader. The devil is in the details, of course, but it shouldn't be hard to do better than the expensive, artificially constrained, and arbitrary system we're saddled with now. The real trick will be convincing entrenched academics to switch -- I'm still not convinced this is actually possible.
davidxc 8y ago
Here's a more simple thought experiment that gets across the point of why p(null | significant effect) /= p(significant effect | null), and why p-values are flawed as stated in the post. Imagine a society where scientists are really, really bad at hypothesis generation. In fact, they're so bad that they only test null hypothesis that are true. So in this hypothetical society, the null hypothesis in any scientific experiment ever done is true. But statistically using a p value of 0.05, we'll still reject the null in 5% of experiments. And those experiments will then end up being published in scientific literature. But then this society's scientific literature now only contains false results - literally all published scientific results are false. Of course, in real life, we hope that our scientists have better intuition for what is in fact true - that is, we hope that the "prior" probability in Bayes' theorem, p(null), is not 1.
- danieltillett 8y ago
  I don’t have to imagine that society since they already exist and are called social scientists. More seriously you do make a good point which is all scientists lie on a spectrum from always generating true hypotheses, to always generating false hypotheses. Scientists in different fields tend to lie more to one or the other of the extremes. My experience is the observational sciences are more shifted to the always false end than the experimental sciences.
  - modbait 8y ago
    This is being down-voted for the shots fired, but the underlying point is almost certainly true to a degree. People aren't ideologically invested in (say) the weight of electrons in the same way that they are in IQ curves across demographic groups.
    - danieltillett 8y ago
      Ideology certainly plays an important role in generating false hypothesis, but all the observational sciences suffer from the problem that you can't run experiments to rigorously test the robustness of your hypothesis. In the experimental sciences you can get far using the rule of thumb that if you need statistics you did the wrong experiment, while in the observational sciences the use of statistics is inherent.
btilly 8y ago
Here is a better way to think about this. The proper role of data is to update our existing beliefs about the world. It is not to specify what our beliefs should be. The question that we really want to answer is, "What is the probability that X is true?" What p-values do is replace that with the seemingly similar but very different, "What is the probability that I'd have the evidence I have against X by chance alone were X true?" Bayesian factors try to capture the idea of how much belief should shift. The conclusion at the end is that replication is better than either approach. I agree. We know that there are a lot of ways to hack p-values. Bayesian factors haven't caught on because they don't match how people want to think. However if we keep consistent research standards, and replicate routinely, the replication rate gives us a sense of how much confidence we should have in a new result that we hear about. (Spoiler. A lot less confidence than most breathless science reporting would have you believe.)
- analog31 8y ago
  My fear is that in 10 years time, we will have learned to hack Bayesian factors.
  - btilly 8y ago
    The way that you hack Bayesian factors is selectively including data. And the selection process can be as simple as publication bias causing some results to be published and others not. This is the fundamental weakness of meta analysis.
- mycall 8y ago
  > The proper role of data is to update our existing beliefs about the world. It is not to specify what our beliefs should be. Create the schema beforehand, I get that. But feature extraction does work, providing models from data. Sometimes takes much time to analyze and understand the models.
  - btilly 8y ago
    If you have enough data and a strong enough signal, then all reasonable belief systems should converge on the same answer. Do not let that fact fool you into believing that raw data is the only thing necessary to make good decisions when faced with realistic situations.
btcindivist 8y ago
p-values that are not in the physics ranges are ridiculous. It's a shame everyone started copying physics but decided for higher acceptance/rejection values. I was a little bit disappointed when I realized that a bunch of valid modern science is just proper experiment design and number crunching. If it's not physics, there's no models of why things work, there's just a p-value on the correlation or some other comparison function. Medicine has turned into a field where you can't know a thing. http://www.cochrane.org/CD005427/BACK_combined-chiropractic-interventions-for-low-back-pain http://www.cochrane.org/CD005427/BACK_combined-chiropractic-... I love reading reports like the above: > There is currently no evidence that supports or refutes that these interventions(chiropractic intervention) provide a clinically meaningful difference for pain or disability in people with [lower back pain] when compared to other interventions. p-values really do not help that much.
- kgwgk 8y ago
  > It's a shame everyone started copying physics but decided for higher acceptance/rejection values. P-values were developed outside of physics, it's not like people took them from physics relaxing the significance thresholds. "If your experiment needs statistics, you ought to have done a better experiment." Lord Ernest Rutherford (maybe)
sykh 8y ago
My favorite probability theory problem is related to this article. You have a test for a disease that is 99% accurate. This means that 99% of the time the test gives a correct result. You test positive for the disease and it is known that 1% of the population has the disease. What is the probability you have the disease? The answer is not at all the one most people think at first when given this problem. This problem is why getting two tests is always a good thing to do when testing positive for a disease. EDIT: I updated the statement of the problem to be one that can be answered!
- joshuamorton 8y ago
  >The answer is not at all the one most people think at first when given this problem. The answer depends on the disease! If it's the common cold, it's probably close to 99%. If it's huntington's disease, the likelyhood is much lower. (when asked, this question is normally posed as "you are given a test, which is 99% accurate, for some rare and deadly disease", the "rare" part is important)
  - sykh 8y ago
    Sorry, you are correct. I'll update the problem.
    - joshuamorton 8y ago
      I'll follow up by mentioning that even as you have it worded now, it's not precise enough to answer. You need to explicitly describe the false positive and negative rates separately. As is, a test that is just "return false" (100% false-negatives, 0% false-positives) will be 99% accurate, but gives no information, whereas a test with 1% false positive rate can have a 1% false negative rate and still be correct 99% of the time, and will provide much more information. Statistics is weird and unintuitive.
      - sykh 8y ago
        I guess I should say that everyone takes the test. I'm not a statistician, I'm a mathematician. The way I understand the phrase, "the test is 99% accurate" is that this means: assuming everyone were to be tested then 99% of the time you get an accurate result. Thus .99(1%) = 0.99% of the people will correctly test positive and 0.01(99%) = 0.99% of the population will incorrectly test positive.
tyrankh 8y ago
I'm not trying to be facetious, but isn't this something you learn in junior-level stats? I had this drilled in in both undergrad math courses and grad machine learning courses; I'm confused to see it warrant an article.
- whatshisface 8y ago
  The harsh reality is that most scientists are not sieving through every statistics book they can get their hands on in order to find out all the reasons they might be wrong. The "individual motivation" to become statistics experts is only present in a few fields, and in the others it is ousted into applied courses taught by other departments. Statistics is directly necessary in ML, so it's a "profit center" and emphasized. In many sciences it's treated like a cost center (something that you need, like IT, but that lies outside of your central expertise.)
  - tyrankh 8y ago
    TIL thanks for the explanation, I guess I never thought about the fact that other STEM fields would not emphasize its meaning.
- pmyteh 8y ago
  It's well known what p-values show. But they are, in practice, used as a gatekeeping mechanism in academic journals in many fields (including mine). Worse, getting p<0.05 is informally seen as a measure of practical significance, rather than simply as one statistical test amongst many passed. So yes, it is something you learn in introductory quantitative methods classes. But I don't think most researchers understand just how much it matters. Also, a key R package for producing regression tables of coefficients for journal articles is called 'stargazer'. Given the unwarranted focus of many readers on those indicia of 'significant' results, I think it's well named. I currently have the opposite problem. Given that I work with very large online datasets (N=1M or so) everything, including the random noise, is statistically significant to p<0.05. It really is effect sizes or busy at that point.
  - 8y ago
learnstats2 8y ago
When I was first taught statistics, I was told that the researcher had to justify a plausible hypothesis first - and then do a hypothesis test/p-value to prove their theory. If this combination of the scientist's intuitive understanding and the p-value test result align, then this is a credible result. On the other hand, the trend now is to conduct every possible test whether or not there is any justification for doing so (corrected for multiple testing, no p-hacking, yes, sure) For example, in tech, we might test every shade of blue. Some of those blues are gonna come up as p-value hits - but since we had no good reason to do this test, this was probably just random noise. Similarly, in genetics, we're gonna test every single gene against everything - just to see what happens (yes, yes, do a Bonferroni correction on each set of tests). Hmm, recent results in genetics don't seem to be very robust or repeatable, for some reason. The likelihood of a truthful link in these tests is incredibly low. When have no particular reason to believe there is a truthful link, and are just blind testing, the false positive rate is very high (as described in the article), and probably even higher than the article speculates with - almost all hits are gonna be false positives. Maybe p-values just don't work well with modern day data. Or, maybe, Big Data just doesn't contain information about mysterious, unexplored, and innovative correlations that we hope it does.
- platz 8y ago
  Yes, Modern day data is not obtained from a controlled clinical trials
- kolpa 8y ago
  If you run an experiment twice and the same shade of blue wins both times, that should persuade you that the winning shade is better. And if you keep replicating the experiment and it keeps winning, that should increase confidence further.
- nonbel 8y ago
  It isn't so much that there is no "truthful link", it is that everything is linked to everything else to some degree and the mathematical models they use for the null hypothesis are just "defaults". These assumptions are almost always violated. The statistical tests detect that, and are providing "true positives".
aaavl2821 8y ago
This was a really interesting article. I've worked with researchers who try to defend a small but statistically significant finding that just doesn't seem likely to be real, and this provides a statistical explanation for my skepticism. The p-value mentality is deeply entrained in a lot of researchers, though The challenge for journal editors seems very real. There's another group that deals with this challenge of interpreting the validity of significant findings for a living, though: biotech VCs. A lot of times trying to reproduce the work is their best way of addressing this, and often the first work done by startups is to try to replicate the academic work. For some other heuristics VCs use to assess "reproducibility risk", see here; https://lifescivc.com/2012/09/scientific-reproducibility-begleys-six-rules/ https://lifescivc.com/2012/09/scientific-reproducibility-beg...
imjustsaying 8y ago
I never understood why people took p-values seriously. They never seemed to mean anything of use. Whenever I brought it up around other academics, no one seemed to want to comment on it. Maybe they were afraid to admit they didn't understand a topic that's apparently important to publishing? Anyone can follow the formula to make a p-value, but there's no requirement to understand its meaning. I'd love to find their use, but I still haven't found it.
- Alex3917 8y ago
  > there's no requirement to understand its meaning. This is a feature, not a bug. The fact that the p-value has nothing to do with whether or not the hypothesis is correct is intended to force people to think for themselves rather than using math as a substitute for logic.
  - k3d 8y ago
    If a feature is working in reverse it's probably a bug though- it might be intended to force independent thinking, but in reality all it does is serve as a gameable goalpost, no thought required.
- kolpa 8y ago
  Before computers could do the complicated math of proper Bayesian updating, it was impossible to get the answer to the question people were asking ("probability of fact given evidence?"), and the science community decided that they'd rather get good answer to bad questions ("probability of evidence given fact?") then bad answers to good questions.
- saalweachter 8y ago
  Well, consider the reverse. A 0.05 p-value might mean anything, but what about a 0.5 p-value? No simple test is going to tell you a hypothesis is definitely correct, but if you can weed out the probably-garbage hypotheses, you can focus on evaluating the remainder more critically.
tw1010 8y ago
Despite using statistics daily, I still feel utterly uncomfortable about its philosophical grounding. Are there any resources HN can suggest to soothe the heart of a sceptic?
- tyrankh 8y ago
  Skeptic * quite different from sceptic :)
  - nkurz 8y ago
    I think it's just an alternate spelling that is preferred in some locales: https://en.wiktionary.org/wiki/sceptic https://en.wiktionary.org/wiki/sceptic
haberman 8y ago
This is especially troubling when combined with confirmation bias. The whole point of data is that it anchors us to reality. Data should be the check that prevents us from believing something simply because we want it to be true. But if we only test theories we already suspect are true, we are already biasing the kinds of false positives we will get.
vcdimension 8y ago
James Abdey wrote his Ph.D. on this subject several year ago and proposed an alternative method for making decisions based on statistical evidence: http://etheses.lse.ac.uk/31/ http://etheses.lse.ac.uk/31/
cassowary37 8y ago
pvals are a lot like the weather - everyone complains about it, nobody does anything about it. Specifically, what tends to be missing from these conversations is a good alternative - the author seems to be asking for false discovery rates/q values. Or maybe effect sizes? The reality is one size doesn't fit all, and the most useful statistic depends on the context. Oh, and the target: good luck submitting your work to a biological journal without pvals. I'm sure the editor will briefly marvel at your courage in taking a stand as she rejects without review. While we're on the subject: there's a tendency to appeal to larger sample sizes, as the author also mentions. Worth remembering that for some of us data isn't a thing you download from the interweb, it's something you generate - and it costs money and time to do so. (And for human subjects research, the stakes are even higher...)
amluto 8y ago
The article says: > Note: this has nothing to do with p-hacking (which is a huge but separate issue). I disagree. p-hacking is when one experimenter checks many statistical tests to find one that is significant. The effect the author is discussing is that many experimenters do many experiments and the significant ones get published. One is more unethical (or maybe just incompetent) than the other, but they’re essentially the same phenomenon.
- anbende 8y ago
  They are the same in that they both create a situation in which the p-value cannot be trusted. However, in one case this is deliberate. In the other it's a problem with the whole enterprise. Also, running multiple tests without correcting for multiple testing (usually by reducing the threshold for significance) is just one form of p-hacking. The more insidious version is when one runs the test after every few participants until random chance makes it "slip over the edge of significance". In that case there might not even be enough variables for multiple testing to have occurred, and it becomes very difficult to detect.
- Fomite 8y ago
  The difference between something that requires a bad actor and one which is an outcome of a system working as intended is pretty huge.
look_lookatme 8y ago
Is there a book (written in plain language) that goes into the history of academic journals and the details the current state of the "replication crisis", "data dredging", etc?
wmnwmn 8y ago
2 solutions: a) stop doing experiments that just look for correlation without any attempt to get at mechanism. Of course sometimes you can't avoid this and then 2) use lower p values. Don't waste thousands or millions (more) dollars following up 5% results.
jssmith 8y ago
> Many researchers are now arguing that we should, more generally, move away from using statistics to make all-or-none decisions and instead use them for "estimation". In other words, instead of asking whether an effect is null or not, we should ask how big the effect is likely to be given the data. I couldn’t agree more with this statement, and even moreso in a business setting than in research. It’s just so easy to get caught up in statistical significance and lose perspective on practical significance. I’ve found confidence intervals most informative and easy to understand.
Karishma1234 8y ago
Sometimes dumbing down a concept can totally screw up a person's learning curve. During early days a lot of Java tutorials (in Indian engineering books) mentioned that the reason Java has Interfaces is because otherwise it is not possible to inherit from multiple classes. While it is true that you can implement multiple interfaces, the whole point of interface is to define "interface" without forcing an implementation. It has nothing to do with the "limitation" of single inheritance. Coming back to p values. A simple google search will find you many articles that say > A small p-value (typically ≤ 0.05) indicates strong evidence against the null hypothesis, so you reject the null hypothesis. The whole idea of p-values is to warn a scientist that they should look for statistical significance. The behaviour of hypothesis over infinite trials is what that matters and hence more data => better reliance. But "more", "better" etc. are subjective ideas and in many cases when everything else considered normal <0.05 might be good but not always. There are far too many factors such as wrong sampling method, things you can not measure vs things you can measure etc. that affect this number truly. I think author nails it when he writes "Replication is the best statistic." Always think of these tests in biological evolution perspective. Do you think this hypothetis would survive the test of time where it has to frequently face the real world ?
- candiodari 8y ago
  It seems to me that a number of things are not clear at all. 1) p-values is a metric on a whole library of procedures, not a single check (although one of them, the Normal distribution is the one usually meant). There's 5 basic ones everyone should know, and you could fill a decent bookshelf with details on all of them. This means a true p-value should be accompanied by a) what the source data was b) how it's distributed, and why (better yet, proof). This should include assurances that there are no attempts to game the measurement (or otherwise any change in the source data directly related to this measurement), as that of course invalidates it. c) a sanity check (like a normality test, or redoing the procedure on generated data and verifying the expected result) d) what the exact claim is (e.g. this is normally distributed with a mean > X) e) what the procedure was to verify this claim (e.g. normality test + mean > X ... you need to COMBINE your p values, because if the data isn't normal your P test is invalid and of course your hypothesis should be rejected even if the numbers in the t-test say it shouldn't) f) a HUGE disclaimer that this measurement and it's actual numerical result only apply to past values, and if it's used to change something it is no longer valid, even if it's numerical value changes as a result, and you can still calculate it. 2) it's not a given that it's possible at all. The truth is that there are many things that don't follow the central limit theorem and therefore cannot correctly be used for statistical approximation. Essentially you need to be utterly convinced that every measurement is the result of somehow combinding a large number of effects that keep reappearing. For instance, planetary orbits do NOT satisfy this criterion. Sure they're the result of large numbers of influences, but almost none of those influences ever repeat (e.g. comets passing in close orbits is a big modifier of orbits, and in 4-5 billion years that'll happen once with any given asteroid. Almost every perturbation is unique and doesn't repeat, so it can't be predicted and won't follow statistical laws. That is the sort of effect that no form of statistics will ever find, and that invalidates your results. Because these are rare it works in the short term, but it doesn't work in the long term. E.g. if you repeat the experiment to determine the speed of light with Jupiter's moons, that won't work with the original data because their orbits have shifted too much)
Malarkey73 8y ago
I'm honestly more tired of essays about p-values than p-values. It's true that like all metrics if it becomes a target then it maybe abused (Goodharts Law). However if you abolished p-values people would start hacking or misunderstanding priors or confidence limits or OR. It's an easy dumb stat that most anyone can do in excel and most everyone recognises. The emphasis should be that it remains a quick shorthand for casual use but that more complex studies have more sophisticated models and probabilistic reasoning. But the emphasis on the p-values is bizarre. As best illustrated by JT Leek the pipeline of data research has multiple points of failure that may lead to false findings or irreproducible research. But we talk very little about them whilst essays about p-values come out every week... https://www.nature.com/news/statistics-p-values-are-just-the-tip-of-the-iceberg-1.17412#/pipe https://www.nature.com/news/statistics-p-values-are-just-the...
keithfma 8y ago
Andrew Gelman's blog provides regular insightful commentary on this issue, I highly recommend it: http://andrewgelman.com/ http://andrewgelman.com/ The post that turned me on to all of this is at: http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/ http://andrewgelman.com/2016/09/21/what-has-happened-down-he...
thanatropism 8y ago
This is an old thread already and I don't know if I'm getting my voice heard. But at any rate: hypothesis testing (slightly different philosophically from p-values, but anyway) is bogus because conjectures-and-refutations falsificationism is bogus. That's not how good science has ever happened, only how bogus research programs have dressed themselves in science. The core of science is "the unity of science". Signal-to-noise measurements tell you very little outside a general coherentist/holistic verificationist framework.
- NPMaxwell 8y ago
  About unity of science: https://en.wikipedia.org/wiki/Unity_of_science https://en.wikipedia.org/wiki/Unity_of_science
rfdearborn 8y ago
I don't think there's any cause to abandon p-values and NHST if you're running experiments with high power and intelligent, deliberate priors. With power = 0.8 and p(h1) = 0.6, p(h0 | p < 0.05) = 0.04. Even if power = 0.8 and p(h1) = 0.2 then p(h0 | p < 0.05) = 0.2.
vadansky 8y ago
Does anyone have that article about how a pro/anti parapsychology both designed a study, analyzed the data, and got conflicting results? (There was a joke about how it was the only paper published that explained a discrepancy by saying the other side cheated)
alan-crowe 8y ago
I remember reading http://andrewgelman.com/2016/11/13/more-on-my-paper-with-john-carlin-on-type-m-and-type-s-errors/ http://andrewgelman.com/2016/11/13/more-on-my-paper-with-joh... with its graph "This is what power = 0.06 looks like". So I got the point that you have to have sufficient statistical power. A useful rule of thumb is that you need a power of at least 0.8. You need to have some idea how big the effect is likely to. Perhaps from previous exploratory research, from claims of other researchers, from reasoning "well, if this is happening the way we think it is, there has to be an effect greater than x waiting to be discovered.". Then you work out how big a sample size you need to use. Then you roll up your sleeves and get down to work. But the reason for using p values rather than Bayesian inference is that it gets you out of the tricky problem of coming up with a prior. You only need to think about the null hypothesis and ask yourself whether the probability of the data, given the null hypothesis, is less than 0.05. So there is a bit of contradiction. p values don't really work unless you ensure that you have sufficient power. To do that you need a plausible effect size, to feed into your power calculation. And that is implicitly an rough approximate prior, 50:50 either null or that effect. You could just do a Bayesian update, stating how much you shifted from 50:50. Basically, if you don't already know enough to have an arguable prior to get a Bayesian approach started, you don't know enough to do a power calculation, so you shouldn't be using p-values either. I went looking on andrewgelman.com for a reference for wanting power = 0.8 and found a more recent post http://andrewgelman.com/2017/12/04/80-power-lie/ http://andrewgelman.com/2017/12/04/80-power-lie/ Oh shit! The situation is much worse than I realised :-(
- thousandautumns 8y ago
  > But the reason for using p values rather than Bayesian inference is that it gets you out of the tricky problem of coming up with a prior. It technically doesn't even do this. Using a frequentist approach is equivalent to a Bayesian approach with an uninformative prior, which is itself an assumption being baked into the analysis, only one that is almost unquestionably incorrect. Its essentially saying you have literally no idea how a data is being generated, which is certainly not true.
stevenjluck 8y ago
Here's a follow-up to the original blog post: https://lucklab.ucdavis.edu/blog/2018/4/28/why-ive-lost-faith-in-p-values-part-2 https://lucklab.ucdavis.edu/blog/2018/4/28/why-ive-lost-fait...

354 points by anacleto 8y ago

The core issue is that p-values are cheaper to get than replicating the study, but replicating the study is the only reliable way to see if it's true or not. Sometimes the expensive/time-consuming way, is the only good way.

lisper 8y ago
Replication by itself is not enough. You need pre-registration too. Otherwise you can p-hack the replications.
- rossdavidh 8y ago
  True, but the article was making a point about "even if we set aside p-hacking for a moment", so I was going with that. But I agree, pre-registration is important as well.
- o2348diuu 8y ago
  Not really disagreeing with you necessarily, but you can hack pre-registration too, in a way that becomes more tenable with smaller studies, which benefit less from pre-registration anyway. The tables are misleading too: of course p(h0=T|p<.05) is going to be about 0.50 if your power is 0.50 and p(h1) = 0.10. That is maybe the bigger problem than p-values per se. Also, he fails to compute the other cell, which is p(h0=T|p>.05), which ends up being about 0.95. The problem in that table is have no power to detect anything. In real life, you don't know p(h0)--that's the point of doing the study. So Bayesianism doesn't really help you. The real problem is lack of power.
- tomrod 8y ago
  Pre-registration?
  - lisper 8y ago
    http://www.apa.org/science/about/psa/2015/08/pre-registration.aspx http://www.apa.org/science/about/psa/2015/08/pre-registratio...

nonbel 8y ago

tldr: The author got a PhD in 1993[1] and is just now figuring out that p-values are not false positive rates [1]http://mindbrain.ucdavis.edu/people/sjluck http://mindbrain.ucdavis.edu/people/sjluck I was lucky and figured it out before getting a degree. Its got to be hard for people in this position to look back on their previous work where the most fundamental aspect of interpreting the results was incorrect. He gets it right that statistics are good for estimation, but there is a part two. You need to come up with a theory that makes a prediction to compare to these estimates, and then test that. Ie, your prediction about the distribution of the results is the "null hypothesis". I think p-values are probably ok for that.

SubiculumCode 8y ago
I've met Steve Luck multiple times over the last decade. He is a very rigorous and insightful scientist, with expertise in a wide range of methods (especially eeg) and psychological phenomena. His wife Prof. Lisa Oaks may be even more impressive in these regards, but that is an aside. The point I would like to make though is that there are lots of aspects to research and stats is just one part to master of many. It should not be up to a neuroscientist to make posts like this. The stats community should be actively pushing the scientific community to alternatives that match interpretative intuition with reality of the statistical metric.
- nonbel 8y ago
  I went through the same thing. I was taught the same half-assed BS statistics with all the wrong interpretations and was surrounded by people just following along with that. Still, it was a point of duty/honor for me to figure out what these statistical results meant. How is interpreting results not a key part of a scientists job? I guess if you don't want to bother learning to interpret results then you can do science by either being a lab tech or theorist.

deleted 8y ago

[deleted]

thaumasiotes 8y ago

Sounds like someone who never understood statistics, still doesn't, and doesn't want to. A particularly glaring issue is this offhand comment: > this is a statement about what happens when the null hypothesis is actually true. In real research, we don't know whether the null hypothesis is actually true. If we knew that, we wouldn't need any statistics! In real research, we have a p value, and we want to know whether we should accept or reject the null hypothesis. That isn't a question that any statistical approach will help you with. There's a reason we talk in terms of "rejecting" or "failing to reject" a hypothesis. We don't do statistical tests to accept hypotheses, only to reject them. The concept of accepting one hypothesis based on a comparison between it and one other hypothesis is ludicrous on its face, suffering exactly the problems associated with Pascal's wager.

nonbel 8y ago
>"That isn't a question that any statistical approach will help you with. There's a reason we talk in terms of "rejecting" or "failing to reject" a hypothesis. We don't do statistical tests to accept hypotheses, only to reject them." In (Neyman-Pearson's) hypothesis testing you definitely can accept an hypothesis. This is not true for (Fisher's) significance testing though. This paper is probably fine to explain it: https://onlinelibrary.wiley.com/doi/pdf/10.1002/0470013192.bsa438 https://onlinelibrary.wiley.com/doi/pdf/10.1002/0470013192.b...
s17n 8y ago
He clearly meant "fail to reject" where he said "accept", you're quibbling about semantics. The whole post is about how p-values (and indeed, known statistical techniques) don't actually help you decide whether or not you should reject the null.
- thaumasiotes 8y ago
  The post is about a confusion between the question that p-values address ("how likely is this data to have come from the null hypothesis?") and another very different question ("given several findings that were unlikely to have come from various different hypotheses, how many are likely to be spurious?").

wrp 8y ago

One of the best articles covering this issues is Meehl[1][2]. You can find discussion in various places like Gelman[3] and Reinhart[4]. [1] Meehl, Paul E (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66(1), 195–244. [2] http://meehl.umn.edu/files/144whysummariespdf http://meehl.umn.edu/files/144whysummariespdf [3] http://andrewgelman.com/2015/03/23/paul-meehl-continues-boss/ http://andrewgelman.com/2015/03/23/paul-meehl-continues-boss... [4] https://www.refsmmat.com/notebooks/meehl.html https://www.refsmmat.com/notebooks/meehl.html

deleted 8y ago

[deleted]

tprice7 8y ago

'The fundamental problem is that p values don't mean what we "need" them to mean, that is p(null | significant effect).' From Bayes' theorem, this more useful probability is given by p * x, where x = p(null) / p(significant effect). Maybe we could just lower the accepted threshold for statistical significance by several orders of magnitude so that, for statistically significant p, p * x is still small even for careful (i.e. big) estimates of x (e.g. maybe a Fermi approximation of the total number of experiments ever performed in the field in question). This doesn't necessarily imply impractically big sample sizes, although obviously this depends on the specifics (I believe the p value for a given value of the t-statistic decays exponentially with sample size).

kkylin 8y ago
This was basically the suggestion here: https://www.nature.com/articles/s41562-017-0189-z.epdf https://www.nature.com/articles/s41562-017-0189-z.epdf Also previous HN discussion: https://news.ycombinator.com/item?id=15192610 https://news.ycombinator.com/item?id=15192610
nonbel 8y ago
I don't follow your argument. You've got two premises: 1) You are saying that people are committing the transposing the conditional fallacy: p(H0|data) != p(data|H0): - OK 2) You say to use Bayes theorem to get the value we want: - OK, but actually a better formulation is p(H_0|data) = p(H_0)*p(data|H_0)/[p(H_0)*p(data|H_0) + p(H_1)*p(data|H_1) + ... + p(H_n)*p(data|H_n)] You probably don't need to add up all the way to hypothesis n since the terms eventually become negligible and can be dropped from the denominator. The point is that you have to compare how likely the result would be under other hypotheses, not just H_0. 3) You propose lowering the threshold for "significance" - How does this follow from the premises? Lets say you get a very low value for p(H_0)p(data|H_0), this can still be much higher than p(H_1)p(data|H_1), etc so it is still the best choice. Ie, you can get a low p-value given H_0 but if there is no better model out there you should still keep H_0.
- edwinksl 8y ago
  Minor point, but you are missing p(H_0)*p(data|H_0) in the denominator in bullet point 2.

piotrkaminski 8y ago

> instead of asking whether an effect is null or not, we should ask how big the effect is likely to be given the data. However, at the end of the day, editors need to make an all-or-none decision about whether to publish a paper Yet another way in which the traditional publishing structure actively harms science.

Bromskloss 8y ago
Do you have an alternative way of publishing in mind?
- piotrkaminski 8y ago
  My criticism is not necessarily constructive. ;) But it's not too hard to imagine something along the lines of arXiv combined with a rating/commenting system not unlike HN itself, combined with a Facebook-ish algorithm for surfacing articles relevant to each reader. The devil is in the details, of course, but it shouldn't be hard to do better than the expensive, artificially constrained, and arbitrary system we're saddled with now. The real trick will be convincing entrenched academics to switch -- I'm still not convinced this is actually possible.

davidxc 8y ago

Here's a more simple thought experiment that gets across the point of why p(null | significant effect) /= p(significant effect | null), and why p-values are flawed as stated in the post. Imagine a society where scientists are really, really bad at hypothesis generation. In fact, they're so bad that they only test null hypothesis that are true. So in this hypothetical society, the null hypothesis in any scientific experiment ever done is true. But statistically using a p value of 0.05, we'll still reject the null in 5% of experiments. And those experiments will then end up being published in scientific literature. But then this society's scientific literature now only contains false results - literally all published scientific results are false. Of course, in real life, we hope that our scientists have better intuition for what is in fact true - that is, we hope that the "prior" probability in Bayes' theorem, p(null), is not 1.

danieltillett 8y ago
I don’t have to imagine that society since they already exist and are called social scientists. More seriously you do make a good point which is all scientists lie on a spectrum from always generating true hypotheses, to always generating false hypotheses. Scientists in different fields tend to lie more to one or the other of the extremes. My experience is the observational sciences are more shifted to the always false end than the experimental sciences.
- modbait 8y ago
  This is being down-voted for the shots fired, but the underlying point is almost certainly true to a degree. People aren't ideologically invested in (say) the weight of electrons in the same way that they are in IQ curves across demographic groups.
  - danieltillett 8y ago
    Ideology certainly plays an important role in generating false hypothesis, but all the observational sciences suffer from the problem that you can't run experiments to rigorously test the robustness of your hypothesis. In the experimental sciences you can get far using the rule of thumb that if you need statistics you did the wrong experiment, while in the observational sciences the use of statistics is inherent.

btilly 8y ago

Here is a better way to think about this. The proper role of data is to update our existing beliefs about the world. It is not to specify what our beliefs should be. The question that we really want to answer is, "What is the probability that X is true?" What p-values do is replace that with the seemingly similar but very different, "What is the probability that I'd have the evidence I have against X by chance alone were X true?" Bayesian factors try to capture the idea of how much belief should shift. The conclusion at the end is that replication is better than either approach. I agree. We know that there are a lot of ways to hack p-values. Bayesian factors haven't caught on because they don't match how people want to think. However if we keep consistent research standards, and replicate routinely, the replication rate gives us a sense of how much confidence we should have in a new result that we hear about. (Spoiler. A lot less confidence than most breathless science reporting would have you believe.)

analog31 8y ago
My fear is that in 10 years time, we will have learned to hack Bayesian factors.
- btilly 8y ago
  The way that you hack Bayesian factors is selectively including data. And the selection process can be as simple as publication bias causing some results to be published and others not. This is the fundamental weakness of meta analysis.
mycall 8y ago
> The proper role of data is to update our existing beliefs about the world. It is not to specify what our beliefs should be. Create the schema beforehand, I get that. But feature extraction does work, providing models from data. Sometimes takes much time to analyze and understand the models.
- btilly 8y ago
  If you have enough data and a strong enough signal, then all reasonable belief systems should converge on the same answer. Do not let that fact fool you into believing that raw data is the only thing necessary to make good decisions when faced with realistic situations.

btcindivist 8y ago

p-values that are not in the physics ranges are ridiculous. It's a shame everyone started copying physics but decided for higher acceptance/rejection values. I was a little bit disappointed when I realized that a bunch of valid modern science is just proper experiment design and number crunching. If it's not physics, there's no models of why things work, there's just a p-value on the correlation or some other comparison function. Medicine has turned into a field where you can't know a thing. http://www.cochrane.org/CD005427/BACK_combined-chiropractic-interventions-for-low-back-pain http://www.cochrane.org/CD005427/BACK_combined-chiropractic-... I love reading reports like the above: > There is currently no evidence that supports or refutes that these interventions(chiropractic intervention) provide a clinically meaningful difference for pain or disability in people with [lower back pain] when compared to other interventions. p-values really do not help that much.

kgwgk 8y ago
> It's a shame everyone started copying physics but decided for higher acceptance/rejection values. P-values were developed outside of physics, it's not like people took them from physics relaxing the significance thresholds. "If your experiment needs statistics, you ought to have done a better experiment." Lord Ernest Rutherford (maybe)

sykh 8y ago

My favorite probability theory problem is related to this article. You have a test for a disease that is 99% accurate. This means that 99% of the time the test gives a correct result. You test positive for the disease and it is known that 1% of the population has the disease. What is the probability you have the disease? The answer is not at all the one most people think at first when given this problem. This problem is why getting two tests is always a good thing to do when testing positive for a disease. EDIT: I updated the statement of the problem to be one that can be answered!

joshuamorton 8y ago
>The answer is not at all the one most people think at first when given this problem. The answer depends on the disease! If it's the common cold, it's probably close to 99%. If it's huntington's disease, the likelyhood is much lower. (when asked, this question is normally posed as "you are given a test, which is 99% accurate, for some rare and deadly disease", the "rare" part is important)
- sykh 8y ago
  Sorry, you are correct. I'll update the problem.
  - joshuamorton 8y ago
    I'll follow up by mentioning that even as you have it worded now, it's not precise enough to answer. You need to explicitly describe the false positive and negative rates separately. As is, a test that is just "return false" (100% false-negatives, 0% false-positives) will be 99% accurate, but gives no information, whereas a test with 1% false positive rate can have a 1% false negative rate and still be correct 99% of the time, and will provide much more information. Statistics is weird and unintuitive.
    - sykh 8y ago
      I guess I should say that everyone takes the test. I'm not a statistician, I'm a mathematician. The way I understand the phrase, "the test is 99% accurate" is that this means: assuming everyone were to be tested then 99% of the time you get an accurate result. Thus .99(1%) = 0.99% of the people will correctly test positive and 0.01(99%) = 0.99% of the population will incorrectly test positive.

tyrankh 8y ago

I'm not trying to be facetious, but isn't this something you learn in junior-level stats? I had this drilled in in both undergrad math courses and grad machine learning courses; I'm confused to see it warrant an article.

whatshisface 8y ago
The harsh reality is that most scientists are not sieving through every statistics book they can get their hands on in order to find out all the reasons they might be wrong. The "individual motivation" to become statistics experts is only present in a few fields, and in the others it is ousted into applied courses taught by other departments. Statistics is directly necessary in ML, so it's a "profit center" and emphasized. In many sciences it's treated like a cost center (something that you need, like IT, but that lies outside of your central expertise.)
- tyrankh 8y ago
  TIL thanks for the explanation, I guess I never thought about the fact that other STEM fields would not emphasize its meaning.
pmyteh 8y ago
It's well known what p-values show. But they are, in practice, used as a gatekeeping mechanism in academic journals in many fields (including mine). Worse, getting p<0.05 is informally seen as a measure of practical significance, rather than simply as one statistical test amongst many passed. So yes, it is something you learn in introductory quantitative methods classes. But I don't think most researchers understand just how much it matters. Also, a key R package for producing regression tables of coefficients for journal articles is called 'stargazer'. Given the unwarranted focus of many readers on those indicia of 'significant' results, I think it's well named. I currently have the opposite problem. Given that I work with very large online datasets (N=1M or so) everything, including the random noise, is statistically significant to p<0.05. It really is effect sizes or busy at that point.
- 8y ago

learnstats2 8y ago

When I was first taught statistics, I was told that the researcher had to justify a plausible hypothesis first - and then do a hypothesis test/p-value to prove their theory. If this combination of the scientist's intuitive understanding and the p-value test result align, then this is a credible result. On the other hand, the trend now is to conduct every possible test whether or not there is any justification for doing so (corrected for multiple testing, no p-hacking, yes, sure) For example, in tech, we might test every shade of blue. Some of those blues are gonna come up as p-value hits - but since we had no good reason to do this test, this was probably just random noise. Similarly, in genetics, we're gonna test every single gene against everything - just to see what happens (yes, yes, do a Bonferroni correction on each set of tests). Hmm, recent results in genetics don't seem to be very robust or repeatable, for some reason. The likelihood of a truthful link in these tests is incredibly low. When have no particular reason to believe there is a truthful link, and are just blind testing, the false positive rate is very high (as described in the article), and probably even higher than the article speculates with - almost all hits are gonna be false positives. Maybe p-values just don't work well with modern day data. Or, maybe, Big Data just doesn't contain information about mysterious, unexplored, and innovative correlations that we hope it does.

platz 8y ago
Yes, Modern day data is not obtained from a controlled clinical trials
kolpa 8y ago
If you run an experiment twice and the same shade of blue wins both times, that should persuade you that the winning shade is better. And if you keep replicating the experiment and it keeps winning, that should increase confidence further.
nonbel 8y ago
It isn't so much that there is no "truthful link", it is that everything is linked to everything else to some degree and the mathematical models they use for the null hypothesis are just "defaults". These assumptions are almost always violated. The statistical tests detect that, and are providing "true positives".

aaavl2821 8y ago

This was a really interesting article. I've worked with researchers who try to defend a small but statistically significant finding that just doesn't seem likely to be real, and this provides a statistical explanation for my skepticism. The p-value mentality is deeply entrained in a lot of researchers, though The challenge for journal editors seems very real. There's another group that deals with this challenge of interpreting the validity of significant findings for a living, though: biotech VCs. A lot of times trying to reproduce the work is their best way of addressing this, and often the first work done by startups is to try to replicate the academic work. For some other heuristics VCs use to assess "reproducibility risk", see here; https://lifescivc.com/2012/09/scientific-reproducibility-begleys-six-rules/ https://lifescivc.com/2012/09/scientific-reproducibility-beg...

imjustsaying 8y ago

I never understood why people took p-values seriously. They never seemed to mean anything of use. Whenever I brought it up around other academics, no one seemed to want to comment on it. Maybe they were afraid to admit they didn't understand a topic that's apparently important to publishing? Anyone can follow the formula to make a p-value, but there's no requirement to understand its meaning. I'd love to find their use, but I still haven't found it.

Alex3917 8y ago
> there's no requirement to understand its meaning. This is a feature, not a bug. The fact that the p-value has nothing to do with whether or not the hypothesis is correct is intended to force people to think for themselves rather than using math as a substitute for logic.
- k3d 8y ago
  If a feature is working in reverse it's probably a bug though- it might be intended to force independent thinking, but in reality all it does is serve as a gameable goalpost, no thought required.
kolpa 8y ago
Before computers could do the complicated math of proper Bayesian updating, it was impossible to get the answer to the question people were asking ("probability of fact given evidence?"), and the science community decided that they'd rather get good answer to bad questions ("probability of evidence given fact?") then bad answers to good questions.
saalweachter 8y ago
Well, consider the reverse. A 0.05 p-value might mean anything, but what about a 0.5 p-value? No simple test is going to tell you a hypothesis is definitely correct, but if you can weed out the probably-garbage hypotheses, you can focus on evaluating the remainder more critically.

tw1010 8y ago

Despite using statistics daily, I still feel utterly uncomfortable about its philosophical grounding. Are there any resources HN can suggest to soothe the heart of a sceptic?

tyrankh 8y ago
Skeptic * quite different from sceptic :)
- nkurz 8y ago
  I think it's just an alternate spelling that is preferred in some locales: https://en.wiktionary.org/wiki/sceptic https://en.wiktionary.org/wiki/sceptic

haberman 8y ago

This is especially troubling when combined with confirmation bias. The whole point of data is that it anchors us to reality. Data should be the check that prevents us from believing something simply because we want it to be true. But if we only test theories we already suspect are true, we are already biasing the kinds of false positives we will get.

vcdimension 8y ago

James Abdey wrote his Ph.D. on this subject several year ago and proposed an alternative method for making decisions based on statistical evidence: http://etheses.lse.ac.uk/31/ http://etheses.lse.ac.uk/31/

cassowary37 8y ago

pvals are a lot like the weather - everyone complains about it, nobody does anything about it. Specifically, what tends to be missing from these conversations is a good alternative - the author seems to be asking for false discovery rates/q values. Or maybe effect sizes? The reality is one size doesn't fit all, and the most useful statistic depends on the context. Oh, and the target: good luck submitting your work to a biological journal without pvals. I'm sure the editor will briefly marvel at your courage in taking a stand as she rejects without review. While we're on the subject: there's a tendency to appeal to larger sample sizes, as the author also mentions. Worth remembering that for some of us data isn't a thing you download from the interweb, it's something you generate - and it costs money and time to do so. (And for human subjects research, the stakes are even higher...)

amluto 8y ago

The article says: > Note: this has nothing to do with p-hacking (which is a huge but separate issue). I disagree. p-hacking is when one experimenter checks many statistical tests to find one that is significant. The effect the author is discussing is that many experimenters do many experiments and the significant ones get published. One is more unethical (or maybe just incompetent) than the other, but they’re essentially the same phenomenon.

anbende 8y ago
They are the same in that they both create a situation in which the p-value cannot be trusted. However, in one case this is deliberate. In the other it's a problem with the whole enterprise. Also, running multiple tests without correcting for multiple testing (usually by reducing the threshold for significance) is just one form of p-hacking. The more insidious version is when one runs the test after every few participants until random chance makes it "slip over the edge of significance". In that case there might not even be enough variables for multiple testing to have occurred, and it becomes very difficult to detect.
Fomite 8y ago
The difference between something that requires a bad actor and one which is an outcome of a system working as intended is pretty huge.

look_lookatme 8y ago

Is there a book (written in plain language) that goes into the history of academic journals and the details the current state of the "replication crisis", "data dredging", etc?

wmnwmn 8y ago

2 solutions: a) stop doing experiments that just look for correlation without any attempt to get at mechanism. Of course sometimes you can't avoid this and then 2) use lower p values. Don't waste thousands or millions (more) dollars following up 5% results.

jssmith 8y ago

> Many researchers are now arguing that we should, more generally, move away from using statistics to make all-or-none decisions and instead use them for "estimation". In other words, instead of asking whether an effect is null or not, we should ask how big the effect is likely to be given the data. I couldn’t agree more with this statement, and even moreso in a business setting than in research. It’s just so easy to get caught up in statistical significance and lose perspective on practical significance. I’ve found confidence intervals most informative and easy to understand.

Karishma1234 8y ago

Sometimes dumbing down a concept can totally screw up a person's learning curve. During early days a lot of Java tutorials (in Indian engineering books) mentioned that the reason Java has Interfaces is because otherwise it is not possible to inherit from multiple classes. While it is true that you can implement multiple interfaces, the whole point of interface is to define "interface" without forcing an implementation. It has nothing to do with the "limitation" of single inheritance. Coming back to p values. A simple google search will find you many articles that say > A small p-value (typically ≤ 0.05) indicates strong evidence against the null hypothesis, so you reject the null hypothesis. The whole idea of p-values is to warn a scientist that they should look for statistical significance. The behaviour of hypothesis over infinite trials is what that matters and hence more data => better reliance. But "more", "better" etc. are subjective ideas and in many cases when everything else considered normal <0.05 might be good but not always. There are far too many factors such as wrong sampling method, things you can not measure vs things you can measure etc. that affect this number truly. I think author nails it when he writes "Replication is the best statistic." Always think of these tests in biological evolution perspective. Do you think this hypothetis would survive the test of time where it has to frequently face the real world ?

candiodari 8y ago
It seems to me that a number of things are not clear at all. 1) p-values is a metric on a whole library of procedures, not a single check (although one of them, the Normal distribution is the one usually meant). There's 5 basic ones everyone should know, and you could fill a decent bookshelf with details on all of them. This means a true p-value should be accompanied by a) what the source data was b) how it's distributed, and why (better yet, proof). This should include assurances that there are no attempts to game the measurement (or otherwise any change in the source data directly related to this measurement), as that of course invalidates it. c) a sanity check (like a normality test, or redoing the procedure on generated data and verifying the expected result) d) what the exact claim is (e.g. this is normally distributed with a mean > X) e) what the procedure was to verify this claim (e.g. normality test + mean > X ... you need to COMBINE your p values, because if the data isn't normal your P test is invalid and of course your hypothesis should be rejected even if the numbers in the t-test say it shouldn't) f) a HUGE disclaimer that this measurement and it's actual numerical result only apply to past values, and if it's used to change something it is no longer valid, even if it's numerical value changes as a result, and you can still calculate it. 2) it's not a given that it's possible at all. The truth is that there are many things that don't follow the central limit theorem and therefore cannot correctly be used for statistical approximation. Essentially you need to be utterly convinced that every measurement is the result of somehow combinding a large number of effects that keep reappearing. For instance, planetary orbits do NOT satisfy this criterion. Sure they're the result of large numbers of influences, but almost none of those influences ever repeat (e.g. comets passing in close orbits is a big modifier of orbits, and in 4-5 billion years that'll happen once with any given asteroid. Almost every perturbation is unique and doesn't repeat, so it can't be predicted and won't follow statistical laws. That is the sort of effect that no form of statistics will ever find, and that invalidates your results. Because these are rare it works in the short term, but it doesn't work in the long term. E.g. if you repeat the experiment to determine the speed of light with Jupiter's moons, that won't work with the original data because their orbits have shifted too much)

Malarkey73 8y ago

I'm honestly more tired of essays about p-values than p-values. It's true that like all metrics if it becomes a target then it maybe abused (Goodharts Law). However if you abolished p-values people would start hacking or misunderstanding priors or confidence limits or OR. It's an easy dumb stat that most anyone can do in excel and most everyone recognises. The emphasis should be that it remains a quick shorthand for casual use but that more complex studies have more sophisticated models and probabilistic reasoning. But the emphasis on the p-values is bizarre. As best illustrated by JT Leek the pipeline of data research has multiple points of failure that may lead to false findings or irreproducible research. But we talk very little about them whilst essays about p-values come out every week... https://www.nature.com/news/statistics-p-values-are-just-the-tip-of-the-iceberg-1.17412#/pipe https://www.nature.com/news/statistics-p-values-are-just-the...

keithfma 8y ago

Andrew Gelman's blog provides regular insightful commentary on this issue, I highly recommend it: http://andrewgelman.com/ http://andrewgelman.com/ The post that turned me on to all of this is at: http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/ http://andrewgelman.com/2016/09/21/what-has-happened-down-he...

thanatropism 8y ago

This is an old thread already and I don't know if I'm getting my voice heard. But at any rate: hypothesis testing (slightly different philosophically from p-values, but anyway) is bogus because conjectures-and-refutations falsificationism is bogus. That's not how good science has ever happened, only how bogus research programs have dressed themselves in science. The core of science is "the unity of science". Signal-to-noise measurements tell you very little outside a general coherentist/holistic verificationist framework.

NPMaxwell 8y ago
About unity of science: https://en.wikipedia.org/wiki/Unity_of_science https://en.wikipedia.org/wiki/Unity_of_science

rfdearborn 8y ago

I don't think there's any cause to abandon p-values and NHST if you're running experiments with high power and intelligent, deliberate priors. With power = 0.8 and p(h1) = 0.6, p(h0 | p < 0.05) = 0.04. Even if power = 0.8 and p(h1) = 0.2 then p(h0 | p < 0.05) = 0.2.

vadansky 8y ago

Does anyone have that article about how a pro/anti parapsychology both designed a study, analyzed the data, and got conflicting results? (There was a joke about how it was the only paper published that explained a discrepancy by saying the other side cheated)

alan-crowe 8y ago

I remember reading http://andrewgelman.com/2016/11/13/more-on-my-paper-with-john-carlin-on-type-m-and-type-s-errors/ http://andrewgelman.com/2016/11/13/more-on-my-paper-with-joh... with its graph "This is what power = 0.06 looks like". So I got the point that you have to have sufficient statistical power. A useful rule of thumb is that you need a power of at least 0.8. You need to have some idea how big the effect is likely to. Perhaps from previous exploratory research, from claims of other researchers, from reasoning "well, if this is happening the way we think it is, there has to be an effect greater than x waiting to be discovered.". Then you work out how big a sample size you need to use. Then you roll up your sleeves and get down to work. But the reason for using p values rather than Bayesian inference is that it gets you out of the tricky problem of coming up with a prior. You only need to think about the null hypothesis and ask yourself whether the probability of the data, given the null hypothesis, is less than 0.05. So there is a bit of contradiction. p values don't really work unless you ensure that you have sufficient power. To do that you need a plausible effect size, to feed into your power calculation. And that is implicitly an rough approximate prior, 50:50 either null or that effect. You could just do a Bayesian update, stating how much you shifted from 50:50. Basically, if you don't already know enough to have an arguable prior to get a Bayesian approach started, you don't know enough to do a power calculation, so you shouldn't be using p-values either. I went looking on andrewgelman.com for a reference for wanting power = 0.8 and found a more recent post http://andrewgelman.com/2017/12/04/80-power-lie/ http://andrewgelman.com/2017/12/04/80-power-lie/ Oh shit! The situation is much worse than I realised :-(

thousandautumns 8y ago
> But the reason for using p values rather than Bayesian inference is that it gets you out of the tricky problem of coming up with a prior. It technically doesn't even do this. Using a frequentist approach is equivalent to a Bayesian approach with an uninformative prior, which is itself an assumption being baked into the analysis, only one that is almost unquestionably incorrect. Its essentially saying you have literally no idea how a data is being generated, which is certainly not true.

stevenjluck 8y ago

Here's a follow-up to the original blog post: https://lucklab.ucdavis.edu/blog/2018/4/28/why-ive-lost-faith-in-p-values-part-2 https://lucklab.ucdavis.edu/blog/2018/4/28/why-ive-lost-fait...