10 ms·
What a debate about p-values shows about science
- paulddraper 9y ago> The case against p<.05 > The case against p<.005 > The real problem isn’t with statistical significance; it’s with the culture of science Right. The substance happens in the last 10% of the article. ...and there isn't any solution or recommendation. Unfortunately, I think this article is really long way of saying, "There exists p-hacking". I was hoping for more :(
- BenoitEssiambre 9y agoThey should change the nomenclature "significant", which in normal language tends to imply a large size to statistically "detectable" or "discernible". P=0.05 is best interpreted as "There is a 95% chance that this data isn't pure noise". Then researchers should always put more emphasis on confidence intervals. Readers of papers can then see if the size is enough to make the results relevant and not likely caused by experimental accident. Plus given the perverse incentives researchers are subject to, maybe assume that the real effect size is probably closer to the lower bound of the interval. Null hypotheses have limited usefulness even at p=0.005. A tiny systematic bias in the experiment can make it cross that threshold and there are _always_ at least small biases. These can be caused by not exactly calibrated instruments or small differences in the way tests are performed by different researchers on the team etc. Inherent in null hypothesis testing is this ridiculous assumption that there are no systematic biases. This is its fatal flaw. The null hypothesis test never tells you that the effect is zero or that it is any other value, it only gives you an hint about whether or not the data is too noisy to say anything at all.
- Retric 9y ago"There is a 95% chance that this data isn't pure noise". That's not what P=0.05 actually means. It's much closer to, this result rises above the noise floor. Suggesting but not implying a possible link between X and Y, fell free to investigate.
- tedsanders 9y ago>P=0.05 is best interpreted as "There is a 95% chance that this data isn't pure noise". No, that's not what P=0.05 means. This misconception is exactly why p values are so problematic. The question we want to answer is unanswerable, so instead we subtly substitute a different question in its place and don't even realize we've swapped concepts.
- BenoitEssiambre 9y agoWhen I say that "There is a 95% chance that this data isn't pure noise" I'm being slightly indirect. More directly it would be something like "There is 5% chance that pure noise generator you would give you this result" but since I want to reason about the actual world that the experiment is trying to study, I think it is legitimate to do the Bayesian flip and infer the more useful first statement that the world is likely too noisy to detect an effect using this experiment.
- andy_wrote 9y agoI think I understand what you're trying to get at, but I don't think describing a p-value in terms of "pure noise" is quite right. The null hypothesis may well carry with it all sorts of specific assumptions: independence, finite variance, maybe even specific probability distributions (and not necessarily the normal distribution). Typically somewhere in there we define a θ parameter and then include something like θ = 0 that provides grounds for possible rejection. We think of "θ = 0" as the null hypothesis, but all the other assumptions are part of the null hypothesis as well; the statement "θ = 0" does not make sense without them.
- nonbel 9y agoIf you see clouds how likely is rain? If you see rain how likely is it there are clouds? Are the answers to these at all the same? No. There is absolutely nothing valid about your "flip" (called transposing the conditional) and the damage done to the human species by people doing the "flip" is too huge for most to comprehend. Sorry, but I really cannot overstate the problems with this "flip". Perhaps people who get paid by grants should get three chances regarding this flip. If they strike out they are banned from ever being funded by NIH, etc for life. The result (either 99+% of current researchers would be culled, or p-values would become appropriately rare) would do more to advance science than any other idea I have seen.
- ak_yo 9y ago> P=0.05 is best interpreted as "There is a 95% chance that this data isn't pure noise". This is incorrect. A p-value is the probability of observing a value of the test statistic that is at least as extreme as the reported value, assuming the null hypothesis is true.
- nils-m-holm 9y agoWhich sounds like a fancy way of saying that "there is a 5% chance of the data being noise (while the null hypothesis is true)".
- imh 9y ago"there is a 5% chance of the data being noise" is a statement on the probability of whether the phenomenon is not real given the data: p(it's just noise | data). A p value is a statement on the probability we'd see this data given that the phenomenon isn't real: p(data | it's just noise). Now, the exact values of a given data set are vanishingly unlikely, so we actually ask about the probability we'd say this data or even more extreme data given that we're just looking at noise. That's a typical p value.
- nils-m-holm 9y agoThat's what I meant: P(data | H0). So there is a 5% probability for the data to arise by chance (being noise) given H0 is true. The whole terminology with values being "extreme" is just unintuitive and unwieldy, IMO.
- DamonHD 9y agoRight at the end is a key thing: reward failure better. Publish negative results. I'm kinda embarrassed by my negative results, such as they are, but I'm clear in this latest case that I wanted the method reviewed most, since others then may be able to use it. (Warning: not a real scientist, but being validated by real scientists in the case in question.)
- tenaciousDaniel 9y agoI'm not a scientist, but this is very surprising to me. I would have imagined that our scientific culture would be one where any result is recognized as something from which valuable knowledge can be gained. The idea that unanticipated results reflects some failure on the part of the researcher seems so strange and unscientific. But again, I'm not a scientist.
- allenz 9y agoIn an ideal world, scientists would be judged solely on their study design and execution without considering the outcome. But that takes a lot of effort. It's much easier to judge based on the results that the scientist achieved ("I discovered the priming effect"). We remember Watson and Crick, and forget many other scientists who worked just as hard.
- KGIII 9y agoScience doesn't really have a place for participation trophies. That and it's pretty natural to judge based on results. When handing out recognition and fame, the results are probably the best criteria - though that should include those who helped and that each stood on the shoulders of giants. That information should be included in the papers for those who are curious.
- allenz 9y agoNegative results are real science, and science is collaborative. When we don't incentivize negative results, scientists don't take the time to analyze and share those findings, which results in a shallower understanding and possibly wasted effort by other scientists. It also biases scientists towards claiming results where none exist (e.g. p-hacking).
- searine 9y agoOnly badly designed experiments overrely on p-values. A robustly designed experiment should produce repeatable, intuitive results. Anyone looking at the presented data should be able to reason the result without needing to conduct a statistical test. Obviously, these tests are important, but people focus too much on the p-value rather than designing an experiment which will produce a meaningful result (regardless of whether the result is positive or negative). Proper experimental design is like a cake, and p-values are the frosting. Everybody wants cake, most often with frosting. Nobody wants frosting without the cake, or at least those that do have questionable taste.
- tlb 9y agoNot all phenomena are strong enough to allow such experiments. Some make only a small difference to an inherently noisy process. These can only be discovered with statistical tests on substantial samples.
- toufka 9y agoCertainly, but a well-designed experiment can identify the wedges that differentiate hypothesis A from hypothesis B without requiring a statistical analysis of noisy data. And true, there are certain kinds of experiments that can only be done statistically. However, depending on the field, it can just be easier to do, and really should just be considered lazy science - again the details matter. In my field (biochemistry/genetics) there really is a very real effort to design experiments that do not require statistics to verify the result. And that effort in design is rewarded amply when the result of the experiment becomes quite clearly binary in nature, from multiple angles, rather that a statistical "probably".
- tlb 9y agoAre there many real examples of studies where statistical tests are needed to see the effect, but a different experiment could demonstrate the same effect without any statistics? I haven't been able to think of one in the fields I know well.
- justwantaccount 9y agoI personally find NHST suspicious, even if the p-value is less than 0.005. It means that, ASSUMING that the null hypothesis is correct, the probability of observing the data is less than 0.5%. That's still not zero, though. For example, if I try to decide that native Hawaiians are US citizens, and the null hypothesis is that they are, but since only ~0.2% of total US population is native Hawaiian, NHST would conclude that native Hawaiians aren't US citizens. Conversely, the p-value being high isn't really meaningful either. For example, if I try to decide if a white person is from South Africa, and the null hypothesis is that they are, and since ~9% of the population in South Africa is white, I would fail to reject the null hypothesis, even if the person's actually from Europe or literally any other country in the world. p-values are dependent on the sample size, too, and become more and more sensitive to smaller differences in the ASSUMED population distribution(s), when the effect size can in fact be small. NSHT seems highly dependent on quality null hypothesis that's correct, which you can't really establish since that's what you're trying to find out. So the approach doesn't really conclude anything either way, and just really seems weak as an approach to scientific testing overall.
- majormajor 9y agoI don't really follow. Your examples seem like weird or poorly thought out experiments (well, surveys, not experiments), not anything to do with significance testing. If I wanted to see if native Hawaiians are citizens and didn't have access to law, real data, anything like that, I'd sample from the population of Hawaiians, not citizens as a whole, and then regardless of the null hypothesis being "are citizens" or "aren't citizens" I think the results would be overwhelmingly one-sided enough to work out. And the South Africa example is asking about 1 single person? That not a question of science, then, just a question?
- justwantaccount 9y agoMaybe my example was contrived, but my main argument was that p-values represent P(D|H), or probability seeing your data conditioned on assuming that the hypothesis is correct, and that approach seems inherently flawed to me. If you're doing an experiment, p-values don't say anything about if the hypothesis is true or false, in fact the approach can't theoretically give that information. If you're doing a basic t-test, it goes something like this: 1. You have a null hypothesis you assume to be true 2. Based on this null hypothesis and the central limit theorem, if you theoretically conduct many experiments you expect some summary statistic to have a specific distribution around the 'true' population summary statistic and 'true' population variance, which you don't have so you assume it to be the null hypothesis 3. You compare your experiment's summary statistic to the hypothesized distribution. If the probability of seeing your summary statistic is below some threshold, you say it's unlikely to see this summary statistic again. But the probability of seeing that summary statistic again actually depends on P(H), which NHST doesn't provide any information on. My examples were meant to highlight the nature of conditional probabilities, rather than how real life experiments are conducted.
- claytonjy 9y agoPretty good take on this for a general-purpose audience. I was ready to complain about this line for implying P(H|D) (rather than P(D|H_0), as actually occurs): > The researcher basically asks: How ridiculous would it be to believe the null hypothesis is the true answer, given the results we’re seeing? but I think they (indirectly) correct it pretty well in the next section. It isn't published yet, but Gelman and co's recent piece on this seems like an important contribution to the debate [1]. Short version: take the p-value down from it's pedestal, abandon the dichotomous view of is/isn't significant, and consider it with all the other evidence and data. [1]: http://andrewgelman.com/2017/09/26/abandon-statistical-significance/ http://andrewgelman.com/2017/09/26/abandon-statistical-signi...
- RA_Fisher 9y agoI really like Gelman's response. We should be using likelihoods, information and and intervals to describe parameters.
- SubiculumCode 9y agoA rebuttal to p< .005 proposal: https://www.researchgate.net/publication/319880949_Justify_your_alpha_a_response_to_Redefine_statistical_significance/figures?lo=1 https://www.researchgate.net/publication/319880949_Justify_y...
- godelski 9y ago538 actually has a good model and discussion about p-hacking. Allows you to play around with values. https://fivethirtyeight.com/features/science-isnt-broken/#part1 https://fivethirtyeight.com/features/science-isnt-broken/#pa...
- bitL 9y agoP-value is such a stupid metrics. Never understood why people still use it and conjure 0.05 magic number as the one that "proves" something. It should be considered a joke, but well, there is economics and a bunch of other "sciences" that are considered "reputable", and people there are daily making living off dubious theories, so nothing will change.
- ggm 9y agoI co-authored a paper into IMC discussing stats and network questions with a statistician. We got bounced for being too polemical and teaching-mode instead of classical science. the whole point of the paper, was to try and argue for a more mature outlook into statistical methods applied to network operations! My motivation to co-write was strongly influenced by being called out for p-jacking. Its remarkably easy to fall from grace. I did. walking back is very hard, when peer review doesn't want to have a polemical discussion.
- cwyers 9y ago> Ideally, Lakens says, the level of statistical significance needed to prove a hypothesis depends on how outlandish the hypothesis is. > Yes, you’d want a very low p-value in a study that claims mental telepathy is possible. But do you need such stringent criteria for a well-worn idea? The high standards could impede young PhDs with low budgets. Watching someone try to be Bayesian without using Bayesian techniques is painful. > There are also new, advanced statistical techniques — like Bayesian analysis — that, in some ways, more directly evaluate a study’s outcome. Almost as painful as watching Bayesian analysis described as "new."
- vinchuco 9y agoNaive question: Is there a good technique for a bot that detects factually wrong statements?
- 09094920394314 9y agoSure, just give me definition of factually wrong and I'build it for you:) Less snarky: not really
- 201709User 9y agoEither general AI or reducing chat into a theorem and then running a prover - but that's for logic not facts. And liberals would have that banned within hours.
- sideshowb 9y agoAhem. Liberal here working on such a system (in concept anyway).
- 201709User 9y agoThen machine learning it is.
- sideshowb 9y agoGive them a break. It's a pop science article. Such articles can describe 1.5 concepts at most, so having used that budget on p-values, it's not available for Bayes, so explaining the concept from a frequentist perspective is reasonable. Also Bayesian analysis (as a mainstream stats technique) is relatively new as it has only become mainstream with the advent of cheap computing power.
- kringldt 9y agoAlong with grit, growth mindset, power posing, priming, and stereotype threat. A lot of non replicating trash in psychology right now.
- Myrmornis 9y agoWhen I looked at the authors list I got the impression that it was mostly sociologists, public health researchers etc, but not very many well known statisticians, or people from the traditional physical and biological sciences. It seems like it would have been better to have more authors from the latter groups to make such a broad suggestion; their absence suggests that the proposal doesn't have much support among those fields.
- teekert 9y agoWhy not just be more clear and instead of writing the "p-value < 0.05", write: The probability that the difference between these two groups is by chance is less than 5%. Then everybody knows that if you do this a bit more than 20 times this results is achieved by chance alone. Then, just show the damn plots and let me see how much distributions overlap. I have seen insanely small P-values with completely overlapping distributions just because of large amounts of data. I find the P-value near useless. Personally I much prefer ROC curves, they are much better at showing the difference between distributions. Still, nothing beats the raw data and healthy skepticism.
- FabHK 9y ago> The probability that the difference between these two groups is by chance is less than 5%. That's a common misunderstanding of the p-value, but that's not what it means. (It means: assuming there is no difference, the probability that we'd nevertheless see a difference as big as we are seeing or even bigger is less than 5%).