11 ms·
Redefine statistical significance
- gattilorenz 9y agoIt can hardly hurt, but it is still a stop gap measure. It won't solve the publication bias people will still change the hypothesis or the test after measurements are done. I think the situation would improve with better teaching of philosophy of science and statistics (this would educate better reviewers too).
- agentofoblivion 9y agoAgree. It doesn't stop p-hacking, it just makes it harder. Definitely treating the symptom instead of the disease. We ultimately need institutional and cultural change, but it's not obvious how to do that in the short term, so making it harder to claim significance might be a step in the right direction. On the other hand, you might expect that new discoveries by nature have less data since data is likely more expensive for brand new research, and by extension a lower likelihood of meeting these sorts of stringent statistical requirements. Decreasing the p-value threshold may be counter-productive if we dismiss legitimate new discoveries due to essentially economic constraints with data gathering, which would have the impact of making it less likely to get funding to pursue the problem in more depth, thereby slowing the advance of discoveries.
- jjoonathan 9y ago> It doesn't stop p-hacking, it just makes it harder. I could see the reverse happening, where higher p-value standards lead to normalization of deviance in the form of worse p-hacking.
- felippee 9y agoI agree with that. In my prediction legitimate, honest research with p~0.05 will become unpublishable, while p-hacked bullshit will prevail in glory.
- PeterisP 9y agoIf you do a honest preliminary study and get a p~0.05 result, then repeating the study with 70% larger sample size should get you p~0.005; but if you've p-hacked, it won't.
- felippee 9y agoBut sometimes getting 70% larger sample will cost you so much money, that you'd need another grant, one you'd get after you'd publish your p=0.05 result, which given the new rules you'd not be able to publish... Statistics is just another method and almost every method can be hacked or abused. Science is not about putting checkmarks in tables but about reading and understanding ideas and reproducing the results. Tweaking some numeric values is not going to help the review process which is fundamentally broken these days.
- zeckalpha 9y agoThis is addressed in the paper. The additional costs of larger sample sizes outweigh the costs of studies based on false positives.
- epistasis 9y agoIt can hurt, in that it can slow the spread of information. If you perform 70% fewer different types of experiments because you have to hit p=0.005 instead of p=0.05, then you explore in fewer directions. This is a classic tradeoff between exploration and exploitation in active learning. If your view of the world is that there are only a very few hypotheses worth exploring, and you have a good lay of the scientific land, then requiring higher bar of proof is probably good. If it's a new field that's extremely complex and where very little is known of the governing principles, then requiring very high stats could severely slow progress and waste lots of research dollars. I completely agree that rather than setting arbitrary barriers for significance, it would seem much better to let people actually understand what was found, at whatever significance it was. Even setting up the null model to get a p-value requires tons and assumptions. The better test is reproducibility and predictive models that can be validated or invalidated. That's where the science is, and not in the p.
- nonbel 9y ago> "It can hurt, in that it can slow the spread of information." I am not at all in favor of this proposal, but one thing it may do is stem the tidal wave of misinformation.
- btilly 9y agoYours is a theoretical concern. The very practical concern is that entire areas of research have been based on studies replicated and backed up entirely through p-hacking and selectively publishing only papers with positive results. This is a proven issue today. See https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis for more. It may be that there is a pendulum that needs to swing a few times to get to a good tradeoff. But it is clear, now, which direction it needs to swing.
- epistasis 9y agoI disagree 100%, having read that Wikipedia page and its sources. It's something that affects a few fields, not all science. And the problem has been completely 100% overblown. If the problem is that things aren't replicating, changing the p-value cutoff for significance isn't going to fix everything. It can just as easily be a bad null model that's the problem,in which case you can't trust any p-value. The MRI scan problem was closely related to that. It's a field-specific and null-model specific thing. Broadly changing the a p-value cutoff for everybody isn't going to fix this issue.
- BeetleB 9y ago>I think the situation would improve with better teaching of philosophy of science and statistics (this would educate better reviewers too). This is necessary, but not sufficient. What's needed is a way to know for sure that the hypothesis was not changed after data collection. I think predeclaring the hypothesis is the way to go.
- gattilorenz 9y agoYeah, but at the end you can still fabricate the data, remove "outliers",... Plus it's almost impossible to imagine a world where, before any experiment in any field, you predeclare it. Not that education can fix all these (you can't prevent evil), but if reviewers and journals and conferences started to accept more the negative results, the incentive in lying would quickly decrease. And people would probably start to "disprove" interesting theories, instead of trying to "prove" niche results...
- BeetleB 9y ago>Yeah, but at the end you can still fabricate the data, remove "outliers",... Fabricating data is essentially fraud. And while it does happen, most of the problems with reproducibility are not problems of fraud. It won't protect against outliers, but removing outliers will not solve most problems. It'll happen, but again, I don't think the majority of irreproducible studies are due to misuse of outliers. >Plus it's almost impossible to imagine a world where, before any experiment in any field, you predeclare it. Not at all. I'm not saying you predeclare every experiment - just every experiment you try to publish. The way it works is: 1. You make observations (i.e. collect data - no predeclaring anything). If you see interesting patterns, you'll form a hypothesis. 2. This is the stage where you predeclare your hypothesis, and the criterion of falsification. 3. You now collect new data and test it against your hypothesis. The hard part is ensuring people won't use some of the old data and claim they collected after their declaration. It's a hard problem, but not an impossible one. People are in the habit these days of collecting a lot of data, seeing patterns, and publishing them. That's really not how a lot of early science was done. Once you see the patterns, you need to conduct more experiments to falsify them.
- 9y ago
- gboudrias 9y ago> It won't solve the publication bias people will still change the hypothesis or the test after measurements are done. As a Psychology student, this is a well-known initiative: https://cos.io/prereg/ https://cos.io/prereg/ (Though I can't confirm or deny its widespread usage.) The publication bias is harder, and pre-registration won't solve this. But I think this is a separate issue, and it's important to address each issue in its own right. I've seen the proposal from TFA before and with my very limited knowledge, I'm still fairly certain it will never come to pass in Psychology, as nearly half of all modern studies have reproducibility issues (!). It would be beneficial to our field, in the way that a band-aid is beneficial to a gaping wound, but it would require a lot more rigor than has been evidently been displayed so far (and more rigor is more work, and time is limited). So... Don't hold your breath. (Sorry if my comment sounds pessimistic, I don't know much and I'm open to being corrected. I still have enough critical thought to be skeptical of some researchers' dedication to intellectual rigor.)
- leemailll 9y agoChange p value from 0.05 to 0.005 won't stop p-hacking. And this might also lead to more grunted graduate students as they then will have to increase sample numbers to satisfy new test, which inevitably increase the already painful long time span for projects to get published
- taeric 9y agoTo be fair, this is a pragmatic, not a technical, solution. Similarly, we limit the speeds we allow in residential areas not because it prevents wreckless driving, but because it decreases the actual risk of it. Similarly, the technical solution involves technology that does not require drivers and has no risk of human error anymore. The pragmatic solution is to just limit the acceptable speeds.
- nkrisc 9y agoA bit of humorous pedantry: We seek to prevent reckless driving. "Wreckless" driving is what we're trying to promote.
- taeric 9y agoThanks for the pedantry. It is amusing to me, because the "w" looks way more correct to me. Can't say why, though.
- noxToken 9y agoThough reck is a word, it isn't commonly used, whereas wreck is a pretty elementary word. Reck and wreck are homophones. That's probably your answer.
- taeric 9y agoRight. But to me the word "wreckless" even looks like what I want in this sentence. I think I take the meaning to be "as if wrecks weren't a thing." As opposed to how the pedantry points out, that if "wreckless" were a word, it would actually be more of the opposite of "reckless."
- aheilbut 9y agoNo one in biology would be able to publish anything.
- pfortuny 9y agoImagine psychology... The end of a science.
- kingkawn 9y agoOr it never was one
- aimager 9y agoit's too far away
- Strilanc 9y agoFunny, I was thinking the opposite. Imagine psychology... done properly. The beginning of a science. (I realize that "beginning" is too harsh, but psychology does have very serious problems with replicability. At the moment, it deserves its tarnished reputation.)
- nonbel 9y agoThere are so many problems with this: 1) The p-value filter leads to publication bias. -You should publish your results anyway, or the study wasn't designed/performed correctly. The raw data and description of methods should be valuable. 2) The null hypothesis is (almost) always false anyway. -Everything in bio/psych/etc has a real (not spurious) non-zero correlation with everything else, so the significance level just determines how much data needs to be collected to reject it. 3) Rejection or not of the null hypothesis does not indicate whether the theory/explanation of interest is correct, so is inappropriate for deciding whether a result is interesting to begin with. -Usually the null hypothesis is very precise and the "alternative statistical hypothesis" that maps to the research hypothesis is very vague, so many alternative research hypotheses may explain the results.
- marcosdumay 9y ago
- imh 9y ago>For a wide range of common statistical tests, transitioning from a P value threshold of α = 0.05 to α = 0.005 while maintaining 80% power would require an increase in sample sizes of about 70%. This proposal is a great pragmatic step forward. Like they say in the paper, it doesn't solve all problems, but it would be an improvement with reasonable cost and tremendous benefits. >Such an increase means that fewer studies can be conducted using current experimental designs and budgets. But Fig. 2 shows the benefit: false positive rates would typically fall by factors greater than two. Hence, considerable resources would be saved by not performing future studies based on false premises.
- stdbrouw 9y agoIn some fields like psychology, power is more likely to already be 10% or 20% for the majority of studies, and in fact P-hacking and low standards for evidence would be far less harmful if power were higher, because low power leads to inflated effect size estimates. Additionally, power calculations are always just a guess and easy to fudge, so it's pretty much a given that current statistical power would not be maintained with more stringent critical values. See http://andrewgelman.com/2014/11/17/power-06-looks-like-get-used/ http://andrewgelman.com/2014/11/17/power-06-looks-like-get-u... So this proposal is really the opposite of pragmatic. Pragmatic would be requiring effect size estimates and confidence intervals in all published papers. It is surprising how many papers will talk about highly significant effects without actually discussing how large the estimated effect is thought to be, which gives authors a lot of leeway when exaggerating the importance of their findings.
- reilly3000 9y agoThe cost of increasing sample size is significant; this is a trade-off that allows smaller projects to still conduct valuable research.
- Klockan 9y agoStudies with low significance are no better than anecdotes.
- tw1010 9y agoThis still doesn't feel satisfying. Part of me is still not really happy with the philosophical foundations of statistics. Does anyone know of any legitimately competing theory to statistics? Maybe something that doesn't rest on the same types of mathematics that Fisher and crew relied on when all this started? Pure mathematics has come a long way in the last fifty years but little has seeped into the applied world.
- 168 9y agoHow & why exactly are you unhappy with statistics?
- ArchReaper 9y agolol what the hell are you on about
- deleted 9y ago[deleted]
- scryder 9y agoStatistics arises from a set of axioms, assumed truths, which can be used to prove all other things in the field. You can take a look at the three axioms people use to justify statistics. If you are willing to accept them, all else that relies on them (without using new axioms) must be true: https://en.wikipedia.org/wiki/Probability_axioms https://en.wikipedia.org/wiki/Probability_axioms This same logic is used to justify development in pure mathematics: choose a set of axioms which you accept as ground truths, and prove things using them. As long as you are unable to prove your axioms are contradictory, and the axiom choice seems acceptable, then the work that you've done (with respect to them) is philosophically justified.
- iovrthoughtthis 9y agoWhen can we have scientific papers formatted for the web? Reading pdf's with many tiny columns spread across each page puts me off reading so much.
- mjpuser 9y agoThis is an interesting point considering the World Wide Web was born from the need to share scientific info.
- folli 9y agoMost journals have a HTML and a PDF version (as does Nature): https://www.nature.com/articles/s41562-017-0189-z https://www.nature.com/articles/s41562-017-0189-z I prefer the PDF version for print outs.
- iovrthoughtthis 9y agoThanks you for this. I had no idea. Now I can actually read the paper!
- s17n 9y agoIt used to be possible to have have a successful academic career without publishing much - for example, one of my philosophy profs in college (at a top 10 school) had never published anything after his dissertation (he got his phd in the early 60s). Of course, this system only worked because academia was a bastion of the male WASP elites that didn't have much pretense of serving the broader public. But at least you didn't have the torrent of mediocre papers that you see today.
- rebuilder 9y agoSo... what the academics did wasnt very helpful, but at least they didn't do much of it?
- marcosdumay 9y agoThose ones teached. Wether is was helpful or not depend on how good was the teaching and how useful is the knowledge. Not everybody in a teaching institution should be required to push humanity's knowledge forward. But then you get the problem of selecting those people without easily measured objective indicators. That's why it worked reasonably well when those were slightly low paying jobs restricted to a caste.
- adekok 9y ago> academia was a bastion of the male WASP elites that didn't have much pretense of serving the broader public Have things really changed? I suspect there are fewer males, but any job that demands 20 years of full-time concerted effort is likely to be dominated by men. Similarly, the western world is overwhelmingly caucasian, so again... the best predictor (now as then) is that white male professors will be represented disproportionately. > at least you didn't have the torrent of mediocre papers that you see today. That certainly is true. Stats for the humanities and social sciences are that 80% of the papers have zero citations. i.e. they have no contribution to the greater body of human work. In Physics (my background), most papers have 2-3 citations, and only a small percentage have 1 or fewer. I would say that if a discipline is dominated by uncited papers, then that discipline is probably a waste of time. And the professors who work in it are a net drain on society.
- jimmar 9y agoNot a huge fan of this idea. For example, people who analyze twitter data can get very small p-values because they analyze millions of tweets even though the effects they find are very small. See https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1336700 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1336700
- amelius 9y agoI don't see the problem as long as you clearly separate significance and effect-size.
- seanwilson 9y agoThis bothers me a lot of media reporting. The headline will be something like "X is good/bad for you" with a tiny effect size but the way it's reported makes you assume it's a large effect size. Usually the effect size won't be discussed in any depth or at all, they just want to sum it up in black and white. If the effect size is tiny it's probably just noise that would disappear in a higher quality future study.
- moultano 9y agoI'd rather hear about small things that are true than large things that are false.
- cropsieboss 9y agoThe things is that these small things might just be noise from some confounding factors. http://jaoa.org/article.aspx?articleid=2517494 http://jaoa.org/article.aspx?articleid=2517494 For example, here sample size is huge, USA population gets significant increased risk, while EU population does not. Mixing the two together would result in a smaller but still significant increased risk. Given the size, it's quite clear that USA population has many other confounding factors that cannot be eliminated by mathematics alone (there is no control).
- nonbel 9y agoStatistical significance doesn't mean an effect is "true", or "real". So getting rid if it shouldn't cause you any issues.
- SubiculumCode 9y agoThis is fine, but without other simultaneous changes, will do harm to young scientists. We need credit for publishing null results, or stop judgment on the basis of publication number. Would lead to larger, more well powered studies (good), but this tends to lead to acquiring multiple measures which can be inappropriately data-mined, and leads to large grants to established investigators, but fewer grants to new investigators.
- jerrytsai 9y agoDefinitely. The main problem is that in the current system no one is being rewarded for good science, but for showing something interesting, bolstered by a declaration of (statistical) significance. The incentives are not aligned with societal objectives. Good science requires a tension between hypothesis generation and skepticism. Perhaps if we rewarded the _debunking_ of findings as much as we do the discovery of findings, things would change.
- adrianratnapala 9y agoWhy doesn't this happen already. The funding bodies etc, who want "quantitive" measures of research look at publications. Why would we expect debunking papers be published if they are debunking something interesting?
- adrianN 9y agoOnce you have debunked something interesting I suppose you don't have trouble getting in published. But you have a hard time writing a grant application that reads something like "I want to replicate study X, no new results are expected."
- adrianratnapala 9y agoI think this hits the nail on the head. That said, it also seems like low hanging fruit. At least in some fields, replication should be a lot cheaper than doing things in the first place -- because a lot of the cost in pissing about trying to find something that even seems to work. For funding bodies to explicitly support replication studies, even if each gets only 20% of the amount the original studies get should be a winner at reasonably low cost.
- rgejman 9y agoAnimal experiments will get A LOT more expensive. Will there be a concomitant increase in agency funding to offset the increased costs?
- siginfo 9y agoThey do briefly mention "the relative cost of type I versus type II errors". Both errors (Type I - false positive, Type II - false negative) have some cost associated. Money saved by using a small sample size is wasted trying to replicate a false positive result and by groups around the world that rely on that false result. Requiring larger sample sizes would mean fewer experiments are carried out but we will have more confidence in the positive results produced. The outcome is fewer experiments wasted on following up on false positives. None of this requires a change in funding.
- rgejman 9y agoI really don't think the proposal to do "fewer, but better" experiments work with animal studies. They are so expensive and so complicated and so much work and only answer singular, small questions that you almost always need a ton of further follow up work. For instance, in the field I work in you have to spend days to months waiting for tumors to grow and then go and treat the animals every day for a couple of weeks with an IV drug (weekends too!). That is a a lot of work and at the end only tells you one piece of information about the drug: does it slow tumor growth in this one experimental model. It may in fact do that -- and you may get a really great p-value if you increase the number of mice -- but you still need to study the drug's pharmacokinetics, tissue distribution, in vivo mechanism of action (assuming you already know the in vitro mechanism of action). These are not just optional experiments that we require today to publish: this kind of work is essential to presenting a story about a new drug. It's not just about what it does, but how it works and universalizable it is.
- JepZ 9y ago> The choice of any particular threshold is arbitrary [...] Sounds scientific, doesn't it? > [...] we judge to be reasonable. And tomorrow someone else judges it differently? Maybe they should not try to redefine significance but simply introduce something called 'well-reproducible' or so.
- logicallee 9y agoCan someone explain why this three-page article has 72 "authors"? That works out to about as much writing per author as this comment.
- eelkefolmer 9y agoIts time to ditch significance levels altogether and use Bayesian inference or analysis.
- analog31 9y agoI'm concerned that prior-hacking will become the new p-hacking.
- Houshalter 9y agoSo you require people report Bayes factors, not posteriors/priors. Those are invariant to the prior.
- stewbrew 9y agoI'd rather say it's time to recall what significance levels actually meant to be and in which context they are useful and to ditch the contemporary aberration thereof. Any game can be gambled. Bayesian statistics just isn't there yet.
- cameronraysmith 9y agoI've had some luck showing John Kruschke's Bayesian estimation supersedes the t-test (BEST) and this simple demonstration http://www.sumsar.net/best_online/ http://www.sumsar.net/best_online/ to people.
- md224 9y agoJust curious: would this have an effect on testing the efficacy of new drugs? I'd hate to see a false negative result for a drug that could actually help people...
- Fomite 9y agoStop worshipping p-values set to an arbitrary threshold, whether it be 0.05 or 0.005, and start actually critically engaging with the statistics and results themselves.
- Houshalter 9y agoThe concept of statistical significance is nonsense. In Bayesian statistics there is only evidence. A p value of 0.05 is roughly equivalent to a factor 20 of evidence. That means you multiply the odds you believe in a hypothesis by 20 (or add 13 decibels.) Similarly a p value of 0.005 is roughly equivalent to 200 units of evidence (23 decibels.) But whether some amount of evidence is "significant" or not is entirely dependent on your prior. If you believe something has about a 50:50 chance of being true to start with, then a factor 20 of evidence is quite enough. Now you believe it 20:1 likely to be true. But for something like xkcd's "green jelly beans cause cancer", your prior should be something like 1 to 100,000 or even smaller. After all, there are a lot of possible foods and a lot of possible diseases. Unless you believe a significant number of them are dangerous, your prior for any specific food causing any specific disease must be pretty low. And then even a factor 200 of evidence is nowhere near enough to convince me that green jelly beans cause cancer.
- maxerickson 9y agoIf it is nonsense you shouldn't be able to translate it coherently into various levels of evidence.
- Houshalter 9y agoP values aren't nonsense and correlate with bayesian evidence. I think that interpreting levels of evidence as "significant" or not is nonsense.
- mnarayan01 9y agoCurious if anyone's done any work to determine if changing the P-value threshold for e.g. Psychology studies (as they call out Psychology in particular) measurably affected replicability with p > 0.005?
- zeckalpha 9y agoThere's overlap in authorship with this paper: http://www.sciencemag.org/content/349/6251/aac4716 http://www.sciencemag.org/content/349/6251/aac4716
- wavegeek 9y ago> For a wide range of common statistical tests, transitioning from a P value threshold of α = 0.05 to α = 0.005 while maintaining 80% power would require an increase in sample sizes of about 70%. This seems unintuitive and the claim is unreferenced. Can anyone explain why this is the case (if true)?