10 ms·

Psychology Journal Bans Significance Testing

152 points by tokenadult 12y ago

phren0logy 12y ago
There was a question yesterday about Evidence Based Medicine vs Science Based Medicine. The SBM criticism of EBM is the over-reliance on Randomized Controlled Trials that meet p=0.05, without looking at the prior probability that a treatment would help. For example, EMB would say that if you have an RCT that shows that a lucky rabbit's foot works, then you have reasonable evidence to put that into practice. The issue is that, as this article points out, even with a statistically significant result for such research, there's no plausible mechanism by which a rabbit's foot makes you lucky. Therefore the RCT is just one part of the whole picture, and subject to a specific type of manipulation.
- srunni 12y ago
  Something related to keep in mind when it comes to p-values is the false discovery rate: > If you run lots of phase 2 trials with different drug candidates where only a minority (lets say 10%) actually work, then with standard trial statistics (80% power and 5% false positive rate) you will get 4.5% false positive and 8% true positives – so less than 2 out of every 3 positive trial results were real. A much lower success rate than the 5% error rate commonly assumed. http://www.forbes.com/sites/davidgrainger/2015/01/29/why-too-many-clinical-trials-fail-and-a-simple-solution-that-could-increase-returns-on-pharma-rd/ http://www.forbes.com/sites/davidgrainger/2015/01/29/why-too... This recent (open access) review in R. Soc. Open Sci. the Forbes article was largely inspired by is worth a read too: http://rsos.royalsocietypublishing.org/content/1/3/140216 http://rsos.royalsocietypublishing.org/content/1/3/140216
- Alex3917 12y ago
  > The SBM criticism of EBM is the over-reliance on Randomized Controlled Trials that meet p=0.05, without looking at the prior probability that a treatment would help. The strength of significance testing is that it purposely doesn't try to tell you how likely something is to be true, only how likely the data you got was the result of chance assuming the treatment is no better than placebo. You're still taking the prior probability into account when trying to figure out the truth, you're just not putting a number on it. My concern with bayesian approaches is that, like with frequentist approaches, the truth is still fundamentally unknowable, only now you're encouraged to put a number on that and pretend that it's science. While bayesian approaches totally make sense in trying to determine a patient's likelihood of having some disease when there is already data available for the prevalence in a population and the sensitivity and specificity of the tests, using bayesian logic to weight clinical trials strikes me as being highly dubious. It would be one thing if SBM actually developed a framework to give a weight to each methodological feature of a trial, but so far I haven't seen much work to build a functioning system. Though if you're really honest about all the ways that you can have positive results without something actually being true, it seems like almost no amount of research will ever have a significant effect on the prior.
  - mdbco 12y ago
    You're absolutely correct that Bayesian approaches are not magical and do not suddenly supply you with vastly more information than frequentist approaches (particularly when you have a really poor prior, in which case the Bayesian approach will be similarly poor). Bayesian statistics is certainly very popular right now, but it should not be looked at as some sort of panacea for all statistical problems. However, I would say that Bayesian approaches do have a big advantage in terms of helping with the interpretation problems that plague frequentist significance testing. Namely, as the OP article points out, Bayesian approaches reformulate the testing question in a way that is more intuitive, i.e. "what is the probability of the hypothesis given both the prior probability and the new data?". So yes, Bayesian methods surely do not fix everything, but since interpretation of statistics is such a major concern, they can be quite beneficial.
  - yummyfajitas 12y ago
    You're still taking the prior probability into account when trying to figure out the truth, you're just not putting a number on it. This is a danger sign - you are doing the same things the Bayesians do, just informally, less explicitly, and probably incorrectly. The fact is that to make a good decision, eventually you need to compute a single number. This is an elementary fact of topology: https://www.chrisstucchio.com/blog/2014/topology_of_decisionmaking.html https://www.chrisstucchio.com/blog/2014/topology_of_decision... That number will be based on some unproveable assumptions. That's a fact of Godel's incompleteness theorem, if nothing else. So given this, why is it "dubious" to make those assumptions explicit and obvious?
    - algorias 12y ago
      > The fact is that to make a good decision, eventually you need to compute a single number. Your linked blog post states that if you make a good decision, then there is a process computing a single number which is equivalent to your process. This is not equivalent to what you claim. As a matter of fact, it's the same kind of confusion that exists around the p-value. It's not the case that a process explicitly computing such a number automatically makes good decisions, which is what you seem to claim implicitly. Also, Gödel has nothing whatsoever to to with this.
- WhitneyLand 12y ago
  But how could a randomized clinical trial make this mistake? What experimental design could make the rabbit's foot look good? "EMB would say that if you have an RCT that shows that a lucky rabbit's foot works, then you have reasonable evidence to put that into practice."
  - sunir 12y ago
    You have heard "19 times out of 20" described in the news? That is the 0.05 restated for laypeople. 1 time out of 20 you will get a false positive, in this case that the rabbit's foot worked.
    - WhitneyLand 12y ago
      Sorry maybe I'm being dense, but who would take 1 out of 20 success to mean they should start buying rabbit feet?
      - Mvandenbergh 12y ago
        Nobody. The problem is that if you conduct 20 trials of the efficacy of rabbit feet, you'd expect 1 trial to show a significant effect. (if you're interpreting P-values the way many people do, incorrectly.)
      - DanBC 12y ago
        When you do a single trial you might show that rabbits feet are effective. You don't yet know if this is a 1 in 20 result or a 19 in 20 result. That's why you replicate.
      - ejain 12y ago
        You'd start buying rabbit feet because only the study that was "successful" is published.
analog31 12y ago
I wonder if it just boils down to this: Exclusive reliance on any single tool by an entire field for a long enough time period will eventually lead to a proliferation of bad results.
- rcthompson 12y ago
  More like exclusive reliance on a single tool will lead to a requirement for everybody to use it, even if they don't understand how to use it properly, which leads to both unintentional misuse (by experimenters who don't understand it) and the inability to catch intentional misuse (because adjudicators don't understand it either).
po 12y ago
The audio is a bit quiet but the 'dance of the p values' video [1] linked to by the article is absolutely fantastic and well worth watching. [1] https://www.youtube.com/watch?v=ez4DgdurRPg https://www.youtube.com/watch?v=ez4DgdurRPg
MengerSponge 12y ago
Good! This is a big step in improving everybody's work. If you aren't up on this research, here are the two articles that you need to read. Why Most Published Research Findings are False: http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 http://journals.plos.org/plosmedicine/article?id=10.1371/jou... Revised Standards for Statistical Evidence: http://www.pnas.org/content/110/48/19313.short http://www.pnas.org/content/110/48/19313.short
mdbco 12y ago
The article is certainly correct that p-values and confidence intervals (or confidence sets, in multi-dimensional contexts) are widely misunderstood, not just in psychology or other social sciences, but in the hard sciences as well. The problem is even worse when you look outside of academia at common practices in more applied settings. As suggested, a good approach is to take p-values not as conclusive or decisive, but rather as a tool that must be supplemented by other statistics. In particular, the article emphasizes Bayesian methods, which can certainly provide additional information, but this approach can also be rather limited when priors are not well-defined or are entirely unknown, which is unfortunately often the case in many problem domains. One potential question is how to determine the nature of the distinction mentioned in the conclusion between "preliminary research" and "confirmatory research", particularly in cases where statistics provide the primary evidence, as in, e.g. psychology. Further studies in the same vein as the preliminary research can certainly provide additional supporting statistical evidence, but this doesn't escape the problem that all of the evidence is probabilistic in nature. The key issue here is that since statistical approaches can only give probabilistic evidence that a hypothesis is correct, then they strictly cannot tell you what is certainly true, so even confirmatory research is quite open to falsification. So we wouldn't want the label of "confirmatory research" to somehow suggest to the public the idea that it is certainly correct.
- deleted 12y ago
  [deleted]
- esfandia 12y ago
  Is the cautious approach then to treat a p-value in the absence of priors on the same level as a p-value in presence of unfavorable priors? When someone tests positive for a cancer test, the priors are known (probability of cancer in the general population is usually very low, and the false positive rate of the test may be relatively high), and so usually that first test is merely indication that further tests are needed. So when you don't know the prior and you observe a low p-value on something, isn't that just "preliminary research" that needs to be further confirmed with other methods or at least the same test but using other data?
  - 12y ago
Kenji 12y ago
The problem is this: You need a solid background in statistics and some mathematics to perform these tests properly. Most apparently, people from that field lack these skills. If someone can't hit a nail on the head with a hammer, blaming and banning the use of hammers won't solve a thing.
- manicdee 12y ago
  Taking hammers away will solve the problem of holes in things which aren't supposed to have them, including thumbs, walls, people's heads, and candy jars.
  - zmjones 12y ago
    there are just other tools that will get abused. bayes factors for example
- viraptor 12y ago
  Depends what are you trying to solve - people's understanding of p-values, or many false results? If you care about getting true results and there are easier ways to get there than p-values, why would it be a bad solution? Similar thing to what happens in software security. Sure, you could tech people how to use memory management properly - and then watch them fail again and again. Or you could just provide an alternative solution like rust which changes the issue completely.
VieElm 12y ago
I kind of which that the widely read publications, like newspapers, that have turned out science articles based on all these now pretty much discredited research articles would issue retractions or at least clarification for each of the articles that has been challenged in this regard.
CountBayesie 12y ago
>The type of analysis being banned is often called a frequentist analysis I find that there is a trend of associating "bad statistics" with "Frequentists Statistics" which isn't really fair. If you found a statistician trained only in Frequentist methods and asked their opinion on experiment design in psychological research they would likely be just as appalled as any Bayesian. I'm a big fan of Bayesian methods, but the solutions of "we'll solve the problem of misunderstanding p-values by removing them!" is still a problem of misunderstanding p-values! The misunderstanding is the issue, not the p-value.
- deleted 12y ago
  [deleted]
- colechristensen 12y ago
  I think what this journal doing is probably a good thing, but only as the lesser of several evils. The truth is something more like... it is easier for soft sciences to abuse frequentist statistics than bayesian. Both have merit, it cannot be argued, but it is simply easier to produce meaningless conclusions with frequentist statistics done wrong. This situation is so bad that it merits banning frequentist for this journal and I think that's reasonable. This doesn't mean that every journal in every field should, but perhaps it will be a useful temporary measure to improve quality.
- stdbrouw 12y ago
  The problem is that p-values are begging to be misunderstood, and in fact you cannot use them as a decisionmaking procedure without "misinterpreting" them – after all, you're deciding whether to accept the hypothesis P(HA|D) based on 1-P(D|H0) on the grounds that, while they're not the same, they're proportional. (In that sense the p-value is like the poor man's likelihood ratio.) There's nothing wrong with p-values as a concept, but there's everything wrong with p-values in hypothesis testing. The misunderstanding is baked in.
  - im3w1l 12y ago
    You can update your posterior based on the p-values yourself though. "Well those eggheads may have disproved X, but X is just common sense, so I'm gonna keep believing it anyway. U-until I see more studies confirming the finding I mean."
rcthompson 12y ago
I spent about a hour explaining p-values to a fellow graduate-level researcher a few weeks ago. I pretty much just kept rephrasing the definition in slightly different ways until the person finally got it. In undergrad, hypothesis testing was more or less taught as "do this inscrutable calculation and if the result is 0.05 or less, you win". The point is, in my experience, a lot of people really don't get p-values, even graduate-level and post-graduate scientists.
- viraptor 12y ago
  I agree. I'm in the pool of people who don't really understand p-values even after taking 2 statistics courses. The main issue I've noticed is that compared to many other calculations, you just have no idea if your result is correct. You can make wrong assumptions, wrong calculation, apply wrong methods, but in the end you get a number... and that's it. It may be a wrong number and you'll never know. This is completely different to many other practical applications of math where you can verify your result in various ways, or validate your answer against the initial assumptions, or test your program against lots of inputs.
  - mdbco 12y ago
    It's actually entirely possible to do some "checking of your answer" for p-values, as well. As you mentioned, for practical math, you can often validate your answer against the initial assumptions. This is true for statistical testing too, as it typically relies on many theoretical assumptions. So what you can do in practice is propose a different set of assumptions, perform the hypothesis test in a manner that follows those new assumptions, and see if you obtain a similar result. Typically, for any given hypothesis that you want to test, there are several possible methods for performing that test, so you can redo your test many times. This is one type of robustness checking, which includes many other things as well (e.g. running your test over subsamples or resamples of the data, checking for sensitivity to outliers, etc). Good statisticians generally like to do lots and lots of robustness checking.
  - yummyfajitas 12y ago
    The main issue I've noticed is that compared to many other calculations, you just have no idea if your result is correct. You can make wrong assumptions, wrong calculation, apply wrong methods, but in the end you get a number... and that's it. The same is true in Bayesian statistics, and even simple formal reasoning with no statistics in sight. If you make wrong assumptions, you'll get the wrong result. The only thing you can expect statistics to do is help you change your opinion about the relative merits of opposing theories. If both your opposing theories are wrong, you will still be equally wrong. The true flaw with frequentist statistics is that it goes out of it's way to hide this fact from you. In contrast, Bayesian stats forces you to explicitly choose a prior, enumerate your assumptions, and accept that your conclusion is based on these things.
cafebeen 12y ago
Here's the key passage: <quote> BASP will require strong descriptive statistics, including effect sizes. We also encourage the presentation of frequency or distributional data when this is feasible. Finally, we encourage the use of larger sample sizes than is typical in much psychology research, because as the sample size increases, descriptive statistics become increasingly stable and sampling error is less of a problem. </quote> In other words: report the effect size, plot the data, and increase the N.
deleted 12y ago
[deleted]
torgoguys 12y ago
Obligatory xkcd: http://xkcd.com/882/ http://xkcd.com/882/
jwilliams 12y ago
I think this Wikipedia section summed it up really well for me: http://en.wikipedia.org/wiki/P-value#Alternating_coin_flips http://en.wikipedia.org/wiki/P-value#Alternating_coin_flips Extreme example, but explains how the context can shift the findings significantly.
kriro 12y ago
This is very interesting for me. I recently switched from AI to Human Computer Interaction which is more cross discipline, specifically with a strong influence from psychology. I'll gladly admit that I don't have the best understanding of NHST but I think I understand it well enough. Interestingly enough there was a post on HN somewhat recently about Bayesian alternatives (BEST). The paper that was linked was: ftp://ftp.sunet.se/pub/lang/CRAN/web/packages/BEST/vignettes/BEST.pdf And the recommended book I settled on was by the author of that paper (Doing Bayesian Data Analysis) I feel like I'm "ahead of the curve" thanks to HN :D
deleted 12y ago
[deleted]
stdbrouw 12y ago
Banning p-values makes sense to me, as they force you to declare an effect as either significant or not significant, rather than looking at the preponderance of the evidence and building knowledge over multiple experiments. It also leads us to focus on statistical uncertainty at the expense of all of the other kinds of uncertainty researchers are faced with: do I have the theory to back this up, am I actually measuring what I am trying to measure, is this coherent with other findings in the field? I do think the editors might be right when they say banning p-values will make the quality of the research go up, not down. But if you read the original editorial, up at http://www.tandfonline.com/doi/pdf/10.1080/01973533.2015.1012991 http://www.tandfonline.com/doi/pdf/10.1080/01973533.2015.101..., you can see that they also reject confidence intervals and Bayesian reasoning with uniform priors (which is really the same thing) without providing any guidance at all on better procedures. I fear that will just lead readers to try and guess the reliability of the data themselves, or worse, interpret the sample statistics as numbers without any associated uncertainty. So they're doing away with poor statistical procedures, but at what cost? It's like that old joke: we've found a 100% reliable cure for cancer – bombing the planet until everybody's dead.
- nabla9 12y ago
  Methodological flexibility in statistical modeling has gone trough the roof. New tools and estimation methods make Bayesian methods easy to use: Jags and Stan, Hamiltonian Monte Carlo, the variational Bayesian approximations, expectation propagation and even probabilistic programming (note: Bayesian in this context does not mean subjectivity. It's the ability to quantify uncertainty in the results and increased flexibility in the modeling). New methods and tools are faster to use and give better results, but they require more statistical knowledge. If the scientist applying these methods or peer review can't understand the advances, it's all for nothing. Many sciences are methodologically very conservative to the extent that it holds the science back. How do you increase the statistical knowledge of the field so much that peer review and researchers can be expected to understand and use the new methods if they can't be trusted with p-value?
Booktrope 12y ago
If the stats aren't based on samples of real-world data, rather than very rigorously designed experimental comparisons, I think there's another reason to be wary of p-values. [Note, in what follows I probably use some terminology wrong, because I'm not a statistician, but I do think the point is important and I don't see much written about it.] In the real world data is not a bunch of independent events, but events (or data points) that are interconnected in ways that are difficult to quantify. I once heard a presentation by an expert in AB testing at a leading tech company who became wary of his results and brought his concerns about non-independence of events that were being tested to corporate statisticians. (He was concerned that, even though the AB testing procedure supposedly randomized the test, interconnectedness within website traffic was not being accounted for.) By his account, the statisticians agreed it was a problem but recommended that he assume that variations due to non-independence would more or less balance each other out. He wasn't satisfied with this and said that he went ahead and did some more measurements, then ran out all the binomial expansions rather than relying on approximations. When he did this more detailed work, he found out that with at least some web-based AB tests, where conventional statistical formulas showed a p-value of .05, he was getting a confidence level of more like 30%. (I don't think you could call his measurement a p-value because he wasn't using the formulas normally used to compute p-values, but I think what he was saying was more or less that the p-value from the forumalae was .05 when it should have been .30 from a more rigorous look) As I'm not a statistician by trade I don't keep up on the literature very well, but interconnectedness of data does seem to me to be a very important issue. I'm wondering if anyone can point me to some helpful reading to understand this side of the issue better. In particular, is there any approach to AB testing that can reliably address the issue of data interconnectedness in the kind of situation described above?
- stdbrouw 12y ago
  Could you expand on what you mean by "interconnectedness within website traffic"? If one person visiting the site has no influence on other people visiting the site, then measurements of their behavior will be independent. If Facebook tests a different interface on half of their users and the changed behavior of those users indirectly has some impact on the behavior of the control group, then your measurements would have some level of dependence – I can imagine that this could happen but it's not clear to me that this would be a common scenario. The same would happen if you measure the behavior of the same person more than once – but in this case there's many procedures for working with paired or autocorrelated data.
chrisseaton 12y ago
I never understood where 0.05 came from. It seems like an arbitrary, magic number. Aren't those bad in science? Shouldn't we have some reason for every number we use? Why 0.05 instead of 0.049?
- nzp 12y ago
  It is somewhat arbitrary. 0.05 corresponds to roughly 2, and 0.01 to roughly 3 standard deviations under the normal distribution. In high energy physics, for example, in the past a result with 3 sigma significance was considered “a discovery”, but it turned out there were a lot of false signals, so it was upped to 5 sigma (~ 3 x 10^{-7} percent chance of getting the effect assuming there is no effect in reality), while 3 sigma (or 4) result is considered “evidence of effect“. Unfortunately in softer sciences, considering their relatively generally poor state of research standards, we have no idea if 0.05 or 0.01 levels really are good standards of “discovery”.
return0 12y ago
i think this is also important on a conceptual level. There's just too much literature in life sciences nowadays which, instead of formulating bold hypotheses with clearly distinct effects, is happy to report miniscule effects that just crossed the arbitrary significance barrier.
MollyR 12y ago
I might be missing the forest from the tree's. The paper talks about how it seems researches are "hacking" (their word) p-values.If the researchers lack the ethics using one form of statistics, what is really stopping them from misusing Bayesian Analysis? Sidenote: While we are talking about bayesian stuff. I recently ran into the sleeping beauty problem ( http://en.wikipedia.org/wiki/Sleeping_Beauty_problem http://en.wikipedia.org/wiki/Sleeping_Beauty_problem ) and it showed how different interpretations of an experiment can lead to people believing in two very different answers in an almost religious way. Some people argued this thought experiment shows a clear flaw in bayesian thinking. I'm not sure either way, I'm still thinking about the puzzle.
- Gravityloss 12y ago
  That's a cool problem. It took a while to understand the angle. I'd say I'm a halfer. I think the extreme sleeping beauty problem highlights how if you're woken up, it's not any more probable it's a head or a tails awakening, since, for a reason I can't explain, the million tails awakenings "don't accumulate". In a sense, it's like the opposite of the Monty Hall problem, since here the sleeping beauty receives no information whatsoever during the experiment.
  - MollyR 12y ago
    I'd say I'm a halfer too, but a lot of physicists/philosophers say 1/3, and it really bothers me that I can't seem to sync up with world leaders. source : http://rfcwalters.blogspot.com/2014/08/the-sleeping-beauty-problem-how.html http://rfcwalters.blogspot.com/2014/08/the-sleeping-beauty-p...
logicallee 12y ago
the main reason to ban significance testing in all fields is because of logicallee's getcher scientific results agency. Our prices are: $10,000 random study, no results guaranteed. Not recommended! Highly likely to be damaging. $20,000 basic study, p=0.5 - study inconclusive but implies it's at least not "more likely" that the damaging/negative result is correct. (Assuming uniform priors.) No scientific value. $100,000 weak FUD. Suggests that the damaging/negative results (assuming bayesian reasoning or uniform priors) may be incorrect, at a suggestive p<0.10 level. Not conclusive and invites further studies which can strengthen damaging/negative result! Not recommended unless further studies are unlikely. Consists of ten studies, one of which is published with remaining buried. Unscientific. $200,000 basic refutation. Refutes the damaging/negative result at a statistically significant p<0.05. Likely to be referenced and accepted. However, due to significance level, links to logicallee's getcher scientific results agency should still be avoided! (Invites skepticism.) Scientific. May take up to six months. $500,000 Silver refutation. Highly significant refutation at p<0.02. Study can be extremely rigorous and sponsorship can be public. The results should be referred to and referenced as widely as possible. Unpublished results (the other 49 studies in the series) to remain unpublished and unreferenced. Due to the number of parallel studies to be involved, may take up to one year for these results. Highly scientific. $1,000,000 Gold refutation. Highly significant refutation at p<0.01, with extreme amounts of data to be published. Should become the gold standard for data in the field, and the most significant effect published. Full refutation of damaging/negative result, with full scientific rigor. Should be widely promoted. May take up to 24 months to produce data. Gold standard of science. ACADEMIC BONUS: prices are free for tenured professors, thousands of whom can do all the studies they want and only publish if they see some significant effect. EDIT: in other words, http://xkcd.com/882/ http://xkcd.com/882/

152 points by tokenadult 12y ago

There was a question yesterday about Evidence Based Medicine vs Science Based Medicine. The SBM criticism of EBM is the over-reliance on Randomized Controlled Trials that meet p=0.05, without looking at the prior probability that a treatment would help. For example, EMB would say that if you have an RCT that shows that a lucky rabbit's foot works, then you have reasonable evidence to put that into practice. The issue is that, as this article points out, even with a statistically significant result for such research, there's no plausible mechanism by which a rabbit's foot makes you lucky. Therefore the RCT is just one part of the whole picture, and subject to a specific type of manipulation.

srunni 12y ago
Something related to keep in mind when it comes to p-values is the false discovery rate: > If you run lots of phase 2 trials with different drug candidates where only a minority (lets say 10%) actually work, then with standard trial statistics (80% power and 5% false positive rate) you will get 4.5% false positive and 8% true positives – so less than 2 out of every 3 positive trial results were real. A much lower success rate than the 5% error rate commonly assumed. http://www.forbes.com/sites/davidgrainger/2015/01/29/why-too-many-clinical-trials-fail-and-a-simple-solution-that-could-increase-returns-on-pharma-rd/ http://www.forbes.com/sites/davidgrainger/2015/01/29/why-too... This recent (open access) review in R. Soc. Open Sci. the Forbes article was largely inspired by is worth a read too: http://rsos.royalsocietypublishing.org/content/1/3/140216 http://rsos.royalsocietypublishing.org/content/1/3/140216
Alex3917 12y ago
> The SBM criticism of EBM is the over-reliance on Randomized Controlled Trials that meet p=0.05, without looking at the prior probability that a treatment would help. The strength of significance testing is that it purposely doesn't try to tell you how likely something is to be true, only how likely the data you got was the result of chance assuming the treatment is no better than placebo. You're still taking the prior probability into account when trying to figure out the truth, you're just not putting a number on it. My concern with bayesian approaches is that, like with frequentist approaches, the truth is still fundamentally unknowable, only now you're encouraged to put a number on that and pretend that it's science. While bayesian approaches totally make sense in trying to determine a patient's likelihood of having some disease when there is already data available for the prevalence in a population and the sensitivity and specificity of the tests, using bayesian logic to weight clinical trials strikes me as being highly dubious. It would be one thing if SBM actually developed a framework to give a weight to each methodological feature of a trial, but so far I haven't seen much work to build a functioning system. Though if you're really honest about all the ways that you can have positive results without something actually being true, it seems like almost no amount of research will ever have a significant effect on the prior.
- mdbco 12y ago
  You're absolutely correct that Bayesian approaches are not magical and do not suddenly supply you with vastly more information than frequentist approaches (particularly when you have a really poor prior, in which case the Bayesian approach will be similarly poor). Bayesian statistics is certainly very popular right now, but it should not be looked at as some sort of panacea for all statistical problems. However, I would say that Bayesian approaches do have a big advantage in terms of helping with the interpretation problems that plague frequentist significance testing. Namely, as the OP article points out, Bayesian approaches reformulate the testing question in a way that is more intuitive, i.e. "what is the probability of the hypothesis given both the prior probability and the new data?". So yes, Bayesian methods surely do not fix everything, but since interpretation of statistics is such a major concern, they can be quite beneficial.
- yummyfajitas 12y ago
  You're still taking the prior probability into account when trying to figure out the truth, you're just not putting a number on it. This is a danger sign - you are doing the same things the Bayesians do, just informally, less explicitly, and probably incorrectly. The fact is that to make a good decision, eventually you need to compute a single number. This is an elementary fact of topology: https://www.chrisstucchio.com/blog/2014/topology_of_decisionmaking.html https://www.chrisstucchio.com/blog/2014/topology_of_decision... That number will be based on some unproveable assumptions. That's a fact of Godel's incompleteness theorem, if nothing else. So given this, why is it "dubious" to make those assumptions explicit and obvious?
  - algorias 12y ago
    > The fact is that to make a good decision, eventually you need to compute a single number. Your linked blog post states that if you make a good decision, then there is a process computing a single number which is equivalent to your process. This is not equivalent to what you claim. As a matter of fact, it's the same kind of confusion that exists around the p-value. It's not the case that a process explicitly computing such a number automatically makes good decisions, which is what you seem to claim implicitly. Also, Gödel has nothing whatsoever to to with this.
WhitneyLand 12y ago
But how could a randomized clinical trial make this mistake? What experimental design could make the rabbit's foot look good? "EMB would say that if you have an RCT that shows that a lucky rabbit's foot works, then you have reasonable evidence to put that into practice."
- sunir 12y ago
  You have heard "19 times out of 20" described in the news? That is the 0.05 restated for laypeople. 1 time out of 20 you will get a false positive, in this case that the rabbit's foot worked.
  - WhitneyLand 12y ago
    Sorry maybe I'm being dense, but who would take 1 out of 20 success to mean they should start buying rabbit feet?
    - Mvandenbergh 12y ago
      Nobody. The problem is that if you conduct 20 trials of the efficacy of rabbit feet, you'd expect 1 trial to show a significant effect. (if you're interpreting P-values the way many people do, incorrectly.)
    - DanBC 12y ago
      When you do a single trial you might show that rabbits feet are effective. You don't yet know if this is a 1 in 20 result or a 19 in 20 result. That's why you replicate.
    - ejain 12y ago
      You'd start buying rabbit feet because only the study that was "successful" is published.

analog31 12y ago

I wonder if it just boils down to this: Exclusive reliance on any single tool by an entire field for a long enough time period will eventually lead to a proliferation of bad results.

rcthompson 12y ago
More like exclusive reliance on a single tool will lead to a requirement for everybody to use it, even if they don't understand how to use it properly, which leads to both unintentional misuse (by experimenters who don't understand it) and the inability to catch intentional misuse (because adjudicators don't understand it either).

po 12y ago

The audio is a bit quiet but the 'dance of the p values' video [1] linked to by the article is absolutely fantastic and well worth watching. [1] https://www.youtube.com/watch?v=ez4DgdurRPg https://www.youtube.com/watch?v=ez4DgdurRPg

MengerSponge 12y ago

Good! This is a big step in improving everybody's work. If you aren't up on this research, here are the two articles that you need to read. Why Most Published Research Findings are False: http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 http://journals.plos.org/plosmedicine/article?id=10.1371/jou... Revised Standards for Statistical Evidence: http://www.pnas.org/content/110/48/19313.short http://www.pnas.org/content/110/48/19313.short

mdbco 12y ago

The article is certainly correct that p-values and confidence intervals (or confidence sets, in multi-dimensional contexts) are widely misunderstood, not just in psychology or other social sciences, but in the hard sciences as well. The problem is even worse when you look outside of academia at common practices in more applied settings. As suggested, a good approach is to take p-values not as conclusive or decisive, but rather as a tool that must be supplemented by other statistics. In particular, the article emphasizes Bayesian methods, which can certainly provide additional information, but this approach can also be rather limited when priors are not well-defined or are entirely unknown, which is unfortunately often the case in many problem domains. One potential question is how to determine the nature of the distinction mentioned in the conclusion between "preliminary research" and "confirmatory research", particularly in cases where statistics provide the primary evidence, as in, e.g. psychology. Further studies in the same vein as the preliminary research can certainly provide additional supporting statistical evidence, but this doesn't escape the problem that all of the evidence is probabilistic in nature. The key issue here is that since statistical approaches can only give probabilistic evidence that a hypothesis is correct, then they strictly cannot tell you what is certainly true, so even confirmatory research is quite open to falsification. So we wouldn't want the label of "confirmatory research" to somehow suggest to the public the idea that it is certainly correct.

deleted 12y ago
[deleted]
esfandia 12y ago
Is the cautious approach then to treat a p-value in the absence of priors on the same level as a p-value in presence of unfavorable priors? When someone tests positive for a cancer test, the priors are known (probability of cancer in the general population is usually very low, and the false positive rate of the test may be relatively high), and so usually that first test is merely indication that further tests are needed. So when you don't know the prior and you observe a low p-value on something, isn't that just "preliminary research" that needs to be further confirmed with other methods or at least the same test but using other data?
- 12y ago

Kenji 12y ago

The problem is this: You need a solid background in statistics and some mathematics to perform these tests properly. Most apparently, people from that field lack these skills. If someone can't hit a nail on the head with a hammer, blaming and banning the use of hammers won't solve a thing.

manicdee 12y ago
Taking hammers away will solve the problem of holes in things which aren't supposed to have them, including thumbs, walls, people's heads, and candy jars.
- zmjones 12y ago
  there are just other tools that will get abused. bayes factors for example
viraptor 12y ago
Depends what are you trying to solve - people's understanding of p-values, or many false results? If you care about getting true results and there are easier ways to get there than p-values, why would it be a bad solution? Similar thing to what happens in software security. Sure, you could tech people how to use memory management properly - and then watch them fail again and again. Or you could just provide an alternative solution like rust which changes the issue completely.

VieElm 12y ago

I kind of which that the widely read publications, like newspapers, that have turned out science articles based on all these now pretty much discredited research articles would issue retractions or at least clarification for each of the articles that has been challenged in this regard.

CountBayesie 12y ago

>The type of analysis being banned is often called a frequentist analysis I find that there is a trend of associating "bad statistics" with "Frequentists Statistics" which isn't really fair. If you found a statistician trained only in Frequentist methods and asked their opinion on experiment design in psychological research they would likely be just as appalled as any Bayesian. I'm a big fan of Bayesian methods, but the solutions of "we'll solve the problem of misunderstanding p-values by removing them!" is still a problem of misunderstanding p-values! The misunderstanding is the issue, not the p-value.

deleted 12y ago
[deleted]
colechristensen 12y ago
I think what this journal doing is probably a good thing, but only as the lesser of several evils. The truth is something more like... it is easier for soft sciences to abuse frequentist statistics than bayesian. Both have merit, it cannot be argued, but it is simply easier to produce meaningless conclusions with frequentist statistics done wrong. This situation is so bad that it merits banning frequentist for this journal and I think that's reasonable. This doesn't mean that every journal in every field should, but perhaps it will be a useful temporary measure to improve quality.
stdbrouw 12y ago
The problem is that p-values are begging to be misunderstood, and in fact you cannot use them as a decisionmaking procedure without "misinterpreting" them – after all, you're deciding whether to accept the hypothesis P(HA|D) based on 1-P(D|H0) on the grounds that, while they're not the same, they're proportional. (In that sense the p-value is like the poor man's likelihood ratio.) There's nothing wrong with p-values as a concept, but there's everything wrong with p-values in hypothesis testing. The misunderstanding is baked in.
- im3w1l 12y ago
  You can update your posterior based on the p-values yourself though. "Well those eggheads may have disproved X, but X is just common sense, so I'm gonna keep believing it anyway. U-until I see more studies confirming the finding I mean."

rcthompson 12y ago

I spent about a hour explaining p-values to a fellow graduate-level researcher a few weeks ago. I pretty much just kept rephrasing the definition in slightly different ways until the person finally got it. In undergrad, hypothesis testing was more or less taught as "do this inscrutable calculation and if the result is 0.05 or less, you win". The point is, in my experience, a lot of people really don't get p-values, even graduate-level and post-graduate scientists.

viraptor 12y ago
I agree. I'm in the pool of people who don't really understand p-values even after taking 2 statistics courses. The main issue I've noticed is that compared to many other calculations, you just have no idea if your result is correct. You can make wrong assumptions, wrong calculation, apply wrong methods, but in the end you get a number... and that's it. It may be a wrong number and you'll never know. This is completely different to many other practical applications of math where you can verify your result in various ways, or validate your answer against the initial assumptions, or test your program against lots of inputs.
- mdbco 12y ago
  It's actually entirely possible to do some "checking of your answer" for p-values, as well. As you mentioned, for practical math, you can often validate your answer against the initial assumptions. This is true for statistical testing too, as it typically relies on many theoretical assumptions. So what you can do in practice is propose a different set of assumptions, perform the hypothesis test in a manner that follows those new assumptions, and see if you obtain a similar result. Typically, for any given hypothesis that you want to test, there are several possible methods for performing that test, so you can redo your test many times. This is one type of robustness checking, which includes many other things as well (e.g. running your test over subsamples or resamples of the data, checking for sensitivity to outliers, etc). Good statisticians generally like to do lots and lots of robustness checking.
- yummyfajitas 12y ago
  The main issue I've noticed is that compared to many other calculations, you just have no idea if your result is correct. You can make wrong assumptions, wrong calculation, apply wrong methods, but in the end you get a number... and that's it. The same is true in Bayesian statistics, and even simple formal reasoning with no statistics in sight. If you make wrong assumptions, you'll get the wrong result. The only thing you can expect statistics to do is help you change your opinion about the relative merits of opposing theories. If both your opposing theories are wrong, you will still be equally wrong. The true flaw with frequentist statistics is that it goes out of it's way to hide this fact from you. In contrast, Bayesian stats forces you to explicitly choose a prior, enumerate your assumptions, and accept that your conclusion is based on these things.

cafebeen 12y ago

Here's the key passage: <quote> BASP will require strong descriptive statistics, including effect sizes. We also encourage the presentation of frequency or distributional data when this is feasible. Finally, we encourage the use of larger sample sizes than is typical in much psychology research, because as the sample size increases, descriptive statistics become increasingly stable and sampling error is less of a problem. </quote> In other words: report the effect size, plot the data, and increase the N.

deleted 12y ago

[deleted]

torgoguys 12y ago

Obligatory xkcd: http://xkcd.com/882/ http://xkcd.com/882/

jwilliams 12y ago

I think this Wikipedia section summed it up really well for me: http://en.wikipedia.org/wiki/P-value#Alternating_coin_flips http://en.wikipedia.org/wiki/P-value#Alternating_coin_flips Extreme example, but explains how the context can shift the findings significantly.

kriro 12y ago

This is very interesting for me. I recently switched from AI to Human Computer Interaction which is more cross discipline, specifically with a strong influence from psychology. I'll gladly admit that I don't have the best understanding of NHST but I think I understand it well enough. Interestingly enough there was a post on HN somewhat recently about Bayesian alternatives (BEST). The paper that was linked was: ftp://ftp.sunet.se/pub/lang/CRAN/web/packages/BEST/vignettes/BEST.pdf And the recommended book I settled on was by the author of that paper (Doing Bayesian Data Analysis) I feel like I'm "ahead of the curve" thanks to HN :D

deleted 12y ago

[deleted]

stdbrouw 12y ago

Banning p-values makes sense to me, as they force you to declare an effect as either significant or not significant, rather than looking at the preponderance of the evidence and building knowledge over multiple experiments. It also leads us to focus on statistical uncertainty at the expense of all of the other kinds of uncertainty researchers are faced with: do I have the theory to back this up, am I actually measuring what I am trying to measure, is this coherent with other findings in the field? I do think the editors might be right when they say banning p-values will make the quality of the research go up, not down. But if you read the original editorial, up at http://www.tandfonline.com/doi/pdf/10.1080/01973533.2015.1012991 http://www.tandfonline.com/doi/pdf/10.1080/01973533.2015.101..., you can see that they also reject confidence intervals and Bayesian reasoning with uniform priors (which is really the same thing) without providing any guidance at all on better procedures. I fear that will just lead readers to try and guess the reliability of the data themselves, or worse, interpret the sample statistics as numbers without any associated uncertainty. So they're doing away with poor statistical procedures, but at what cost? It's like that old joke: we've found a 100% reliable cure for cancer – bombing the planet until everybody's dead.

nabla9 12y ago
Methodological flexibility in statistical modeling has gone trough the roof. New tools and estimation methods make Bayesian methods easy to use: Jags and Stan, Hamiltonian Monte Carlo, the variational Bayesian approximations, expectation propagation and even probabilistic programming (note: Bayesian in this context does not mean subjectivity. It's the ability to quantify uncertainty in the results and increased flexibility in the modeling). New methods and tools are faster to use and give better results, but they require more statistical knowledge. If the scientist applying these methods or peer review can't understand the advances, it's all for nothing. Many sciences are methodologically very conservative to the extent that it holds the science back. How do you increase the statistical knowledge of the field so much that peer review and researchers can be expected to understand and use the new methods if they can't be trusted with p-value?

Booktrope 12y ago

If the stats aren't based on samples of real-world data, rather than very rigorously designed experimental comparisons, I think there's another reason to be wary of p-values. [Note, in what follows I probably use some terminology wrong, because I'm not a statistician, but I do think the point is important and I don't see much written about it.] In the real world data is not a bunch of independent events, but events (or data points) that are interconnected in ways that are difficult to quantify. I once heard a presentation by an expert in AB testing at a leading tech company who became wary of his results and brought his concerns about non-independence of events that were being tested to corporate statisticians. (He was concerned that, even though the AB testing procedure supposedly randomized the test, interconnectedness within website traffic was not being accounted for.) By his account, the statisticians agreed it was a problem but recommended that he assume that variations due to non-independence would more or less balance each other out. He wasn't satisfied with this and said that he went ahead and did some more measurements, then ran out all the binomial expansions rather than relying on approximations. When he did this more detailed work, he found out that with at least some web-based AB tests, where conventional statistical formulas showed a p-value of .05, he was getting a confidence level of more like 30%. (I don't think you could call his measurement a p-value because he wasn't using the formulas normally used to compute p-values, but I think what he was saying was more or less that the p-value from the forumalae was .05 when it should have been .30 from a more rigorous look) As I'm not a statistician by trade I don't keep up on the literature very well, but interconnectedness of data does seem to me to be a very important issue. I'm wondering if anyone can point me to some helpful reading to understand this side of the issue better. In particular, is there any approach to AB testing that can reliably address the issue of data interconnectedness in the kind of situation described above?

stdbrouw 12y ago
Could you expand on what you mean by "interconnectedness within website traffic"? If one person visiting the site has no influence on other people visiting the site, then measurements of their behavior will be independent. If Facebook tests a different interface on half of their users and the changed behavior of those users indirectly has some impact on the behavior of the control group, then your measurements would have some level of dependence – I can imagine that this could happen but it's not clear to me that this would be a common scenario. The same would happen if you measure the behavior of the same person more than once – but in this case there's many procedures for working with paired or autocorrelated data.

chrisseaton 12y ago

I never understood where 0.05 came from. It seems like an arbitrary, magic number. Aren't those bad in science? Shouldn't we have some reason for every number we use? Why 0.05 instead of 0.049?

nzp 12y ago
It is somewhat arbitrary. 0.05 corresponds to roughly 2, and 0.01 to roughly 3 standard deviations under the normal distribution. In high energy physics, for example, in the past a result with 3 sigma significance was considered “a discovery”, but it turned out there were a lot of false signals, so it was upped to 5 sigma (~ 3 x 10^{-7} percent chance of getting the effect assuming there is no effect in reality), while 3 sigma (or 4) result is considered “evidence of effect“. Unfortunately in softer sciences, considering their relatively generally poor state of research standards, we have no idea if 0.05 or 0.01 levels really are good standards of “discovery”.

return0 12y ago

i think this is also important on a conceptual level. There's just too much literature in life sciences nowadays which, instead of formulating bold hypotheses with clearly distinct effects, is happy to report miniscule effects that just crossed the arbitrary significance barrier.

MollyR 12y ago

I might be missing the forest from the tree's. The paper talks about how it seems researches are "hacking" (their word) p-values.If the researchers lack the ethics using one form of statistics, what is really stopping them from misusing Bayesian Analysis? Sidenote: While we are talking about bayesian stuff. I recently ran into the sleeping beauty problem ( http://en.wikipedia.org/wiki/Sleeping_Beauty_problem http://en.wikipedia.org/wiki/Sleeping_Beauty_problem ) and it showed how different interpretations of an experiment can lead to people believing in two very different answers in an almost religious way. Some people argued this thought experiment shows a clear flaw in bayesian thinking. I'm not sure either way, I'm still thinking about the puzzle.

Gravityloss 12y ago
That's a cool problem. It took a while to understand the angle. I'd say I'm a halfer. I think the extreme sleeping beauty problem highlights how if you're woken up, it's not any more probable it's a head or a tails awakening, since, for a reason I can't explain, the million tails awakenings "don't accumulate". In a sense, it's like the opposite of the Monty Hall problem, since here the sleeping beauty receives no information whatsoever during the experiment.
- MollyR 12y ago
  I'd say I'm a halfer too, but a lot of physicists/philosophers say 1/3, and it really bothers me that I can't seem to sync up with world leaders. source : http://rfcwalters.blogspot.com/2014/08/the-sleeping-beauty-problem-how.html http://rfcwalters.blogspot.com/2014/08/the-sleeping-beauty-p...

logicallee 12y ago

the main reason to ban significance testing in all fields is because of logicallee's getcher scientific results agency. Our prices are: $10,000 random study, no results guaranteed. Not recommended! Highly likely to be damaging. $20,000 basic study, p=0.5 - study inconclusive but implies it's at least not "more likely" that the damaging/negative result is correct. (Assuming uniform priors.) No scientific value. $100,000 weak FUD. Suggests that the damaging/negative results (assuming bayesian reasoning or uniform priors) may be incorrect, at a suggestive p<0.10 level. Not conclusive and invites further studies which can strengthen damaging/negative result! Not recommended unless further studies are unlikely. Consists of ten studies, one of which is published with remaining buried. Unscientific. $200,000 basic refutation. Refutes the damaging/negative result at a statistically significant p<0.05. Likely to be referenced and accepted. However, due to significance level, links to logicallee's getcher scientific results agency should still be avoided! (Invites skepticism.) Scientific. May take up to six months. $500,000 Silver refutation. Highly significant refutation at p<0.02. Study can be extremely rigorous and sponsorship can be public. The results should be referred to and referenced as widely as possible. Unpublished results (the other 49 studies in the series) to remain unpublished and unreferenced. Due to the number of parallel studies to be involved, may take up to one year for these results. Highly scientific. $1,000,000 Gold refutation. Highly significant refutation at p<0.01, with extreme amounts of data to be published. Should become the gold standard for data in the field, and the most significant effect published. Full refutation of damaging/negative result, with full scientific rigor. Should be widely promoted. May take up to 24 months to produce data. Gold standard of science. ACADEMIC BONUS: prices are free for tenured professors, thousands of whom can do all the studies they want and only publish if they see some significant effect. EDIT: in other words, http://xkcd.com/882/ http://xkcd.com/882/