26 ms·

P values are not as reliable as many scientists assume

• gwern 13y ago
  Further reading: http://lesswrong.com/lw/g13/against_nhst/ http://lesswrong.com/lw/g13/against_nhst/
• 66ff99 13y ago
  >The more implausible the hypothesis — telepathy, aliens, homeopathy — the greater the chance that an exciting finding is a false alarm Implausible according to who?
• michaelochurch 13y ago
  Many frequentists tend to Bayeslessly trust them.
• snowwrestler 13y ago
  Statistics are descriptive, not predictive--period. I'm continually surprised at how many people either don't know, or don't internalize, that. Look at how often "risk factors"--which are a descriptive concept--are converted to advice--which is predictive. Doing so in the absence of a causal hypothesis is a basic violation of "correlation does not equal causation." If you want to construct a scientific theory you must be able to articulate some predictive tests, and that means you must hypothesize a causal mechanism.
  • jules 13y ago
    Yet the only reason we are interested in statistics is for the predictive aspect. This is exactly where hypothesis testing goes wrong: as much as you may claim that you're doing something purely descriptive, the whole point of the exercise is to make decisions. Hypothesis tests simply don't give us the right information to base a decision on, but in practice people still do make decisions based on them in a fundamentally incorrect way.
  • baddox 13y ago
    > Statistics are descriptive, not predictive--period. Does the law of the large numbers somewhat unify descriptiveness and predictiveness?
  • cscheid 13y ago
    That's because your short statement is not really representative of good statistical practice. For example, people spend a lot of time researching http://en.wikipedia.org/wiki/Generalization_error http://en.wikipedia.org/wiki/Generalization_error, and models like http://en.wikipedia.org/wiki/Probably_approximately_correct_learning http://en.wikipedia.org/wiki/Probably_approximately_correct_... worry a lot about things like VC dimension exactly because they characterize the behavior of statistical models to unseen data. Or maybe you don't think of prediction as "behavior under unseen data"?
  • cschmidt 13y ago
    As in "I used to think correlation implied causation. Then I took a statistics class. Now I don't..." http://xkcd.com/552/ http://xkcd.com/552/
    • snowwrestler 13y ago
      The reason that comic is funny is because there is a known mechanism that we would expect to cause that effect: teaching.
      • dredmorbius 13y ago
        Which is why the alt text on that particular comic is so key. Correlation suggests that if you're looking for causation, it might be somewhere over here. It doesn't insist that the two are the same, but if you're looking for clues, it's a hell of a dowsing rod.
  • mtdewcmu 13y ago
    True-- Correlation does not imply causation. But-- Correlation implies publication.
  • dllthomas 13y ago
    Reliable correlation is all you need to make predictions. If you understand causation, you can be more confident about the reliability (or lack thereof) of the correlation under changing conditions.
• relaunched 13y ago
  The last sentence in this paragraph is hilarious: P values have always had critics. In their almost nine decades of existence, they have been likened to mosquitoes (annoying and impossible to swat away), the emperor's new clothes (fraught with obvious problems that everyone ignores) and the tool of a “sterile intellectual rake” who ravishes science but leaves it with no progeny3. One researcher suggested rechristening the methodology “statistical hypothesis inference testing”3, presumably for the acronym it would yield.
  • pcrh 13y ago
    As is: "One researcher suggested rechristening the methodology “statistical hypothesis inference testing”, presumably for the acronym it would yield."
    • Nicholas_C 13y ago
      I'm guessing you didn't read the last sentence of the paragraph in the post you were replying to.
      • pcrh 13y ago
        Oops!
• loderunner 13y ago
  Another reason to be skeptical of the statistics thrown around in popular news.
  • pacaro 13y ago
    Unfortunately it's not just the nightly news one needs to be skeptical of, it's also (amongst many others) your doctor—who probably isn't particularly well educated w.r.t. statistics, but who will nonetheless give advice and prescribe treatments based on them.
    • mattfenwick 13y ago
      This is a great -- and scary -- point. These statistical fallacies and misunderstandings are so deeply ingrained into our scientific and medical systems that it's hard to see how and when they will be removed. I can attest from personal experience that many scientists 1) don't understand these statistical tests, 2) don't care to find out, and 3) don't think there's a problem.
• JoshTriplett 13y ago
  I'd love to see a comprehensive article that shows what a research paper's analysis would look like using Bayesian methods. I've seen plenty of general hints about Bayesian methods, discussion of priors, and similar, but I haven't found any specific guide on how to apply those methods to the types of research papers that would traditionally use a null hypothesis significance test with a p value.
  • Homunculiheaded 13y ago
    Not articles but there are two very excellent books on the subject that I can't recommend enough: If you read calculus with about the same fluency as the comic books then "Data Analysis: A Bayesian Tutorial" is awesome http://www.amazon.com/Data-Analysis-A-Bayesian-Tutorial/dp/0198568320 http://www.amazon.com/Data-Analysis-A-Bayesian-Tutorial/dp/0... And if you would like a little more exposition (but still a mathematically sophisticated treatment) "Doing Bayesian Data Analysis: A Tutorial with R and BUGS" is fantastic http://www.amazon.com/Doing-Bayesian-Data-Analysis-Tutorial/dp/0123814855/ http://www.amazon.com/Doing-Bayesian-Data-Analysis-Tutorial/... The latter will also give you more details of how to approach classical, frequentest tests and summary statistics with their Bayesian equivalent. Honestly I would say get both books as they're cheap and provide different insights. You only need to read a few chapters of each to see how you approach basic experiments from a Bayesian perspective.
  • klodolph 13y ago
    As a complement to pure Bayesian reasoning, I recommend Mayo's "Error and the Growth of Experimental Knowledge". http://www.amazon.com/Experimental-Knowledge-Science-Conceptual-Foundations/dp/0226511987 http://www.amazon.com/Experimental-Knowledge-Science-Concept...
  • Fomite 13y ago
    There's a number of tutorials scattered throughout the International Journal of Epidemiology, Epidemiology and the American Journal of Epidemiology, as well as good "exemplar" articles either using Bayesian methods, or using both approaches.
  • gwern 13y ago
• Blahah 13y ago
  This is probably the most annoying problem in my daily life (yeah I know, first-world problems). I have daily conversations with biologists where I've analysed some data and associated (say) a posterior probability with each condition in the model. They insist I give them p-values or something they can present as though they were p-values. At the beginning of my PhD I complied. Then they throw out all the nuance in the data, put one asterisk for p < 0.05, two asterisks for p < 0.01, etc., and (this is the horrifying part) _believe_ that an asterisk indicates that something is true. They put stupid asterisks all over my beautiful plots and then think their arbitrary cutoffs mean something biologically meaningful. I die a little inside every time I see an asterisk on a plot. Now I refuse to use p-values and deliberately construct analyses that are incompatible with Fisherian statistics. And rather than giving people raw numbers, I produce a massive document of interpretation. Takes a huge amount of time, but I'm hoping it will mean my publishing track will contain significantly (ha!) fewer false results than most biologists'.
  • km3k 13y ago
    > _believe_ that an asterisk indicates that something is true Attitudes like this bug me so much. From my experience it is a big part of what makes people that are not familiar with science not trust scientists.
    • vacri 13y ago
      The story is so much simpler than that. People not familiar with science don't trust scientists because they want to stick to their own known story. Change is scary, and people generally don't like it unless it significantly improves their living conditions. It doesn't matter how rigorous scientists are with their methodology, my 'new age' mother isn't going to stop believing in fairies in the trees because scientists say there's no proof. Or when she talks about higher and lower vibrations, she is puzzled when I ask her "vibrations of what? what is it that's vibrating?". This isn't important to her story, and is merely a detail that makes life less enjoyable, so she ignores it. A less obnoxious version of ICP's "scientists are ruining life by explaining things" during their 'how do magnets work' phase. Saying that people don't trust scientists because of p values is a bit like saying people don't trust plumbers when they use blue plumber's tape instead of gray. People don't like plumbers because they don't show up on time and charge through the nose - things that change your own story.
• socrates1998 13y ago
  I have always thought people rely on the normal distribution too much. Does it work? Sometimes. The problem is that people tend to believe something they use a lot. Even the 0.05 threshold is sort of made up. Correlation does not mean causation.
  • jheriko 13y ago
    p-values need not arise from the normal distribution - but any distribution - selecting the right one is another cause for error when producing them. the normal distribution is also quite well justified by the central limit theorem. i do however agree that a p-value of 0.05 is not worth very much.
  • jessaustin 13y ago
    Careful, don't poke the hornet's nest: https://news.ycombinator.com/item?id=7064435 https://news.ycombinator.com/item?id=7064435
• jawns 13y ago
  This is the reason why I don't include P values on http://www.correlated.org http://www.correlated.org. They would muddle my otherwise irreproachable statistics.
  • mbateman 13y ago
    Holy crap this site is fantastic. Thanks.
  • klodolph 13y ago
    I bet the p-values on that site would be very high if properly calculated. Generally, if you are reporting correlations between a large number of variables, the p-values shoot through the roof. Of course, TONS of people forget this and publish a p-value as if those two variables are the only ones under consideration. Which is just sad.
    • singingfish 13y ago
      With correlation statistics the p value is proportional to the sample size.
      • klodolph 13y ago
        I think we're talking about different experiments. I'm not talking about, "here are the correlation statistics for 100 variables", I'm talking about, "I tested 100 variables, and these are the five pairs which are most strongly correlated."
        • singingfish 13y ago
          At which point you need to do some variant on principal components analysis on the covariance matrix.
  • moontear 13y ago
    Woah, why didn't I know about this site? Awesome! Would be great if you'd include links to you original sources. Even though you don't include P-values in the graphics, on the detail page I would still love them just as much as Chi-square. Your tagline should be "correlation does not imply causation" ;-)
• gabemart 13y ago
  I was recently trying to explain Bayesian logic to a friend, and came up with the following analogy. I would be interested to hear feedback on it. --- Imagine everyone in the USA gets sudden amnesia. We want to find out who the President is, but no one can remember. A scientist comes up with a test to determine if someone is the President. If they are the President, there is a 100% chance the test will say they are the President and a 0% chance the test will say they are not the President. If they are not the President, there is a 99.999% chance the test will say that are not the President, and a 0.001% chance the test will falsely say they are the President. Giving the test to the person sitting in the big chair in the Oval Office is useful, because it's already quite likely this person is the President. If the test is positive for Presidency, it's extremely likely that person is the president. Giving the test to the 10 people nearest the oval office is useful, because it's fairly likely the President is one of these people. A positive result will indicate strongly that that person is the President, and if no-one in that group is actually the President, there's a 99.99% chance the test will say so. Giving the test to the 1000 people in the White House is pretty useful, because it's pretty likely the President is in the White House, and if none of these people are the president, there's still a 99% chance the test will be correct. A positive result for any one person will indicate quite strongly that that person is the President. But giving the test to everyone in America is not very useful at all, because it's very unlikely that any particular person is the President, and we can expect the test will give a positive result for around 3200 people. For any particular person in this group, it's much more likely they're not the President than they are. --- Is this a broadly correct, if non-rigorous, analogy? I realize most HNers will be much more familiar with this stuff than I am, I'm interested chiefly in whether or not I misled my friend.
  • tlarkworthy 13y ago
    yeah its similar to a classic rare disease screening example in text books: http://www.math.hmc.edu/funfacts/ffiles/30002.6.shtml http://www.math.hmc.edu/funfacts/ffiles/30002.6.shtml (sorry its a bit informal, it was first google hit, but I have seen that in real textbooks for sure) Although its a bit confusing associating the prior with geometric proximity to the oval chair.
  • klodolph 13y ago
• Finster 13y ago
  > In 2005, epidemiologist John Ioannidis of Stanford University in California suggested that most published findings are false; since then, a string of high-profile replication problems has forced scientists to rethink how they evaluate results. That's what is supposed to happen, though, right? You publish your findings. Others try to reproduce. They publish THEIR findings, etc. etc. If most published findings are false, it sounds like the process is working as designed.
  • pessimizer 13y ago
    Bad papers can be generated. published, and cited a lot faster than failures to replicate.
    • Finster 13y ago
      So, doesn't that mean we should be blaming idiotic news media outlets that tend to generalize and publish scoops on scientific papers too quickly?
      • capnrefsmmat 13y ago
        Yes and no. Yes, the media sucks, but no, most scientific papers are not replicated -- or if they are, it's by a hapless grad student using the results in their research, only to find they don't hold up. The hapless grad student usually doesn't get to publish this, because negative results are boring and not usually published in prestigious journals. Most scientists have better things to do than replicate previous findings, unless that previous finding directly bears on their own work.
• cwyers 13y ago
  If you're interested in the subject, some additional reading (warning, PDF): http://www.deirdremccloskey.com/docs/jsm.pdf http://www.deirdremccloskey.com/docs/jsm.pdf
• TempleOSV2 13y ago
  Niggers. God says... mocking dang_it these_cans_are_defective oil envy BBC news_to_me ouch slumin whale et_tu pet you_hoser here_now earnest ouch hypocrite astrophysics bye arrogant didn't_I_say_that gluttony
• jheriko 13y ago
  a p-value of 0.05 or even 0.01 is stupidly high. it only takes a little thought experiment about what that means in reality to realise how permissive it is and you can find demonstrations of this without going particularly far, looking very hard or being especially well educated... consider the wikipedia example with heads vs. tails. http://en.wikipedia.org/wiki/P-value#Examples http://en.wikipedia.org/wiki/P-value#Examples the idea that 5 coin tosses can produce a p-value < 0.05 that 'demonstrates' that the coin is biased towards heads is intuitively 'obviously wrong'. even if we take it to 10 coin tosses (the p-value you get is 0.001 - which looks really strong if we accept that 0.01 is acceptable) it clashes with my own ideals for what statistical significance should mean. this is in a loose way a proof by contradiction that p-values of 0.05 or 0.01 do not have utility (at least for these kinds of small n). aside from that consider running the experiment 5 times or 20 times. how many false positives do you expect? what is the expected number of false positives? is that significant? it also bothers me how connected to the problem formulation that the value itself is. if we analyse the same situation with an identical test but a different formulation of the problem that the values differ? why is five heads in a row less significant as a result when the test is whether a coin is biased at all rather than a test that it is biased towards heads only? sure i understand the probability involved there that we have all these potential coins biased towards tails that mean nothing in the first case - but there is something very deeply wrong with that. shouldn't this be the other way around? if 5 consecutive heads is good evidence that a coin is biased towards heads, isn't it equally good evidence that it is biased at all? classical logic says that it is because being biased towards heads is a subset of being biased in either direction. the truth is that it really is equally good evidence - i challenge someone to explain why it is not! ( actually i kinda want to be wrong about that because i might learn something new then :) ) probability is counter-intuitive and useless for the kinds of small n usually used in experiments - the intuition about it recovers when we deal with sensible n - numbers like 1000 or 10000 - but these are still small n really if you need to scale up, or be confident that your result is correct. even at 100 samples its obvious that our idealisation of percentage and what happens in reality do not marry up neatly... to make a very crude software analogy what about those 1 in 10,000 bugs? they are still a very real problem if you have millions of customers... or - IMO even 10,000 is a very exceedingly small n to try and draw robust conclusions from.
• shas3 13y ago
  The root of this problem is that most data sets in psychology, anthropology, and epidemiology are not as large in terms of sample size as what computer scientists and electrical engineers encounter. p-values are a surrogate to explicitly describing the data using probability distributions or as random processes. In essence, you sacrifice granularity for simplicity. If you look at the original works of Fisher, etc. and their widespread utility, a large part of early statistics is intended for 'practical statisticians' who seldom encounter data-sets that are 'small' in terms of sample size. As someone who works in electrical engineering/computer science, I've never used the p-value because: 1. The field, in general, demands far more mathematical rigor when dealing with statistics. 2. The demand for mathematical rigor is justified because most data sets we deal with are many orders of magnitude larger than what psychologists and others encounter. So predictions based on limit theorems, etc. are often testable.
  • Fomite 13y ago
    As an Epidemiologist, some of the most appalling uses of statistics I've ever seen have been in electrical engineering and CS.
    • shas3 13y ago
      Can you elaborate?
      • Fomite 13y ago
        A little late in coming (blame the Southeast U.S.'s snowstorm), but I work in a very stats heavy field and interact with a number of CS types because I work on computational models. I've gotten a fair amount of "I just need a p-value" requests, and some assumptions that everything can be hit with a t-test or an ANOVA and it'll all work out fine.
• mandor 13y ago
  I fully agree with the critics about the p-values, but what are the best alternatives to analyze and compare data? Most of the time, scientists have to compare the outcome of treatment 1 versus treatment 2; how should they do it "properly"? What is the HN recommandation?
  • Fomite 13y ago
    Effect measures. Don't just report your p-value, report the actual effect measure, and a measure of uncertainty around it, be it a frequentist confidence interval, Bayesian posterior distribution, etc. More information is better.
    • capnrefsmmat 13y ago
      Agreed. A good reference is Geoff Cumming's "Understanding the new statistics," although its focus on analysis in Excel may put off the typical HN audience. Effect sizes and confidence intervals are much more useful than a p value or two.
      • dewarrn1 13y ago
        Cumming had a short treatment of the subject published in Psych Science recently: http://pss.sagepub.com/content/25/1/7.short http://pss.sagepub.com/content/25/1/7.short ; doi: 10.1177/0956797613504966.
• yread 13y ago
  I don't understand how did they get to the number 71% percent for 0.05 p-value? 0.05 p-value means that there is 5% probability that (for a t-test as example) a difference in averages of two sequences (the statistic) is by chance and not because of difference in means of their underlying normal distribution. I assume that the "toss-up" means that there is no difference in the means in reality (so the null hypothesis is true). Am I understanding it correctly? Shouldn't in this case the probability of getting p-value < 0.05 be in fact less than 5% and not 29%? How did they get the 29%?
  • a_bonobo 13y ago
    By Bayesian interpretation, for which they cite this paper (PDF): http://swfsc.noaa.gov/uploadedFiles/Divisions/PRD/Programs/ETP_Cetacean_Assessment/Of_P_Values_and_Bayes__A_Modest_Proposal.6.pdf http://swfsc.noaa.gov/uploadedFiles/Divisions/PRD/Programs/E... I'm not 100% sure how it works, and I've never seen it applied in biology, but this is medicine.
  • runarberg 13y ago
    No, you've got it wrong. A p-value of .05 means that given a true null-hypotheses, you have a 5% chance of getting this result anyway. That is, if there is no difference in the population mean, than you still have a 5% chance of finding a sample that contains this difference anyway. Do you see the difference. The p-value doesn't tell you anything about the population, it only gives you information about your sample -- a confusion that this article was pinpointing. So what the 29% is telling you is that, the chances of finding a statistically significant result (p = 0.05) given a population factor of µ = 0 are up to 29%. Whereas given a population factor of µ = 0 the chances of getting a significant result (p = 0.05) is 5%. EDIT: erased a wrong and confusing claim.
    • deleted 13y ago
      [deleted]
    • yread 13y ago
      Thanks, can you also explain why 29%? EDIT: also, what is population factor µ? The difference in means?
• analog31 13y ago
  "If your experiment needs statistics, you ought to have done a better experiment." -- Ernest Rutherford
• tokenadult 13y ago
  As the article reports, "Perhaps the worst fallacy is the kind of self-deception for which psychologist Uri Simonsohn of the University of Pennsylvania and his colleagues have popularized the term P-hacking; it is also known as data-dredging, snooping, fishing, significance-chasing and double-dipping. 'P-hacking,' says Simonsohn, 'is trying multiple things until you get the desired result' — even unconsciously." Simonsohn, has a whole website about "p-hacking" and how to detect it. http://www.p-curve.com/ http://www.p-curve.com/ He and his colleagues are concerned about making scientific papers more reliable. You can use the p-curve software on that site for your own investigations into p values found in published research. Many of the interesting issues brought up by the comments on the article kindly submitted here become much more clear after reading Simonsohn's various articles http://opim.wharton.upenn.edu/~uws/ http://opim.wharton.upenn.edu/~uws/ about p values and what they mean, and other aspects of interpreting published scientific research. He also has a paper http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2259879 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2259879 on evaluating replication results with more specific tips on that issue. "Abstract: "When does a replication attempt fail? The most common standard is: when it obtains p>.05. I begin here by evaluating this standard in the context of three published replication attempts, involving investigations of the embodiment of morality, the endowment effect, and weather effects on life satisfaction, concluding the standard has unacceptable problems. I then describe similarly unacceptable problems associated with standards that rely on effect-size comparisons between original and replication results. Finally, I propose a new standard: Replication attempts fail when their results indicate that the effect, if it exists at all, is too small to have been detected by the original study. This new standard (1) circumvents the problems associated with existing standards, (2) arrives at intuitively compelling interpretations of existing replication results, and (3) suggests a simple sample size requirement for replication attempts: 2.5 times the original sample."
• yetanotherphd 13y ago
  Most of the objections here and in the article are not inherent problems with frequentist p-values. First, the reported p-value might be wrong. E.g. basing it on assumptions of normality when the data is non-normal. However modern non-parametric approaches like the bootstrap can avoid this issue. Second, testing multiple hypotheses. If you test 10 hypotheses then you cannot reject the null (that all 10 null hypotheses hold) simply because one single hypothesis is rejected in isolation. But this is well known, and failing to account for it is an issue with the researcher, not with frequentist statistics. I actually think that the main practical difference between Bayesian and Frequentist statistics is whether accounting for the issue of multiple hypotheses is done formally or informally.
  • hootener 13y ago
    The article doesn't bash the p-value as a statistical test specifically, more its use and interpretation by scientists over the years. You're absolutely correct about using non-parametric tests, and more scientists should be using them. The normality assumption is flat out laughable when using real-world data most of the time. You're also correct about multiple hypothesis testing. Accounting for familywise error (e.g., Holms adjustments) can help to keep your p-value reporting honest. That doesn't negate the underlying problem, though. A p-value is simply an indication, nothing more. The p-value never promised to be more than that. The issue isn't in the p-value's construction, the issue lies in its misuse and how easily it can be abused in statistical reporting (see: p-hacking). The p-value as a test statistic is perfectly honest in my opinion. But like many other statistical methods, it comes with its own set of baggage that I feel gets conveniently glossed over more often than it should.
• milliams 13y ago
  This is why, to be on the safe side, in particle physics we have a requirement of a p-value of 0.0000003 for a discovery.
  • stinos 13y ago
    But isn't the whole point that no matter how low the P-value is, it is not a reliable measure?
  • crocowhile 13y ago
    (facepalm)
• mrcactu5 13y ago
  i routinely get jobs from doctors at prestigious universities who say, "here's at study with 3 samples, see if we can get p < 0.05"