11 ms·

Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

302 points by tokenadult 11y ago

chimeracoder 11y ago
For context, I have a degree in statistics, and I did research with Andrew Gelman (one of the statisticians quoted in the article). Glad to see this is gaining traction! I've been saying this for years: the world would actually be in a better place if we just abandoned p-values altogether. Hypothesis testing is taught in introductory statistics courses because the calculations involved are deceptively easy, whereas more sophisticated statistical techniques would be difficult without any background in linear algebra or calculus. Unfortunately, this enables researchers in all sorts of fields to make incredibly spurious statistical analyses that look convincing, because "all the calculations are right", even though they're using completely the wrong tool. Andrew Gelman, quoted in the article, feels very strongly that F-tests are always unnecessary[0]. I'd go as far as to extend that logic to the Student's t-test and any other related test as well. You can get into all sorts of confusing "paradoxes" with p-values. One of my favorites: Alice wants to figure out the average height of population. Her null hypothesis is 65 inches. She conducts a simple random sample, performs a t-test, and determines that the sample mean is 70 inches, with a p-value of .01. In an alternate universe, Bob does the same thing, with the same null hypothesis (65 inches). He determines that the sample mean is 90 inches, with a p-value of .000001. Some questions: A) Does Bob's experiment provide stronger evidence for rejecting the null hypothesis than Alice's does? B) In Bob's universe, is the true population mean higher than it is in Alice's universe? By pure hypothesis testing alone, the correct answer to both questions is "no", even though the intuitive answer to both questions is "yes"[1]. [0] http://andrewgelman.com/2009/05/18/noooooooooooooo/ http://andrewgelman.com/2009/05/18/noooooooooooooo/ [1] Part of the problem is that we do expect that, in Bob's universe, the true population mean is highly likely to be higher, and this is supported by the data. Trouble is, the reason we expect that is not formally related to hypothesis testing and t-tests/p-values.
- eanzenberg 11y ago
  If you're doing a measurement, why have a null hypothesis? Alice should sample the population at random, take the height measurements, calculate the average, plot the distribution, calculate the variance. If the distribution is not sufficiently smooth then continue to take measurements until it's smooth, or unchanging. Then Alice is done discovering all there is to know about the height distribution of the population she sampled. Same with Bob.
  - chimeracoder 11y ago
    > Alice should sample the population at random, take the height measurements, calculate the average, plot the distribution, calculate the variance A student-t test is basically a lossy encoding of that information you describe (the sample mean, the variance, and the sample size). > If the distribution is not sufficiently smooth then continue to take measurements until it's smooth, or unchanging. From a frequentist standpoint, this would be considered sloppy and bad methodology. You don't keep sampling until you get the results you want (or until the null hypothesis is invalidated). That said, as I mention in a comment above, the fact that you can keep sampling until the null hypothesis is invalidated (and that it is mathematically always guaranteed to happen eventually) is a big problem with the concept of hypothesis testing in the first place.
    - deleted 11y ago
      [deleted]
  - ryanmonroe 11y ago
    Creating an 100(1-a)% interval estimate of the population mean based on the sample's mean and variance is equivalent to examining the set of all values which produce a p-value below a%. Your view of the way Alice and Bob should be making inference about the true mean is actually exactly what hypothesis testing is. It's just that there is often a "status quo" belief about the parameter which has important theoretical implications. And academic papers are usually about that belief specifically, so it makes sense that they talk about it as a "hypothesis test". And that's the logical headline, the statement about the specific value. But standard deviations and distributions are often given, making it trivial to create the interval as a reader.
- btilly 11y ago
  For giggles and grins, my aunt and uncle are an actual example of one of the classic frequentist vs bayesian examples where frequentist statistics says that something utterly irrelevant should matter. Scenario 1 (true). Bill and Lorena had 7 children. 6 were boys, 1 was a girl. Are they biased towards having one gender? A 2-sided p-value says that there are 16 possibilities of this strength or more, each of which has probability 1/2^7, for a p-value of 2^4/2^7 = 1/2^3 = 0.125. We therefore fail to reject the null hypothesis at a p-value of 0.05. Scenario 2 (also true): Bill and Lorena decided to have children until they had a boy and a girl. They had 6 boys then a girl. Are they biased towards having one gender? An event this unlikely could only happen if they had 6 of the same gender in a row, which is 2 possibilities of probability 1/2^6, for a probability of 2/2^5 = 1/2^5 = 0.03125. We therefore reject the null hypothesis at a p-value of 0.05. Now the Bayesian gotcha. According to Bayes' theorem, the intent of Bill and Lorena can have absolutely NO impact on ANY calculation of posterior probabilities from prior expectations. There is no logical way in which this fact should matter at all. And yet it did!
  - thinkmoore 11y ago
    Is this actually true for the Bayesian model? Throw in a parameter for whether or not Bill and Lorena were trying to have children until they had a boy and a girl. Now your answer will depend quite heavily on your prior!
    - btilly 11y ago
      This is entirely true. The posterior odds are computed ENTIRELY from the odds of the observed events under the prior beliefs. There is NO WAY in which might-have-beens and didn't-happens enter in. Therefore the posterior probabilities cannot depend on the knowledge of what they would have done if something different had happened. Of course frequentist statistics are heavily affected by what would have happened if something different had happened.
      - thinkmoore 11y ago
        No, even as a Bayesian I could use either of two models to understand my data: In world one, I use bayesian update on the model P(# boys | Bias) = Mutltinomial(n = 7, p = Bias) In world two, I use bayesian update on the model P(# children | Bias) = Geometric(p = Bias) Might-have-beens and didn't happens do play in, in my choice of model. I should choose the one that I believe, and if I'm not certain, I should use an even more complicated model that incorporates my beliefs about what models might be appropriate.
- bunderbunder 11y ago
  This paradox is a great example of why I'm uncomfortable with the idea of coming up with a p-value when you're comparing a sample mean to a point value in the first place. If your null hypothesis amounts to little more than a scalar value that exists in a vacuum, you haven't really got a null hypothesis. If your null hypothesis is that your results will match what was seen in some other data set collected by some other person who may or may not have been using the same protocol as you, your null hypothesis describes a parallel universe and there's no way to draw an apples-to-apples comparison to it and your alternative hypothesis, which concerns data that did not come from that parallel universe. So I guess my answer to all questions would be, "Mu."
- tryitnow 11y ago
  Thanks! This is one of the best explanations of the problem with using p-values. Furthermore, I can recall many times when my stats professors would make it very clear what the "right" answer was, despite it being counter-intuitive but without explaining
- thinkmoore 11y ago
  If Alice just wants to know the average height of the population, why is she doing a hypothesis test that the height isn't 65 inches? Since her hypothesis test is designed to help answer the question of whether or not the mean population height is 65 inches, why should we expect it to tell us anything about the mean population height other than whether or not it being 65 is consistent with the data observed?
- statsaresimple 11y ago
  I can't understand what is the problem with p-values. You should never talk about the results, the essential point is that the method provides valid conclusions 95% and invalid conclusions 5% of the time. If your conclusion is wrong (you are in the 5% part) that is not a paradox. Also you should not try to prove things since your conclusion can be wrong, you should be glad to have a procedure that gives you useful information but is not infallible. By being humble you solve the problem.
- statsaresimple 11y ago
  I can't understand what is the problem with p-values. You should never talk about the results, the essential point is that the method provides valid conclusions 95% and invalid conclusions 5% of the time. If your conclusion is wrong (you are in the 5% part) that is not a paradox. Also you should not try to prove things since your conclusion can be wrong, you should be glad to have a procedure that gives you useful information but is not infallible. By being humble you solve the problem.
- kgwgk 11y ago
  Where is the "paradox"? I assume that you mean that by pure hypothesis testing both Alice and Bob reject the null hypothesis at the alpha=0.05 level, for example. The fact that the p-value is irrelevant is by design: you perform the test and either you reject the null hypothesis or you don't. But in fact most people won't do "pure" hypothesis testing and will conclude (somewhat incorrectly) that the evidence against the null hypothesis is indeed stronger in Bob's experiment. Edit: and why would be the answer to the second question "no"? The hypothesis testing procedure doesn't provide any point estimate at all so the question doesn't really mean anything in that setting.
carbocation 11y ago
Statistical significance is difficult to ensure. Certainly, one should be suspicious if 0.05 is ever used as a significance threshold, because it's unlikely that exactly one hypothesis under one regime was tested in any given paper. I am glad that the article's headline is clear that it's time to stop misusing P-values. Tests for statistical significance should still be used, and to abandon them would be foolish. In a sense, though, they are the beginning, not the end, of assessment.
daodedickinson 11y ago
Wow, I remember having these reservations about p-values when I took classes in stats but whenever I brought them up a prof. would wave their hands and be dismissive. They gave me a degree in political science, but I felt that political science was an oxymoron and it left me with no respect for the field.
- chimeracoder 11y ago
  > They gave me a degree in political science, but I felt that political science was an oxymoron and it left me with no respect for the field. For what it's worth, Andrew Gelman (quoted in the article) is one of the most pre-eminent Bayesian statisticians alive, and is a professor in both the department of Statistics and Political Science! "Political Science" need not be an oxymoron, even if a lot of self-professed political scientists use rather unscientific methods.
eanzenberg 11y ago
5% (1 in 20) is a pretty weak threshold to pass. Let's go 5 sigma (p < 3e-7) for discoveries and reserve 0.05 < p < 3e-7 for stuff we should take closer looks at.
- carbocation 11y ago
  > 5% (1 in 20) is a pretty weak threshold to pass. Let's go 5 sigma (p < 3e-7) for discoveries and reserve 0.05 < p < 3e-7 for stuff we should take closer looks at. This would still end up leading to misuse of P-values. Let's say you're doing a genome-wide association study on several hundred thousand SNPs. The traditional threshold is 5e-8 (0.05 / 1,000,000 effective tests). So using 3e-7 for the threshold for "discovery", you'd count many things as discovery that shouldn't be so. On the other hand, let's say you do a study with 20 people with cancer. You give 10 of them a drug, the other 10 a placebo. All 10 with the drug survive; all 10 with the placebo die. Your P value is 0.0002. This doesn't count as discovery, but clinically I know what my judgment is going to be. This is all to say that the misuse of P-values does not just come from the threshold.
  - deleted 11y ago
    [deleted]
  - eanzenberg 11y ago
    Multiple comparisons? 1 million independent tests? Hello? >Your P value is 0.0002. This doesn't count as discovery, but clinically I know what my judgment is going to be. >Let's go 5 sigma (p < 3e-7) for discoveries and reserve 0.05 < p < 3e-7 for stuff we should take closer looks at. ^Means you should start a new trial with n > 20 given the same placebo/drug split.
    - minimaxir 11y ago
      In the scientific world, it is economically infeasable (cost/time) for 1 million tests for a given hypothesis. Even n > 20 can be difficult for certain studies. Bootstrapping the results to simulate 1 million trials won't fix the aforementioned issue either.
btilly 11y ago
I agree as well! Here is what probability theory teaches us. The proper role of data is to adjust our prior beliefs about probabilities to posterior beliefs through Bayes' theorem. The challenge is how to best communicate this result to people who may have had a wide range of prior beliefs. p-values capture a degree of surprise in the result. Naively, a surprising result should catch our attention and cause us to rethink things. This is not a valid statistical procedure, but it IS how we naively think. And the substitution of a complex question for a simpler one is exactly how our brains are set up to handle complex questions about our environment. (I'm currently working through Thinking Fast and Slow which has a lot to say about this.) Simple Bayesian approaches take the opposite approach. You generally start with some relatively naive prior, and then treat the posterior as being the conclusion. Which is not very realistic if the real prior was something quite different. Both approaches have a fundamental mistake. The mistake is that we are taking a data set and asking what it TELLS us about the world. When in probability theory the real role of data is how to UPDATE our views about the world. This is why I have come to believe that for simple A/B testing, thinking about p-values is a mistake. The only three pieces of information that you need is how much data you are willing to collect, how much you have collected, and how big the performance difference is. Stop either when you have hit the maximum amount of data you're willing to throw at the test, or when the difference exceeds the square root of that maximum amount. This is about as good as any simple rule can do. If you try to be clever with p-values you will generally wind up putting yourself at saving yourself some effort in return for a small risk per test of making very bad mistakes. Accepting small risk per test over many tests for a long time puts you at high odds of eventually making a catastrophic mistake. This is a very bad tradeoff. I've personally seen a bad A/B test with a low p-value rolled out that produced a 15% loss in business for a company whose revenues were in the tens of millions annually. It. Was. Not. Pretty. (The problem was eventually found and fixed..a year later and after considerable turnover among the executive team.)
- danenania 11y ago
  Thanks for the very interesting thoughts. Could you explain a bit how your rule of thumb works and why it's better than p-values? Why is the difference vs. the square root of the max available sample size a meaningful measure?
eanzenberg 11y ago
p-value analysis has its big caveats like multiple comparisons, but Bayesian has its own, such as it's extremely hard to calculate priors. Both are challenging to use in difficult analysis and both can be abused.
- arcanus 11y ago
  > such as it's extremely hard to calculate priors. I think you mean it is difficult to formulate priors? Typically, calculating a prior only involves sampling from a distribution.
  - Fomite 11y ago
    Though technically, obtaining that distribution, if you're doing anything above "Eh, it's probably normal..." could be described as "hard".
  - eanzenberg 11y ago
    Let's use this example: http://betterexplained.com/articles/an-intuitive-and-short-explanation-of-bayes-theorem/ http://betterexplained.com/articles/an-intuitive-and-short-e... -1% of women have breast cancer (and therefore 99% do not). -80% of mammograms detect breast cancer when it is there (and therefore 20% miss it). -9.6% of mammograms detect breast cancer when it’s not there (and therefore 90.4% correctly return a negative result). For generating a prior, I posit it's really hard to determine that 1% of women have breast cancer, and this prior is almost linearly sensitive to whether you actually have cancer: prior | % likelihood of cancer 2% | 16% 1% | 7.8% 0.5% | 4% Why do I think it's hard to determine the prior? Imagine you are a 34 yrs old woman in SF. Do you use the rate of breast cancer for only 34 yr olds? For the range of 30-40 yr olds? 30-65 yr olds? For women only in SF? In the bay area? In California? Data from 2000-2016? 2010-2016? This becomes hard because positive cases are rare (thank god), so the population of which you calculate the prior from can be noisy. Imagine if the positive result occurs 1e-5 times.. you are talking about 10's of cases every million. Vary the prior probability and your posterior probability will vary just as much.
    - 11y ago
darawk 11y ago
Is the p-value really not the probability of your results being due to chance? Is that not a perfectly valid definition of it? I suppose 'chance' is a little hand-wavy, but isn't a p-value just the probability of your data given that your hypothesis is false? Isn't that literally and precisely the probability that they occurred by chance?
- chimeracoder 11y ago
  > I suppose 'chance' is a little hand-wavy, but isn't a p-value just the probability of your data given that your hypothesis is false? It is the probability of achieving a result at least as far from the hypothesized value exclusively due to random variation that is uncorrelated with the explanatory variable(s) at hand [and subject to a number of other assumptions].
  - darawk 11y ago
    Ya, but is that not equivalent to the statement 'due to chance'? I think the common understanding of 'chance' would include 'due to random variation uncorrelated with the explanatory variables'.
    - chimeracoder 11y ago
      > I think the common understanding of 'chance' would include 'due to random variation uncorrelated with the explanatory variables'. Yes, but the problem is that it also includes more than that. One of the many problems with p-values is that people assume that the p-value encodes a lot more information than it actually does. Another problem: it doesn't really tell you any "new" information. All equality-based null hypotheses are false, and we know this due to continuity theory[0]. Really, all a hypothesis test does is tell us if the sample size is large enough to reflect this knowledge. Literally any null hypothesis can be rejected, as long as the sample size is sufficiently large. It's worse than that, though. Yes, hypothesis testing doesn't tell us anything about the practical significance of the results. But it also doesn't actually tell us what most people thing it does: a measure of how wrong our hypothesis is. Rejection is a binary state: there's no concept of "strongly rejecting" a null hypothesis[1]. [0] We could reject continuity on the basis that the real world is actually discrete at the quantum level, but a lot of the math underpinning hypothesis testing falls apart if you don't assume continuity, so it's broken either way. [1] Many people - including statisticians - will sometimes imply this colloquially ("the p-value was .00001, so our hypothesis was completely wrong"). It's not always "wrong", because largely you can measure this concept in other ways, but you cannot do so with a p-value.
tokenadult 11y ago
The article submitted here leads to the American Statistical Association statement on the meaning of p values,[1] the first such methodological statement ever formally issued by the association. It's free to read and download. The statement summarizes into these main points, with further explanation in the text of the statement. "What is a p-value? "Informally, a p-value is the probability under a specified statistical model that a statistical summary of the data (for example, the sample mean difference between two compared groups) would be equal to or more extreme than its observed value. "Principles "1. P-values can indicate how incompatible the data are with a specified statistical model. "2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone. "3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold. "4. Proper inference requires full reporting and transparency. "5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result. "6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis." [1] "The ASA's statement on p-values: context, process, and purpose" http://amstat.tandfonline.com/doi/abs/10.1080/00031305.2016.1154108 http://amstat.tandfonline.com/doi/abs/10.1080/00031305.2016....
- thearn4 11y ago
  I remember the feeling after my first undergraduate course in statistics essentially being that we stated these principles, then spent the remaining weeks essentially invalidating them without offering any real alternatives. My professor may have been more careful than I remember, but the subtleness was lost on me at the time if that was the case. The testing of statistical hypotheses always seemed like an odd area of the mathematical sciences to me, even after later taking a graduate mathematical statistics sequence. Like an academic squabble between giants in the field of frequentist inference (Fisher vs. Neyman and Pearson) that ended suddenly without resolution, with the scientific community decided to sloppily merge the two positions for the purposes of publication and forge onward.
- wdewind 11y ago
  > "3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold. There is no lower threshold at which the data becomes non-predictive?
amluto 11y ago
If they held the same meeting 20 times, would they reach the same conclusion in 19 of those meetings? On a more serious note, I think that the use of the word "significant" to mean "the effect is reasonably likely to exist by some standard" should be abolished. Webster's 1913 dictionary says: > Deserving to be considered; important; momentous; as, a significant event. Statisticians don't use "significant" to mean important at all -- they use it to mean "I could detect it". This is bad when someone publishes a paper saying "I found that some drug significantly reduces such-and-such" -- this could just mean that they did a HUGE study and found that the drug reliably had some completely unimportant effect. It's much worse when it's negated, though. Think about all the headlines that say that some treatment "did not have a significant effect". This is basically meaningless. I could do a study finding that exercise has no significant effect on fitness, for example, by making the study small enough. A good friend of mine suggested that statisticians replace "significant" with "discernible". So next time someone does a small study, they might find that "eating fat had no discernible effect on weight gain", and perhaps readers would then ask the obvious question, which is "how hard did you look?". This would also help people doing very good research make less wishy-washy conclusions. For example, suppose that "vaccines have no discernible effect on autism rates". This is probably true in a number of studies, but it's the wrong analysis. If researchers who did these studies had to state the conclusions in a silly manner like that, maybe they'd find a more useful analysis to do. Hint: doing big studies just so you can fail to find an effect is nonsensical. Instead, do big studies so you can put a tight upper bound on the effect. Don't tell me that vaccines don't have a significant (or discernible) effect on autism. Tell me that, with 99.9% confidence, you have ruled out the possibility that vaccines have caused more than ten autism cases in the entire history of vaccines, and that, most likely, they've caused no cases whatsoever (or whatever the right numbers are). Edit: fixed an insignificant typo.
- evolsb 11y ago
  There is a significant (pun) difference between statistically significant and economically significant. The conclusion in the drug paper conflates the two. In finance, we could find plenty of statistically significant results (e.g. small cap stocks outperform large cap stocks on Fridays, with a small p-value if you like), but most results were not economically significant--they were not usable for a trading system because they were too small to overcome real-world costs. In short, they weren't meaningful in a real world sense, even though the result was detectable statistically.
FrankyHollywood 11y ago
Next step is explaining social science students the meaning of 'randomness' :) Really, the amazing amount of bullshit social studies I have seen 'proven' by statistics. Amazing new insights like 'if children wear green shirts, while the teacher has a blue shirt the cognitive attention span is 12.3% higher than children wearing purple shirts. The effect was measured with a significance of bla bla bla.' Software like SPSS facilitates this even more. People with no notion of random effects or probablilty theory click on the 'proof my research' button and even get it published. So a lot more work in this area!
johan_larson 11y ago
It's not just p-values. Some people just don't understand even very basic statistics. I remember talking to one person in marketing who ran surveys of the company's users. They would send out a survey to all registered users, get back responses from 1% of them or something, and then proceed to report findings based on the responses. They were really happy, since a 1% response rate is great for surveys like this. I tried to explain to them that all of this statistical machinery relies on having a random sample, and a self-selected sample is not that. No effect whatsoever. Surveys like this are standard practice in the industry. Why are you making trouble, geek-boy?
- wonderlust 11y ago
  Had an indergrad ecology course to teach me this. The first 4 labs were plug and chug equations sampling a square with computer generated randomly placed dots. All we had to do is get a sample (we could choose the size), and figure out a 95% likelihood that the correct number was within our range. No one could do it for 3 labs, before he told us how. Good samples are REALLY important, even under basically perfect conditions.
- roel_v 11y ago
  " Surveys like this are standard practice in the industry." So, how does one exploit this apparent bad practice? I.e., how does one make a profit from others making this mistake? The answer is, of course, one doesn't - otherwise others would've done so. So what does this tell us about the state of affairs? Is it that bad sampling doesn't matter for practical purposes, or that marketing research is useless? I don't know, and I'm not trying to be belligerent here. A situation similar to this shows up in dozens of places every day - more often than not, it doesn't matter if things are done 'right', there are large margins within which 'good enough' is indistinguishable from 'right'. This bothers me greatly, but it's hard to argue against this conclusion, empirically. How to deal with this cognitive dissonance? I mean this is the exact topic at least half of the blog posts that make it to the HN front page deal are fundamentally about.
rcthompson 11y ago
In undergrad, I learned how to about p-values but never quite understood how they were actually useful. Now, as a bioinformatics graduate student, I've come to understand that my original instinct was right all along.
mikeskim 11y ago
If people would just learn to resample (cross validate, use subsampling or the bootstrap), we wouldn't be having this pointless discussion at all.
- vasilipupkin 11y ago
  In theory, you are correct, but some data sets are too small or make it too difficult to cross validate in a way, that is meaningful to the original problem
Houshalter 11y ago
P-values are so weird. Studies should instead report a likelihood ratio. A likelihood ratio is mathematically correct, and tells you exactly how much to update a hypothesis. You can convert p-values to likelihood ratios, and they are quite similar. But its not perfect. A p value of 0.05 becomes 100:5, or 20:1. Which means it increases the odds of a hypothesis by 20. So a probability of 1% updates to 17%, which is still quite small. But that assumes that the hypothesis has a 100% chance of producing the same or greater result, which is unlikely. Instead it might only be 50% which is half as much evidence. In the extreme case, it could be only 5% likely to produce the result, which means the likelihood ratio is 5:5 and is literally no evidence, but still has a p value of 0.05. Anyway likelihood ratios accumulate exponentially, since they multiply together. As long as there is no publication bias, you can take a few weak studies and produce a single very strong likelihood update.
altrego99 11y ago
The article tends to imply P-value should not be used at all, rather than misused. P-value definitely means something. For example, if the p-value is 1e-10 (which is often possible), you know for sure that the hypothesis generating has been disproved. So let me rephrase the title of the article - "It's time to use P-Value correctly."
purpled_haze 11y ago
...and I could not find the word "correlation" anywhere in this article discussion statistics, harm, misunderstanding, and studies. Shame.
ianamartin 11y ago
This is one of the best conversation threads I've ever seen on HN. It's both polite and informative. I want to toss in my own thoughts here. Since I've spent the vast majority of my tech career in the Market Research industry (hello, bias!), I'm tempted to say that one of the most frequent intersections between statistical science and business decisions happens in that world. Product testing, shopper marketing, A/B testing . . . these are pretty common fare these days. But I feel like the MR people are sort of their own worst enemy in many cases. It's a fairly recent development that MR people are even allowed a seat at the table for major product or business decisions. And when the data nerds show up at the meeting, we have to make human communication decisions that are difficult. I can't show up at the C-suite and lecture company executives about the finer points of statistical philosophy. When I'm presenting findings to stake-holders, it's my job to abstract the details and present something that makes a coherent case for a decision, based on the data we have available. It is sinfully attractive to go tell your boss's boss's boss that we have a threshold--a number we can point to. If this number turns out to be smaller than .05, this project is a go. Three months later, you go back to that boss and tell him the number came back and it was .0499999. The boss says, "Okay, go!" And then you are all, "Wait, wait, wait. Hang on a second. Let's talk about this." My god, what have I done? The practical reality of the intersection of statistics and business is a harsh one. We have to do better. In terms of leaky abstractions, the communication of data science to business decision makers is quite possibly the leaky-est of all. Why is it so leaky? I have two points about this. 1) Statistics is one of the most existentially depressing fields of study. There is no acceptance; there is no love; there is nothing positive about it. Ever. Statistics is always about rejection and failure. We never accept or affirm a hypothesis. We only ever reject the null hypothesis or we fail to reject it. That's it. 2) In business, we tend to be very very sloppy about formulating our hypotheses. Sometimes we don't even really think about them at all. Take a common case for market research. New product testing. We do a rep sample with a decent size (say, 1800 potential product buyers) and we randomly show five different products, one of which is the product the person already owns/uses (because that's called control /s). The other 4 products are variations on a theme with different attributes. What's the null hypothesis here? Does it ever get discussed? What's the alternative hypothesis? The implicit and never-talked-about null is that all things being equal, there is no difference between the distribution of purchase likelihood among all products. The alternative is that there is a real difference on a scale of likely to purchase. The implicit and intuitive assumption is that there is something about that feature set that drives the difference. (I'm looking at you, Max Diff) But that's not real. It's not a part of the test. The only test you can do in that situation is to check if those aggregate distributions are different from each other. The real null is that they are the same, and the alternative is that they are different. All you can do with statistics is tell if two distributions are isomorphic. Now, who wants to try to explain any of that to your CEO? No one does. Your CEO doesn't want it, you don't want it, your girlfriend doesn't want it. No one wants it. So we try to abstract, and I feel like we mostly fail at doing a good job of that. This is getting really long, and I don't want to rant. So to finish up, an idea for more effective uses of data science as it interacts with the business world: I agree, let's stop talking about p values. Let's work harder and funnel the results of those MR studies into practical models of the business' future. Let's take the research and pipe it into Bayesian expected value models. Let's stop showing stacked bar charts to execs and expecting them to make good decisions based on weak evidence we got from hypotheses we didn't really think about in the first place. Some of this might come across as a rant. I hope it is not taken that way. This is a real problem that I've been thinking about for a long time. And I don't mean to step on anyone's toes. I have certainly committed many of the data sins that I'm deriding above. Edited to add: The real workings of statistics are unintuitive. I'm not saying that they are wrong. But in working with people for years now, I understand the confusion. It's a psychological problem. Hypotheses are either not really well though out or not considered in an organized way, in my experience. A hypothesis is not concrete in many practical cases. It's a thought. An idea, perhaps. It's often a thing that floats around in your mind, or maybe you paid some lip service and tossed it into your note-taking app. Data seem much more real. You download a few gigabytes of data and start working on it. It's quite easy to get confused. I have real data! This is tangible stuff. Thinking of things properly and evaluating the probability of your data given the hypothesis is hard. Your data seems much more concrete. These are real people answering real questions about X. Even for people who are really hell-bent on statistical rigor, this is a challenge.
benjaminmhaley 11y ago
The #1 problem is with p-values is the word "significant". We should use "detectable" instead. Significant implies meaningful to most people, but not in a statistical context. This is quite confusing. Detectable is better because the mainstream meaning aligns with the jargon. So: > "Discovering statistically significant biclusters in gene expression data" becomes: > "Discovering statistically detectable biclusters in gene expression data" This rephrasing makes it evident that "statistically detectable" adds little to the title. So the title becomes > "Discovering biclusters in gene expression data" A better title.
giardini 11y ago
This excellent and amusing article by Gerd Gigerenzer discusses the history of p-values and their (mis)use: "Mindless Statistics" http://library.mpib-berlin.mpg.de/ft/gg/GG_Mindless_2004.pdf http://library.mpib-berlin.mpg.de/ft/gg/GG_Mindless_2004.pdf or http://www.unh.edu/halelab/BIOL933/papers/2004_Gigerenzer_JSE.pdf http://www.unh.edu/halelab/BIOL933/papers/2004_Gigerenzer_JS...

302 points by tokenadult 11y ago

For context, I have a degree in statistics, and I did research with Andrew Gelman (one of the statisticians quoted in the article). Glad to see this is gaining traction! I've been saying this for years: the world would actually be in a better place if we just abandoned p-values altogether. Hypothesis testing is taught in introductory statistics courses because the calculations involved are deceptively easy, whereas more sophisticated statistical techniques would be difficult without any background in linear algebra or calculus. Unfortunately, this enables researchers in all sorts of fields to make incredibly spurious statistical analyses that look convincing, because "all the calculations are right", even though they're using completely the wrong tool. Andrew Gelman, quoted in the article, feels very strongly that F-tests are always unnecessary[0]. I'd go as far as to extend that logic to the Student's t-test and any other related test as well. You can get into all sorts of confusing "paradoxes" with p-values. One of my favorites: Alice wants to figure out the average height of population. Her null hypothesis is 65 inches. She conducts a simple random sample, performs a t-test, and determines that the sample mean is 70 inches, with a p-value of .01. In an alternate universe, Bob does the same thing, with the same null hypothesis (65 inches). He determines that the sample mean is 90 inches, with a p-value of .000001. Some questions: A) Does Bob's experiment provide stronger evidence for rejecting the null hypothesis than Alice's does? B) In Bob's universe, is the true population mean higher than it is in Alice's universe? By pure hypothesis testing alone, the correct answer to both questions is "no", even though the intuitive answer to both questions is "yes"[1]. [0] http://andrewgelman.com/2009/05/18/noooooooooooooo/ http://andrewgelman.com/2009/05/18/noooooooooooooo/ [1] Part of the problem is that we do expect that, in Bob's universe, the true population mean is highly likely to be higher, and this is supported by the data. Trouble is, the reason we expect that is not formally related to hypothesis testing and t-tests/p-values.

eanzenberg 11y ago
If you're doing a measurement, why have a null hypothesis? Alice should sample the population at random, take the height measurements, calculate the average, plot the distribution, calculate the variance. If the distribution is not sufficiently smooth then continue to take measurements until it's smooth, or unchanging. Then Alice is done discovering all there is to know about the height distribution of the population she sampled. Same with Bob.
- chimeracoder 11y ago
  > Alice should sample the population at random, take the height measurements, calculate the average, plot the distribution, calculate the variance A student-t test is basically a lossy encoding of that information you describe (the sample mean, the variance, and the sample size). > If the distribution is not sufficiently smooth then continue to take measurements until it's smooth, or unchanging. From a frequentist standpoint, this would be considered sloppy and bad methodology. You don't keep sampling until you get the results you want (or until the null hypothesis is invalidated). That said, as I mention in a comment above, the fact that you can keep sampling until the null hypothesis is invalidated (and that it is mathematically always guaranteed to happen eventually) is a big problem with the concept of hypothesis testing in the first place.
  - deleted 11y ago
    [deleted]
- ryanmonroe 11y ago
  Creating an 100(1-a)% interval estimate of the population mean based on the sample's mean and variance is equivalent to examining the set of all values which produce a p-value below a%. Your view of the way Alice and Bob should be making inference about the true mean is actually exactly what hypothesis testing is. It's just that there is often a "status quo" belief about the parameter which has important theoretical implications. And academic papers are usually about that belief specifically, so it makes sense that they talk about it as a "hypothesis test". And that's the logical headline, the statement about the specific value. But standard deviations and distributions are often given, making it trivial to create the interval as a reader.
btilly 11y ago
For giggles and grins, my aunt and uncle are an actual example of one of the classic frequentist vs bayesian examples where frequentist statistics says that something utterly irrelevant should matter. Scenario 1 (true). Bill and Lorena had 7 children. 6 were boys, 1 was a girl. Are they biased towards having one gender? A 2-sided p-value says that there are 16 possibilities of this strength or more, each of which has probability 1/2^7, for a p-value of 2^4/2^7 = 1/2^3 = 0.125. We therefore fail to reject the null hypothesis at a p-value of 0.05. Scenario 2 (also true): Bill and Lorena decided to have children until they had a boy and a girl. They had 6 boys then a girl. Are they biased towards having one gender? An event this unlikely could only happen if they had 6 of the same gender in a row, which is 2 possibilities of probability 1/2^6, for a probability of 2/2^5 = 1/2^5 = 0.03125. We therefore reject the null hypothesis at a p-value of 0.05. Now the Bayesian gotcha. According to Bayes' theorem, the intent of Bill and Lorena can have absolutely NO impact on ANY calculation of posterior probabilities from prior expectations. There is no logical way in which this fact should matter at all. And yet it did!
- thinkmoore 11y ago
  Is this actually true for the Bayesian model? Throw in a parameter for whether or not Bill and Lorena were trying to have children until they had a boy and a girl. Now your answer will depend quite heavily on your prior!
  - btilly 11y ago
    This is entirely true. The posterior odds are computed ENTIRELY from the odds of the observed events under the prior beliefs. There is NO WAY in which might-have-beens and didn't-happens enter in. Therefore the posterior probabilities cannot depend on the knowledge of what they would have done if something different had happened. Of course frequentist statistics are heavily affected by what would have happened if something different had happened.
    - thinkmoore 11y ago
      No, even as a Bayesian I could use either of two models to understand my data: In world one, I use bayesian update on the model P(# boys | Bias) = Mutltinomial(n = 7, p = Bias) In world two, I use bayesian update on the model P(# children | Bias) = Geometric(p = Bias) Might-have-beens and didn't happens do play in, in my choice of model. I should choose the one that I believe, and if I'm not certain, I should use an even more complicated model that incorporates my beliefs about what models might be appropriate.
bunderbunder 11y ago
This paradox is a great example of why I'm uncomfortable with the idea of coming up with a p-value when you're comparing a sample mean to a point value in the first place. If your null hypothesis amounts to little more than a scalar value that exists in a vacuum, you haven't really got a null hypothesis. If your null hypothesis is that your results will match what was seen in some other data set collected by some other person who may or may not have been using the same protocol as you, your null hypothesis describes a parallel universe and there's no way to draw an apples-to-apples comparison to it and your alternative hypothesis, which concerns data that did not come from that parallel universe. So I guess my answer to all questions would be, "Mu."
tryitnow 11y ago
Thanks! This is one of the best explanations of the problem with using p-values. Furthermore, I can recall many times when my stats professors would make it very clear what the "right" answer was, despite it being counter-intuitive but without explaining
thinkmoore 11y ago
If Alice just wants to know the average height of the population, why is she doing a hypothesis test that the height isn't 65 inches? Since her hypothesis test is designed to help answer the question of whether or not the mean population height is 65 inches, why should we expect it to tell us anything about the mean population height other than whether or not it being 65 is consistent with the data observed?
statsaresimple 11y ago
I can't understand what is the problem with p-values. You should never talk about the results, the essential point is that the method provides valid conclusions 95% and invalid conclusions 5% of the time. If your conclusion is wrong (you are in the 5% part) that is not a paradox. Also you should not try to prove things since your conclusion can be wrong, you should be glad to have a procedure that gives you useful information but is not infallible. By being humble you solve the problem.
statsaresimple 11y ago
I can't understand what is the problem with p-values. You should never talk about the results, the essential point is that the method provides valid conclusions 95% and invalid conclusions 5% of the time. If your conclusion is wrong (you are in the 5% part) that is not a paradox. Also you should not try to prove things since your conclusion can be wrong, you should be glad to have a procedure that gives you useful information but is not infallible. By being humble you solve the problem.
kgwgk 11y ago
Where is the "paradox"? I assume that you mean that by pure hypothesis testing both Alice and Bob reject the null hypothesis at the alpha=0.05 level, for example. The fact that the p-value is irrelevant is by design: you perform the test and either you reject the null hypothesis or you don't. But in fact most people won't do "pure" hypothesis testing and will conclude (somewhat incorrectly) that the evidence against the null hypothesis is indeed stronger in Bob's experiment. Edit: and why would be the answer to the second question "no"? The hypothesis testing procedure doesn't provide any point estimate at all so the question doesn't really mean anything in that setting.

carbocation 11y ago

Statistical significance is difficult to ensure. Certainly, one should be suspicious if 0.05 is ever used as a significance threshold, because it's unlikely that exactly one hypothesis under one regime was tested in any given paper. I am glad that the article's headline is clear that it's time to stop misusing P-values. Tests for statistical significance should still be used, and to abandon them would be foolish. In a sense, though, they are the beginning, not the end, of assessment.

daodedickinson 11y ago

Wow, I remember having these reservations about p-values when I took classes in stats but whenever I brought them up a prof. would wave their hands and be dismissive. They gave me a degree in political science, but I felt that political science was an oxymoron and it left me with no respect for the field.

chimeracoder 11y ago
> They gave me a degree in political science, but I felt that political science was an oxymoron and it left me with no respect for the field. For what it's worth, Andrew Gelman (quoted in the article) is one of the most pre-eminent Bayesian statisticians alive, and is a professor in both the department of Statistics and Political Science! "Political Science" need not be an oxymoron, even if a lot of self-professed political scientists use rather unscientific methods.

eanzenberg 11y ago

5% (1 in 20) is a pretty weak threshold to pass. Let's go 5 sigma (p < 3e-7) for discoveries and reserve 0.05 < p < 3e-7 for stuff we should take closer looks at.

carbocation 11y ago
> 5% (1 in 20) is a pretty weak threshold to pass. Let's go 5 sigma (p < 3e-7) for discoveries and reserve 0.05 < p < 3e-7 for stuff we should take closer looks at. This would still end up leading to misuse of P-values. Let's say you're doing a genome-wide association study on several hundred thousand SNPs. The traditional threshold is 5e-8 (0.05 / 1,000,000 effective tests). So using 3e-7 for the threshold for "discovery", you'd count many things as discovery that shouldn't be so. On the other hand, let's say you do a study with 20 people with cancer. You give 10 of them a drug, the other 10 a placebo. All 10 with the drug survive; all 10 with the placebo die. Your P value is 0.0002. This doesn't count as discovery, but clinically I know what my judgment is going to be. This is all to say that the misuse of P-values does not just come from the threshold.
- deleted 11y ago
  [deleted]
- eanzenberg 11y ago
  Multiple comparisons? 1 million independent tests? Hello? >Your P value is 0.0002. This doesn't count as discovery, but clinically I know what my judgment is going to be. >Let's go 5 sigma (p < 3e-7) for discoveries and reserve 0.05 < p < 3e-7 for stuff we should take closer looks at. ^Means you should start a new trial with n > 20 given the same placebo/drug split.
  - minimaxir 11y ago
    In the scientific world, it is economically infeasable (cost/time) for 1 million tests for a given hypothesis. Even n > 20 can be difficult for certain studies. Bootstrapping the results to simulate 1 million trials won't fix the aforementioned issue either.

btilly 11y ago

I agree as well! Here is what probability theory teaches us. The proper role of data is to adjust our prior beliefs about probabilities to posterior beliefs through Bayes' theorem. The challenge is how to best communicate this result to people who may have had a wide range of prior beliefs. p-values capture a degree of surprise in the result. Naively, a surprising result should catch our attention and cause us to rethink things. This is not a valid statistical procedure, but it IS how we naively think. And the substitution of a complex question for a simpler one is exactly how our brains are set up to handle complex questions about our environment. (I'm currently working through Thinking Fast and Slow which has a lot to say about this.) Simple Bayesian approaches take the opposite approach. You generally start with some relatively naive prior, and then treat the posterior as being the conclusion. Which is not very realistic if the real prior was something quite different. Both approaches have a fundamental mistake. The mistake is that we are taking a data set and asking what it TELLS us about the world. When in probability theory the real role of data is how to UPDATE our views about the world. This is why I have come to believe that for simple A/B testing, thinking about p-values is a mistake. The only three pieces of information that you need is how much data you are willing to collect, how much you have collected, and how big the performance difference is. Stop either when you have hit the maximum amount of data you're willing to throw at the test, or when the difference exceeds the square root of that maximum amount. This is about as good as any simple rule can do. If you try to be clever with p-values you will generally wind up putting yourself at saving yourself some effort in return for a small risk per test of making very bad mistakes. Accepting small risk per test over many tests for a long time puts you at high odds of eventually making a catastrophic mistake. This is a very bad tradeoff. I've personally seen a bad A/B test with a low p-value rolled out that produced a 15% loss in business for a company whose revenues were in the tens of millions annually. It. Was. Not. Pretty. (The problem was eventually found and fixed..a year later and after considerable turnover among the executive team.)

danenania 11y ago
Thanks for the very interesting thoughts. Could you explain a bit how your rule of thumb works and why it's better than p-values? Why is the difference vs. the square root of the max available sample size a meaningful measure?

eanzenberg 11y ago

p-value analysis has its big caveats like multiple comparisons, but Bayesian has its own, such as it's extremely hard to calculate priors. Both are challenging to use in difficult analysis and both can be abused.

arcanus 11y ago
> such as it's extremely hard to calculate priors. I think you mean it is difficult to formulate priors? Typically, calculating a prior only involves sampling from a distribution.
- Fomite 11y ago
  Though technically, obtaining that distribution, if you're doing anything above "Eh, it's probably normal..." could be described as "hard".
- eanzenberg 11y ago
  Let's use this example: http://betterexplained.com/articles/an-intuitive-and-short-explanation-of-bayes-theorem/ http://betterexplained.com/articles/an-intuitive-and-short-e... -1% of women have breast cancer (and therefore 99% do not). -80% of mammograms detect breast cancer when it is there (and therefore 20% miss it). -9.6% of mammograms detect breast cancer when it’s not there (and therefore 90.4% correctly return a negative result). For generating a prior, I posit it's really hard to determine that 1% of women have breast cancer, and this prior is almost linearly sensitive to whether you actually have cancer: prior | % likelihood of cancer 2% | 16% 1% | 7.8% 0.5% | 4% Why do I think it's hard to determine the prior? Imagine you are a 34 yrs old woman in SF. Do you use the rate of breast cancer for only 34 yr olds? For the range of 30-40 yr olds? 30-65 yr olds? For women only in SF? In the bay area? In California? Data from 2000-2016? 2010-2016? This becomes hard because positive cases are rare (thank god), so the population of which you calculate the prior from can be noisy. Imagine if the positive result occurs 1e-5 times.. you are talking about 10's of cases every million. Vary the prior probability and your posterior probability will vary just as much.
  - 11y ago

darawk 11y ago

Is the p-value really not the probability of your results being due to chance? Is that not a perfectly valid definition of it? I suppose 'chance' is a little hand-wavy, but isn't a p-value just the probability of your data given that your hypothesis is false? Isn't that literally and precisely the probability that they occurred by chance?

chimeracoder 11y ago
> I suppose 'chance' is a little hand-wavy, but isn't a p-value just the probability of your data given that your hypothesis is false? It is the probability of achieving a result at least as far from the hypothesized value exclusively due to random variation that is uncorrelated with the explanatory variable(s) at hand [and subject to a number of other assumptions].
- darawk 11y ago
  Ya, but is that not equivalent to the statement 'due to chance'? I think the common understanding of 'chance' would include 'due to random variation uncorrelated with the explanatory variables'.
  - chimeracoder 11y ago
    > I think the common understanding of 'chance' would include 'due to random variation uncorrelated with the explanatory variables'. Yes, but the problem is that it also includes more than that. One of the many problems with p-values is that people assume that the p-value encodes a lot more information than it actually does. Another problem: it doesn't really tell you any "new" information. All equality-based null hypotheses are false, and we know this due to continuity theory[0]. Really, all a hypothesis test does is tell us if the sample size is large enough to reflect this knowledge. Literally any null hypothesis can be rejected, as long as the sample size is sufficiently large. It's worse than that, though. Yes, hypothesis testing doesn't tell us anything about the practical significance of the results. But it also doesn't actually tell us what most people thing it does: a measure of how wrong our hypothesis is. Rejection is a binary state: there's no concept of "strongly rejecting" a null hypothesis[1]. [0] We could reject continuity on the basis that the real world is actually discrete at the quantum level, but a lot of the math underpinning hypothesis testing falls apart if you don't assume continuity, so it's broken either way. [1] Many people - including statisticians - will sometimes imply this colloquially ("the p-value was .00001, so our hypothesis was completely wrong"). It's not always "wrong", because largely you can measure this concept in other ways, but you cannot do so with a p-value.

tokenadult 11y ago

The article submitted here leads to the American Statistical Association statement on the meaning of p values,[1] the first such methodological statement ever formally issued by the association. It's free to read and download. The statement summarizes into these main points, with further explanation in the text of the statement. "What is a p-value? "Informally, a p-value is the probability under a specified statistical model that a statistical summary of the data (for example, the sample mean difference between two compared groups) would be equal to or more extreme than its observed value. "Principles "1. P-values can indicate how incompatible the data are with a specified statistical model. "2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone. "3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold. "4. Proper inference requires full reporting and transparency. "5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result. "6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis." [1] "The ASA's statement on p-values: context, process, and purpose" http://amstat.tandfonline.com/doi/abs/10.1080/00031305.2016.1154108 http://amstat.tandfonline.com/doi/abs/10.1080/00031305.2016....

thearn4 11y ago
I remember the feeling after my first undergraduate course in statistics essentially being that we stated these principles, then spent the remaining weeks essentially invalidating them without offering any real alternatives. My professor may have been more careful than I remember, but the subtleness was lost on me at the time if that was the case. The testing of statistical hypotheses always seemed like an odd area of the mathematical sciences to me, even after later taking a graduate mathematical statistics sequence. Like an academic squabble between giants in the field of frequentist inference (Fisher vs. Neyman and Pearson) that ended suddenly without resolution, with the scientific community decided to sloppily merge the two positions for the purposes of publication and forge onward.
wdewind 11y ago
> "3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold. There is no lower threshold at which the data becomes non-predictive?

amluto 11y ago

If they held the same meeting 20 times, would they reach the same conclusion in 19 of those meetings? On a more serious note, I think that the use of the word "significant" to mean "the effect is reasonably likely to exist by some standard" should be abolished. Webster's 1913 dictionary says: > Deserving to be considered; important; momentous; as, a significant event. Statisticians don't use "significant" to mean important at all -- they use it to mean "I could detect it". This is bad when someone publishes a paper saying "I found that some drug significantly reduces such-and-such" -- this could just mean that they did a HUGE study and found that the drug reliably had some completely unimportant effect. It's much worse when it's negated, though. Think about all the headlines that say that some treatment "did not have a significant effect". This is basically meaningless. I could do a study finding that exercise has no significant effect on fitness, for example, by making the study small enough. A good friend of mine suggested that statisticians replace "significant" with "discernible". So next time someone does a small study, they might find that "eating fat had no discernible effect on weight gain", and perhaps readers would then ask the obvious question, which is "how hard did you look?". This would also help people doing very good research make less wishy-washy conclusions. For example, suppose that "vaccines have no discernible effect on autism rates". This is probably true in a number of studies, but it's the wrong analysis. If researchers who did these studies had to state the conclusions in a silly manner like that, maybe they'd find a more useful analysis to do. Hint: doing big studies just so you can fail to find an effect is nonsensical. Instead, do big studies so you can put a tight upper bound on the effect. Don't tell me that vaccines don't have a significant (or discernible) effect on autism. Tell me that, with 99.9% confidence, you have ruled out the possibility that vaccines have caused more than ten autism cases in the entire history of vaccines, and that, most likely, they've caused no cases whatsoever (or whatever the right numbers are). Edit: fixed an insignificant typo.

evolsb 11y ago
There is a significant (pun) difference between statistically significant and economically significant. The conclusion in the drug paper conflates the two. In finance, we could find plenty of statistically significant results (e.g. small cap stocks outperform large cap stocks on Fridays, with a small p-value if you like), but most results were not economically significant--they were not usable for a trading system because they were too small to overcome real-world costs. In short, they weren't meaningful in a real world sense, even though the result was detectable statistically.

FrankyHollywood 11y ago

Next step is explaining social science students the meaning of 'randomness' :) Really, the amazing amount of bullshit social studies I have seen 'proven' by statistics. Amazing new insights like 'if children wear green shirts, while the teacher has a blue shirt the cognitive attention span is 12.3% higher than children wearing purple shirts. The effect was measured with a significance of bla bla bla.' Software like SPSS facilitates this even more. People with no notion of random effects or probablilty theory click on the 'proof my research' button and even get it published. So a lot more work in this area!

johan_larson 11y ago

It's not just p-values. Some people just don't understand even very basic statistics. I remember talking to one person in marketing who ran surveys of the company's users. They would send out a survey to all registered users, get back responses from 1% of them or something, and then proceed to report findings based on the responses. They were really happy, since a 1% response rate is great for surveys like this. I tried to explain to them that all of this statistical machinery relies on having a random sample, and a self-selected sample is not that. No effect whatsoever. Surveys like this are standard practice in the industry. Why are you making trouble, geek-boy?

wonderlust 11y ago
Had an indergrad ecology course to teach me this. The first 4 labs were plug and chug equations sampling a square with computer generated randomly placed dots. All we had to do is get a sample (we could choose the size), and figure out a 95% likelihood that the correct number was within our range. No one could do it for 3 labs, before he told us how. Good samples are REALLY important, even under basically perfect conditions.
roel_v 11y ago
" Surveys like this are standard practice in the industry." So, how does one exploit this apparent bad practice? I.e., how does one make a profit from others making this mistake? The answer is, of course, one doesn't - otherwise others would've done so. So what does this tell us about the state of affairs? Is it that bad sampling doesn't matter for practical purposes, or that marketing research is useless? I don't know, and I'm not trying to be belligerent here. A situation similar to this shows up in dozens of places every day - more often than not, it doesn't matter if things are done 'right', there are large margins within which 'good enough' is indistinguishable from 'right'. This bothers me greatly, but it's hard to argue against this conclusion, empirically. How to deal with this cognitive dissonance? I mean this is the exact topic at least half of the blog posts that make it to the HN front page deal are fundamentally about.

rcthompson 11y ago

In undergrad, I learned how to about p-values but never quite understood how they were actually useful. Now, as a bioinformatics graduate student, I've come to understand that my original instinct was right all along.

mikeskim 11y ago

If people would just learn to resample (cross validate, use subsampling or the bootstrap), we wouldn't be having this pointless discussion at all.

vasilipupkin 11y ago
In theory, you are correct, but some data sets are too small or make it too difficult to cross validate in a way, that is meaningful to the original problem

Houshalter 11y ago

P-values are so weird. Studies should instead report a likelihood ratio. A likelihood ratio is mathematically correct, and tells you exactly how much to update a hypothesis. You can convert p-values to likelihood ratios, and they are quite similar. But its not perfect. A p value of 0.05 becomes 100:5, or 20:1. Which means it increases the odds of a hypothesis by 20. So a probability of 1% updates to 17%, which is still quite small. But that assumes that the hypothesis has a 100% chance of producing the same or greater result, which is unlikely. Instead it might only be 50% which is half as much evidence. In the extreme case, it could be only 5% likely to produce the result, which means the likelihood ratio is 5:5 and is literally no evidence, but still has a p value of 0.05. Anyway likelihood ratios accumulate exponentially, since they multiply together. As long as there is no publication bias, you can take a few weak studies and produce a single very strong likelihood update.

altrego99 11y ago

The article tends to imply P-value should not be used at all, rather than misused. P-value definitely means something. For example, if the p-value is 1e-10 (which is often possible), you know for sure that the hypothesis generating has been disproved. So let me rephrase the title of the article - "It's time to use P-Value correctly."

purpled_haze 11y ago

...and I could not find the word "correlation" anywhere in this article discussion statistics, harm, misunderstanding, and studies. Shame.

ianamartin 11y ago

This is one of the best conversation threads I've ever seen on HN. It's both polite and informative. I want to toss in my own thoughts here. Since I've spent the vast majority of my tech career in the Market Research industry (hello, bias!), I'm tempted to say that one of the most frequent intersections between statistical science and business decisions happens in that world. Product testing, shopper marketing, A/B testing . . . these are pretty common fare these days. But I feel like the MR people are sort of their own worst enemy in many cases. It's a fairly recent development that MR people are even allowed a seat at the table for major product or business decisions. And when the data nerds show up at the meeting, we have to make human communication decisions that are difficult. I can't show up at the C-suite and lecture company executives about the finer points of statistical philosophy. When I'm presenting findings to stake-holders, it's my job to abstract the details and present something that makes a coherent case for a decision, based on the data we have available. It is sinfully attractive to go tell your boss's boss's boss that we have a threshold--a number we can point to. If this number turns out to be smaller than .05, this project is a go. Three months later, you go back to that boss and tell him the number came back and it was .0499999. The boss says, "Okay, go!" And then you are all, "Wait, wait, wait. Hang on a second. Let's talk about this." My god, what have I done? The practical reality of the intersection of statistics and business is a harsh one. We have to do better. In terms of leaky abstractions, the communication of data science to business decision makers is quite possibly the leaky-est of all. Why is it so leaky? I have two points about this. 1) Statistics is one of the most existentially depressing fields of study. There is no acceptance; there is no love; there is nothing positive about it. Ever. Statistics is always about rejection and failure. We never accept or affirm a hypothesis. We only ever reject the null hypothesis or we fail to reject it. That's it. 2) In business, we tend to be very very sloppy about formulating our hypotheses. Sometimes we don't even really think about them at all. Take a common case for market research. New product testing. We do a rep sample with a decent size (say, 1800 potential product buyers) and we randomly show five different products, one of which is the product the person already owns/uses (because that's called control /s). The other 4 products are variations on a theme with different attributes. What's the null hypothesis here? Does it ever get discussed? What's the alternative hypothesis? The implicit and never-talked-about null is that all things being equal, there is no difference between the distribution of purchase likelihood among all products. The alternative is that there is a real difference on a scale of likely to purchase. The implicit and intuitive assumption is that there is something about that feature set that drives the difference. (I'm looking at you, Max Diff) But that's not real. It's not a part of the test. The only test you can do in that situation is to check if those aggregate distributions are different from each other. The real null is that they are the same, and the alternative is that they are different. All you can do with statistics is tell if two distributions are isomorphic. Now, who wants to try to explain any of that to your CEO? No one does. Your CEO doesn't want it, you don't want it, your girlfriend doesn't want it. No one wants it. So we try to abstract, and I feel like we mostly fail at doing a good job of that. This is getting really long, and I don't want to rant. So to finish up, an idea for more effective uses of data science as it interacts with the business world: I agree, let's stop talking about p values. Let's work harder and funnel the results of those MR studies into practical models of the business' future. Let's take the research and pipe it into Bayesian expected value models. Let's stop showing stacked bar charts to execs and expecting them to make good decisions based on weak evidence we got from hypotheses we didn't really think about in the first place. Some of this might come across as a rant. I hope it is not taken that way. This is a real problem that I've been thinking about for a long time. And I don't mean to step on anyone's toes. I have certainly committed many of the data sins that I'm deriding above. Edited to add: The real workings of statistics are unintuitive. I'm not saying that they are wrong. But in working with people for years now, I understand the confusion. It's a psychological problem. Hypotheses are either not really well though out or not considered in an organized way, in my experience. A hypothesis is not concrete in many practical cases. It's a thought. An idea, perhaps. It's often a thing that floats around in your mind, or maybe you paid some lip service and tossed it into your note-taking app. Data seem much more real. You download a few gigabytes of data and start working on it. It's quite easy to get confused. I have real data! This is tangible stuff. Thinking of things properly and evaluating the probability of your data given the hypothesis is hard. Your data seems much more concrete. These are real people answering real questions about X. Even for people who are really hell-bent on statistical rigor, this is a challenge.

benjaminmhaley 11y ago

The #1 problem is with p-values is the word "significant". We should use "detectable" instead. Significant implies meaningful to most people, but not in a statistical context. This is quite confusing. Detectable is better because the mainstream meaning aligns with the jargon. So: > "Discovering statistically significant biclusters in gene expression data" becomes: > "Discovering statistically detectable biclusters in gene expression data" This rephrasing makes it evident that "statistically detectable" adds little to the title. So the title becomes > "Discovering biclusters in gene expression data" A better title.

giardini 11y ago

This excellent and amusing article by Gerd Gigerenzer discusses the history of p-values and their (mis)use: "Mindless Statistics" http://library.mpib-berlin.mpg.de/ft/gg/GG_Mindless_2004.pdf http://library.mpib-berlin.mpg.de/ft/gg/GG_Mindless_2004.pdf or http://www.unh.edu/halelab/BIOL933/papers/2004_Gigerenzer_JSE.pdf http://www.unh.edu/halelab/BIOL933/papers/2004_Gigerenzer_JS...