5 ms·
Statistics: P values are just the tip of the iceberg
- hnnewguy 12y agoTangentially related, but interesting, EconTalk episode that goes into p-values related to investment strategies, LHC experiments and other stuff: Campbell Harvey of Duke University talks with EconTalk host Russ Roberts about his research evaluating various investment and trading strategies and the challenge of measuring their effectiveness. Topics discussed include skill vs. luck, self-deception, the measures of statistical significance, skewness in investment returns, and the potential of big data. http://www.econtalk.org/archives/2015/03/campbell_harvey.html http://www.econtalk.org/archives/2015/03/campbell_harvey.htm...
- tedsanders 12y agoPersonally, I thought that episode was rather poor. Harvey made a number of claims that were just plain false. For example, he said that the p value is the probability that your result is true (!). He also said that the reason that physicists at the LHC require 5 sigma is because their data is so noisy (!). I think the first one was a miswording, but the second was a serious conceptual error. I wish Russ had pushed back a little harder on that episode.
- mellavora 12y agoStatistics are informative on the data you collected. But if you collected the wrong data, or collected it in the wrong way (i.e. biased sampling), they are less helpful at getting to the truth. Assuming, of course, you want the truth. Lies, damn lies, and statistics as they say.
- remarkEon 12y agoP-value related articles have been popping up on HN recently, but I'd like to say that another place needing scrutiny is probably back at the beginning in model specification. Bias introducing model design at the beginning render the rest of the analysis essentially meaningless. I know you're joking, but sometimes I wonder if "lies, damn lies, and statistics" isn't one of the worst phrases ever uttered in the English language. It's been a way for people to just say "hey, it's just a statistic. who cares!" and continue doing whatever they're doing. It's become this terrible meme where someone who openly admits that they "aren't a math person" gets to thrown down this trump card saying that your analysis is bullshit. But hey, I'm a baseball fan.
- sesqu 12y agoThe whole brouhaha over P-values has really been annoying me, specifically because 95% (Bayesian statistic) of the condemnations I've seen have been about people picking bad models until they get convenient P-values, and 5% have been about important misunderstandings about what they measure.
- remarkEon 12y agoThat is exactly what I'm saying.
- artumi-richard 12y agoI stopped with the statistics element of my math degree as soon as I could because of the disgust I had for the p-value decision thresholds.
- kriro 12y agoI'd recommend the following books (in that order) to anyone entering a p-value laden academic discipline: "What is a p-value anyway?" -> "Doing Bayesian Data Analysis" [I found the latter after reading one of the linked "no to p-values" articles here and I don't want to go back.]
- hessenwolf 12y agop-values are extremely useful for two things: 1. Teaching people who specialise in context areas (chemists, engineers) a pragmatic way to make a decision from data in the presence of randomness. 2. Automating analyses. (remember your Benjamini-Hochberg) I'd never actually use it myself. Jeepers, just look at the picture.
- travjones 12y agoI agree with #2, but I don't agree with #1. Why would one need a p-value to make a decision about the significance of a change in a dependent variable? Confidence intervals provide more information than p-values, and you can use them to identify significant effects. Moreover, p-values do not indicate the effect size that was observed. Thus, a very small effect in a very large sample can have a low/significant p-value.
- hessenwolf 12y agop-value testing and confidence interval testing are not interchangeable? also, p-values are used to determine the number of effects in a mixed model, or the number of factors in an ANOVA, etc. Sometimes confidence intervals on abstract parameters can be a bit less useful.
- travjones 12y agoYou are correct, they are not interchangeable. I didn't suggest that they were. I was offering CIs as an alternative for examining effects in general (e.g., comparing two groups); however, your choice of data representation will always depend on the question that you're asking. Nonetheless, p-values in and of themselves provide no information about the magnitude of an effect. Why not show the actual data?
- hessenwolf 12y agoI guess you mean clinical significance. http://www.bmj.com/content/348/bmj.g2130 http://www.bmj.com/content/348/bmj.g2130 This is a secondary issue. For most non-stats people, I wouldn't recommend the confidence interval as a measure of clinical significance. I'd rather look at the actual distribution of the two effects and view quantiles of the predictive interval. And, yes, in hypthoesis testing, p-values and confidence intervals are generally interchangeable. All they speak of is the overlap of the sample distribution of means, really.
- T-zex 12y agoI have an offtopic question, but believe I could find an answer here. Does anybody know a statistics equivalent book for Calculus Made Easy by Silvanus Phillips Thompson [1] Where everything is explained in layman's terms. [1] http://www.amazon.co.uk/Calculus-Made-Silvanus-Phillips-Thompson/dp/1456531980 http://www.amazon.co.uk/Calculus-Made-Silvanus-Phillips-Thom...
- pjmorris 12y ago'The Cartoon Guide to Statistics', Gonick might be useful. A number of smart people [1][2] recommend it. [1] http://www.stat.cmu.edu/~cshalizi/36-220/syllabus.pdf http://www.stat.cmu.edu/~cshalizi/36-220/syllabus.pdf [2] http://mathbabe.org/2014/07/11/the-lede-program-students-are-rocking-it/ http://mathbabe.org/2014/07/11/the-lede-program-students-are...
- mhuffman 12y ago"The Drunkard's Walk" and "Fooled by Randomness". TDW, in particular, kind of "tricks" you into learning statistics, in order to read the book.
- edtechdev 12y agoThere are some interactive books/courses/sites that are nice. They don't necessarily use 'layman terms' like that 100 year old calculus book: http://oli.cmu.edu/courses/free-open/statistics-course-details/ http://oli.cmu.edu/courses/free-open/statistics-course-detai... http://www.socr.ucla.edu/ http://www.socr.ucla.edu/ http://cast.massey.ac.nz/core/index.html?book=general http://cast.massey.ac.nz/core/index.html?book=general https://www.kno.com/book/details/productId/txt9780983804905 https://www.kno.com/book/details/productId/txt9780983804905
- ryanmonroe 12y agoStatistics in Plain English is a good one. http://www.amazon.com/Statistics-Plain-English-Third-Edition/dp/041587291X http://www.amazon.com/Statistics-Plain-English-Third-Edition...
- nextos 12y agoPrinciples of Statistics, by Bulmer. Dover.
- gregunderscorem 12y agoI think one of the core problems is summed up down near the end: the idea of statistics being taught by an apprenticeship model. There's a slightly more fleshed out version of this idea, also by Dr. Leek, over at SimplyStatistics[1] It might be my own statistics-porous media filter, but it seems like this issue is finally getting the attention it deserves. Hopefully this discussion will lead to bigger and better things. [1]http://simplystatistics.org/2015/04/29/data-analysis-subcultures/ http://simplystatistics.org/2015/04/29/data-analysis-subcult...
- craigching 12y agoThere was an awesome comment at the end by Stephen Heard: Agreed! Much of the "problem" with P-values comes from them being expected to do too much; they aren't (for example) indicators of effect size or final proof/disproof of an effect's reality. And careful attention to the P-value can't overcome problems at the earlier steps you lay out here. I would differ in one spot, though: a P-value should not be "the last of these [inferential] steps" but rather closer to the beginning of critical thought! (More about all this here: http://wp.me/p5x2kS-Y.) I like the "p-value should be the beginning of critical though", I'm going to remember that one.
- CWuestefeld 12y agoI was just reading a related article in economics. From that article (warning, while the topic is general, it uses minimum wage research as an example): It is indisputable that a theory that is inconsistent with empirical data is a poor theory. No theory should be accepted merely because of the beauty of its logic or because it leads to conclusions that are ideologically welcome or politically convenient. Yet it is naive in the extreme to suppose that facts – especially the facts of the social sciences – speak for themselves. Not only is it true that sound analysis is unavoidably a judgment-laden mix of rigorous reasoning (“theory”) with careful observation of the facts; it is also true that the facts themselves are in large part the product of theorizing. ... No one who reads this work – and there’s plenty of it – can possibly decide, based on the data alone, whether minimum-wage legislation does or does not reduce the employment options of low-skilled workers. It’s not just that the conclusions drawn from the empirical evidence differ wildly. It’s more that coming to a conclusion requires reasoning about - theorizing about – the data. These data (like all data) do not speak for themselves. It’s just not what data do. http://cafehayek.com/2015/04/theorizing-about-the-facts-theres-no-escaping-theory.html http://cafehayek.com/2015/04/theorizing-about-the-facts-ther...
- baldfat 12y ago> politically convenient I am involved in local politics and currently running for School Board in my city (We have 14 people running for 5 seats) Political convenient = how things work locally. If you make a proposal you have to see it to 4 other people to get it past. If they know their email or Facebook or Twitter would explode they won't budge. If you show that it is idea is convenient to them specifically you are a genius.
- cafebeen 12y agoPerhaps the most important point from the article: "Statistical research largely focuses on mathematical statistics, to the exclusion of the behaviour and processes involved in data analysis" If there's a major problem in data collection, cleaning, or hypothesis generation, it doesn't matter whether NHST or Bayesian methods are used
- Osmium 12y agoIs there a canonical statistics textbook to recommend to the professional scientist?
- pjmorris 12y agoI don't have a definitive answer, but I'll start the bidding with 'Statistics for Experimenters' [1] [1] 'Statistics for Experimenters', G. Box, W.G. Hunter, J.S. Hunter, Wiley
- a_bonobo 12y agoI'll raise you Motulsky's Intuitive Biostatistics It doesn't teach you how to calculate the steps of a t-test, but it will teach you in which cases to apply what test, the pitfalls of the test, and how to interpret the results
- rm999 12y agoIn my experience around a lot of professional data analysts, shoddy scientific design is ubiquitous; I was definitely guilty of it all the time. A mentor gave me some advice that helped tremendously: formulate what you're trying to answer very precisely (i.e. the hypothesis), and remember you are only trying to answer that question during every step of your experiment. It's very easy to lose focus or rush through important steps that seem insignificant. This is dangerous because bad methodology often looks a lot like good methodology but leads to very different results. When you have a precise hypothesis, it's much simpler to look at a step and say "no, that's not answering the question correctly".
- louden 12y agoThis has been my experience as well. To combat this, I always start with a written analysis plan that forces me to think out all of my analyses.