9 ms·
Statistics Done Wrong – The woefully complete guide
- mathattack 13y agoI like that he references Huff's "How to lie with statistics" in the first sentence of the intro. That was the book that came to mind when I saw the subject. Also reminds me of the Twain quote, "There are three types of lies: Lies, Damned Lies, and Statistics." But despite this, statistics done well are very powerful.
- neuralk 13y agoWith respect to that Twain/Disraeli quote, my friend who is a professor of statistics tells me that he cannot go to a party and say what he does for a living without someone repeating it smirkingly.
- dragonwriter 13y agoIsn't that why the name "Data Science" was invented?
- neuralk 13y agoData science sure sounds sexier...ish. The irony of people who use the "damned lies and statistics" quote snidely is that the "statistics" part is not referring to the field Statistics but the plural version of the noun statistics, which of course are easily abused. The field of Statistics is all about NOT abusing statistics.
- mathattack 13y agoExactly. If you're doing statistics for yourself, it's good to know the tricks so that you don't fool yourself by mistake. Many times people use statistics to support their positions, rather than to make up their minds. If done right, data science or statistics is about the latter.
- jessaustin 13y ago...ish. Yeah, it always makes me wonder when they have to put the "science" in the name. "Computation" seems so much more timeless and elegant than "computer science", for instance. It's almost like "Democratic Republic" for nations.
- dragonwriter 13y agoYou know, lotsfields are named that way. Its just that the some of them are less obvious because both the part of the name identifying the domain and the part saying its the science (or "study", but with the same intended meaning) are in Latin, instead of plain English.
- mathattack 13y agoIt gets worse with Political Science, or perhaps the broader Social Sciences.
- daughart 13y agoAs a graduate student in the life sciences, I was required to take a course on ethical conduct of science. This gave me the tools to find ethical solutions to complex issues like advisor relations, plagiarism, authorship, etc. We were also taught to keep good notes and use ethical data management practices - don't throw out data, use the proper tests, etc. Unfortunately, we weren't really taught how to do statistics "the right way." It seems like this is equally important to ethical conduct of science. Ignorance is no excuse for using bad statistical practices - it's still unethical. By the way, this is at (what is considered to be) one of the best academic institutions in the world.
- dbaupp 13y ago> Unfortunately, we weren't really taught how to do statistics "the right way." Learning the right way takes a lot of work, there's a lot of ways to analyse things, each one wrong/right in different situations. (Even teaching something as "simple" as the correct interpretation of a p-value is hard.)
- daughart 13y agoI'm sure it does, but they don't have a problem assigning other required classes, such as a one-hour-a-week communication of science class. One hour a week for a year is enough to cover a lot of material.
- ultrasaurus 13y agoOne thing that constantly saddens me about statistics is that a large amount of energy is expended using is almost correctly to "prove" something that was already the gut feel. Even unbiased practitioners can be lead astray [1] but standards on how not to intentionally lie with statistics are very useful. [1] http://euri.ca/2012/youre-probably-polluting-your-statistics-more-than-you-think/index.html http://euri.ca/2012/youre-probably-polluting-your-statistics...
- onion2k 13y agoThere's no way to tell whether or not that "gut feel" is accurate without proof. Often it's right, but occasionally it's very, very wrong (cancer risk and Bayes theory provides a good illustration: http://betterexplained.com/articles/an-intuitive-and-short-explanation-of-bayes-theorem/ http://betterexplained.com/articles/an-intuitive-and-short-e...). Consequently it's still worthwhile proving things even when they're seemly obvious.
- darkxanthos 13y ago"Consequently it's still worthwhile proving things even when they're seemly obvious." Not true. It's all about risk vs. payoff. Some things are low risk enough we can go by gut, others we need more evidence. It's all about tuning for false positives and negatives. EDIT: Added quote of what I was responding to.
- hsitz 13y agoHe said he's saddened by people using statistics "almost" correctly to prove what was already "gut feel". I can't quite tell whether his real concern is what you thought he was saying (i.e., wasted effort), or whether it's the "almost" (but not quite correctly) part, i.e., that researchers use statistics to wrongly prove the gut feel. If it's the latter, then I think a big part of the problem is that people don't understand statistics well enough, not that they intentionally misuse it.
- pessimizer 13y agoI think his point was that people seem to "prove" common sense statistically all of the time - but when doing so make a lot of thoughtless assumptions about representativeness, significance, definitions, etc. stemming from the unspoken assumption of a particular outcome being inevitable. Or maybe I'm projecting?
- ambiate 13y agoThe greatest problem of statistical analysis is throwing out observations which do not fit the bill. All analysis should be thoroughly documented with postmortems.
- glutamate 13y agoOne way to do statistics less wrong is to move from statistical testing to statistical modelling. This is what we are trying to support with BayesHive at https://bayeshive.com https://bayeshive.com Other ways of doing this include JAGS (http://mcmc-jags.sourceforge.net/ http://mcmc-jags.sourceforge.net/) and Stan (http://mc-stan.org/ http://mc-stan.org/) The advantage of statistical modelling is that it makes your assumptions very explicit, and there is more of an emphasis on effect size estimation and less on reaching arbitrary significance thresholds.
- jimmar 13y agoOne of the many challenges in science is that there is no publication outlet for experiments that just didn't pan out. If you do an experiment and don't find statistical significance, there aren't many journals that want to publish your work. That alone helps contribute to a bias toward publishing results that might have been found by chance. If 20 independent researchers test the same hypothesis, and there is no real effect, 1 might find statistical significance. That 1 researcher will get published. The 19 just move on.
- essayist 13y agoThey are working on it. There was an effort at Oxford to track perinatal trials -- started in the 1980's. It looks like it hasn't happened yet, but that various major players (PLOS, Center for Evidence-based Medicine) want to expand the brief to cover all clinical trials: http://www.alltrials.net/about/ http://www.alltrials.net/about/ http://www.cochrane.org/about-us/history http://www.cochrane.org/about-us/history
- capnrefsmmat 13y agoFor many types of clinical trials, pre-registration and publication of results through ClinicalTrials.gov is required by the FDA. I think it's been five or ten years now. Unfortunately, compliance isn't great -- something like 80% of studies registered on ClinicalTrials.gov never have results published there. 20% of registered studies never have results published anywhere.
- RogerL 13y agoThis is called the "file drawer effect".
- r0muald 13y agoThere's Figshare <http://figshare.com/> http://figshare.com/> to publish negative data.
- capnrefsmmat 13y agoHey everyone, I'm the author of this guide. It's come full circle -- I posted it a week ago in a "what are you working on?" Ask HN post, someone posted it to Metafilter and reddit, and it made its way to Boing Boing and Daily Kos before coming back here. I'm currently working on expanding the guide to book length, and considering options for publication (self-publishing, commercial publishers, etc.). It seems like a broad spectrum of people find it useful. I'd appreciate any suggestions from the HN crowd. (A few folks have already emailed me with tips and suggestions. Thanks!) (Also, I'm sure glad I added that email signup a couple weeks ago)
- mey 13y agoIs there any way to print the entire book as a document? Do you accept donations?
- sudont 13y agohttp://sphinx-doc.org/latest/builders.html http://sphinx-doc.org/latest/builders.html Sphinx can output Latex for printing to PDF. We'd just need the source. BTW, excellent refresher for statistical screw-ups, I had forgotten a quarter of these (and never learned the rest.)
- capnrefsmmat 13y agoNot yet. I intend to get it published so you can buy a printed copy, but that hasn't happened. I don't accept donations, but sign up with your email and I'll let you know when you can pay me for a copy.
- kisielk 13y agoYou might want to take a look at Leanpub (https://leanpub.com https://leanpub.com) if you haven't already. Seems like it may suit you.
- capnrefsmmat 13y agoUnfortunately, Leanpub exclusively uses Markdown, which doesn't support things like using a BibTeX bibliography for references. I also don't think there's a way to customize the design of the print book, apart from cover images and such. I'm writing using reStructuredText and Sphinx because they give me a bibliography, an index, full-text search, cross-referencing, custom environments (e.g. boxes for examples, tips, etc.), and all sorts of other cool features. For example, if I needed matplotlib plots, I can embed the code in my documents and Sphinx will generate them. I'd immediately put the book on Leanpub if I could get some of these features. A Leanpub using a customized Sphinx would be incredibly useful for people writing textbooks.
- Tycho 13y agoIf I was a billionaire, I would set up some sort of screening lab for scientific/academic/research papers. There would be a statistics division for evaluating the application of statistical methods being used; a replication division for checking that experiments do actually replicate; and a corruption division for investigating suspicious influences on the research. It would be tempting to then generate some sort of credibility rating for each institution based on the papers they're publishing, but that would probably invite too much trouble, so best just to publish the results and leave it at that. Arguably this would be a greater benefit to humanity than all the millions poured charitably into cancer research etc.
- davmre 13y agoSomething like that idea has actually already been the inspiration for at least one startup: MetaMed (http://en.wikipedia.org/wiki/MetaMed http://en.wikipedia.org/wiki/MetaMed, http://nymag.com/health/bestdoctors/2013/metamed-personalized-health-care/ http://nymag.com/health/bestdoctors/2013/metamed-personalize...) does meta-level analysis of the medical literature to determine which treatments seem effective for rare conditions, taking into account the sample size, statistical methodology, funding sources, etc. of each study. Of course, medicine might be unique as a domain in which individuals are willing to pay vast sums of money to obtain slightly more trustworthy research conclusions, and the profit motive has obvious conflicts with "benefit to humanity" (if someone pays you to research a treatment for their disease, do you post the findings when done? Or hold them privately for the next person with the same problem?). But maybe there are other domains in which the market could support a (non-billionaire's) project for better-validated research.
- gwern 13y agoMetaMed, as far as I know, does basically just customized literature reviews; it isn't doing anything I'd recognize as 'meta-level analysis' like the work done by the Cochrane Collaboration or using meta-analytic techniques to directly estimate the reliability of existing medical treatments or beliefs.
- merckx 13y ago
- jmatt 13y agoNorvig's "Warning Signs in Experimental Design and Interpretation" is also worth reading and covers the higher level problem of bad research and results. Including mentioning bad statistics. http://norvig.com/experiment-design.html http://norvig.com/experiment-design.html
- Paradigma11 13y agoQuite a few years ago i devised an ambitious method to achieve significance while sitting through another braindead thesis presentation (psychology): If you are interested in the difference of a metric scaled quantity between two groups do the following: 1.) Add 4-5 plausible control variables that you do not document in advance (questionaire, sex, age...). 2.) Write a r-script that helps you do the following: Whenever you have tested a person increment your dataset with the persons result and run a: t-test u-test ordinal logistic regression over some possible bucket combinations. 3.) Do this over all permutations of the control variables. Have the script ring a loud bell when significance is achieved so data collection is stopped immediately. An added bonus is that you will likely get a significant result with a small n which enables you to do a reversed power analysis. Now you can report that your theoretical research implied a strong effect size so you choose an appropriate small n which, as expected, yielded a significant result ;)
- chrislipa 13y agoXKCD did it first: http://xkcd.com/882/ http://xkcd.com/882/
- Aloisius 13y agoIf you want to see truly egregious uses of statistics, take a look at any paper on diet or nutrition. Be prepared to be angry. At this point, if someone published a study stating that we needed to eat not to die, I'd be skeptical of it.
- capnrefsmmat 13y agoYou might enjoy this: Schoenfeld, J. D., & Ioannidis, J. P. A. (2013). Is everything we eat associated with cancer? A systematic cookbook review. American Journal of Clinical Nutrition, 97(1), 127–134. doi:10.3945/ajcn.112.047142 They did a review of cookbook ingredients and found that most of them had studies showing they increased your risk of cancer, while also having studies showing they decrease your risk of cancer. I think bacon was a notable exception -- everyone agreed that it increases your cancer risk.
- WettowelReactor 13y agoWhat is puzzling to me is that many of the statistical errors showing up in all the science literature are well understood. The problem is not all the junk science that is being generated but that the current tools and culture are not readily naming and shamming these awful studies. Just as we have basic standards in other fields such as GAAP in finance why can' we have an agreed upon standard for data collection and analysis of scientific data?
- VladRussian2 13y agowhenever there is discussion about statistics role in science (sometimes even going as far as crossing into how science is statistics) i always remember this: http://en.wikipedia.org/wiki/Oil_drop_experiment#Fraud_allegations http://en.wikipedia.org/wiki/Oil_drop_experiment#Fraud_alleg...
- evacuationdrill 13y agoMore revealing than the fraud allegations is the next section discussing the way results had pressure from his and other experiments' results for years, delaying our arrival at a more precise measurement. It reads as though it wasn't so much malice as it was self-doubt that lead to the scientists' actions.
- tokenadult 13y agoI see the author of this interesting site is active in this thread. You may already know about this, but for onlookers I will mention that Uri Simonsohn and his colleagues http://opim.wharton.upenn.edu/~uws/ http://opim.wharton.upenn.edu/~uws/ have published a lot of interesting papers advising psychology researchers how to avoid statistical errors (and also how to detect statistical errors, up to and including fraud, by using statistical techniques on published data).
- capnrefsmmat 13y agoThanks. I had seen some of his work, but browsing his list of publications I found a few more interesting papers. I've already worked one into my draft.
- Anon84 13y ago"Statistical significance does not mean your result has any practical significance."
- knassy 13y agoThat was an excellent read. Thank you. I'll admit I'm often reluctant to read to much in to data I deal with daily (web analytics), as I'm unsure of how to measure its significance accurately. I'm going to dive in and learn more about this.