7 ms·
Statistics Done Wrong
- ASpring 8y agoHighly recommend this book. I would consider myself experienced with many statistical methods but this book was still chock full of brilliant examples that let me look at things with fresh eyes. It was helpful also in giving me language to explain technical concepts to less technical folks.
- HillaryBriss 8y agoyeah. the "base rate fallacy" example sheds some light on the pros/cons of mammogram results. one interesting thing about that section of the book is that it says medical doctors fall prey to the fallacy more often than not. i always kind of wonder when i see a medical doctor explaining statistics to a general audience: do they really have this right?
- misiti3780 8y agoI also found this passage about mammographers from "Moonwalking with Einstein" interesting: For most mammographers, practicing medicine is not deliberate practice, according to Ericsson. It’s more like putting into a tin cup than working with a coach. That’s because mammographers usually only find out if they missed a tumor months or years later, if at all, at which point they’ve probably forgotten the details of the case and can no longer learn from their successes and mistakes. One field of medicine in which this is definitively not the case is surgery. Unlike mammographers, surgeons tend to get better with time. What makes surgeons different from mammographers, according to Ericsson, is that the outcome of most surgeries is usually immediately apparent—the patient either gets better or doesn’t—which means that surgeons are constantly receiving feedback on their performance.
- hinkley 8y agoMany of us in the software community know that if the feedback loop isn't fast enough, it doesn't work. I wonder if you made mammographers spend a day a week analyzing mammograms from 5 years ago and then showing them the outcomes if they would get more accurate?
- hydrox24 8y agoGiving them the information could certainly help, and it is a good first step. Mammographers would still be missing the motivation surgeons get when they see a patient die hours or days later. The emotional kick in the pants is harder to provide.
- heavenlyblue 8y agoWell, that’s pretty much like training a neural network, except that the integration of the matrix coefficients happens not from the immediate output data, but from data which was output by the network millions of epochs ago.
- SkyPuncher 8y ago> i always kind of wonder when i see a medical doctor explaining statistics to a general audience: do they really have this right? My wife is in her last year of medical school. I wonder the same question frequently when talking with her peers and seniors. They certainly get taught the basics of interpreting statistics, but I don't really see the "average" doc discussing things in a critical manner. The system seems deeply focused on the results without critically thinking about the context and appropriateness of the studies.
- rscho 8y agoI can answer that clearly for you. 99+% of docs don't have the slightest clue about how statistics are done, or even about what they mean (am MD with special interest in stats).
- r4um 8y agoAnother good curation https://web.ma.utexas.edu/users/mks/statmistakes/StatisticsMistakes.html https://web.ma.utexas.edu/users/mks/statmistakes/StatisticsM... How to Lie with Statistics by Darrell Huff https://en.wikipedia.org/wiki/How_to_Lie_with_Statistics https://en.wikipedia.org/wiki/How_to_Lie_with_Statistics
- pitzahoy 8y agoLove the Huff read. +1 to that.
- hinkley 8y agoI'm fairly certain that most of my opening salvos in evaluating the quality of someone's charts come to me from people regurgitating Huff's book. I'd put it in a class with Fred Brooks; dead obvious, but somehow needing to be constantly re-explained to a new group of people (or the same people at a later date).
- yesenadam 8y ago>dead obvious, but somehow needing to be constantly re-explained Because it's not all 'dead obvious', maybe?
- slavik81 8y ago"Science is made up of so many things that appear obvious after they are explained." ~ Pardot Kynes, Dune
- yesenadam 8y agoThere's a great book about this, Everything is Obvious (Once You Know The Answer) about how many scientific studies come up with obvious-seeming findings, but if they'd found the opposite, that would have seemed obvious to us too..
- hermitdev 8y ago
- wirrbel 8y agoI think I never was in a developed field that was as error prone as statistics, both intentional (fraud, p value hacking, etc) and with honest mistakes.
- tarabanga 8y agofinance?
- RA_Fisher 8y agoStatistics might be seen as the process of rooting out errors and falsehoods. To come into contact with them is the goal (to eliminate them). :)
- dcl 8y agoIndeed. It's the good statisticians that use good statistical methodology and theory to identify out errors and falsehoods. Honest mistakes occur because what looks like a simple problem cannot always be analysed very easily (or in an obvious way). Abuse occurs because it's quite easy to fool others that an analysis is sound - most people aren't sophisticated enough to identify problems, even if they are given the data!
- edoo 8y agoI really loved xkcd 2059.
- danenania 8y agoThis is a great book. I read it a couple years ago and I remember a couple takeaways that apply well to AB testing: 1 - Monitoring tests on an ongoing basis and then calling them as soon as they hit some confidence threshold (like 95%) will give you biased results. It's important to determine your sample size up front and then let the test run all the way through, or at least be aware that the results are less reliable if you stop early. 2 - Testing for multiple metrics requires a much larger sample. If you run a test and then compare conversion rate, purchase amount, pageviews per session, retention, etc. etc., you'll have a much higher error rate since the more things you measure, the more likely you are to get an outlier. You either need to run a separate test for each metric or increase your sample size a lot to account for this effect (iirc the math for exactly how much is in the book).
- capnrefsmmat 8y agoThanks, I'm glad you enjoyed the book! (Author here -- the website got its first publicity here on HN.) Regarding AB testing, you might be interested in this recent research, which uses real data from Optimizely to estimate how often people get AB test false positives because they stopped as soon as they hit significance: https://ssrn.com/abstract=3204791 https://ssrn.com/abstract=3204791 > Specifically, about 73% of experimenters stop the experiment just when a positive effect reaches 90% confidence. Also, approximately 75% of the effects are truly null. Improper optional stopping increases the false discovery rate (FDR) from 33% to 40% among experiments p-hacked at 90% confidence
- Legogris 8y agoHi, I didn't see anything on the page about "intended audience" - would you say this is appropriate for someone who has done the basic classes of statistics in uni but is now pretty rusty or would you need a more solid foundation to be able to grasp the content fully?
- montecarl 8y agoWhile it may be possible take the frequentist approach to AB testing, Bayesian inference is becoming the way to go with this.[1] Instead of directly setting up a yes-no hypothesis test with the nearly impossible to use correctly p-values[2], Bayesian approaches aim to directly estimate whatever quantity you want. With the Bayesian approach, you get estimates for A, B, and A-B (or whatever combination you want, e.g. (A-B)/A). Each of those estimates are properly called posterior probability distributions and describe the range of possible values. The end result is that instead of saying "A is better than B (p < 0.05)" you get a probability distribution of A-B. From that probability distribution you can answer any question you want: the most likely difference between A and B (the average), the probability that A is better than B (just integrate the area above 0), or whatever is needed to make a decision. [1] https://conversionxl.com/blog/bayesian-frequentist-ab-testing/ https://conversionxl.com/blog/bayesian-frequentist-ab-testin... [2] https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf
- fiatjaf 8y agoI thought this was going to a be a collection of statistics people really did wrong.
- devit 8y agoIsn't most or all of this avoided by explicitly using Bayes' theorem along with a correct formalization of the domain? E.g. for the mammogram: P(cancer) = 0.8% P(~cancer) = 1 - P(cancer) = 99.2% P(positive_mammogram | cancer) = 90% P(positive_mammogram | ~cancer) = 7% P(cancer | positive_mammogram) = P(positive_mammogram | cancer) P(cancer) / (P(positive_mammogram | cancer) P(cancer) + P(positive_mammogram | ~cancer) (1 - P(cancer))) = 90% * 0.8% / (90% * 0.8% + 7% * 99.2%) = 9.39457%
- HillaryBriss 8y agoi think the answer is yes, it is avoided in the way that you demonstrate. i guess it's a matter of using an approach that reaches your audience. the book explains it with prose: Imagine 1,000 randomly selected women who choose to get mammograms. Eight of them (0.8%) have breast cancer. The mammogram correctly detects 90% of breast cancer cases, so about seven of the eight women will have their cancer discovered. However, there are 992 women without breast cancer, and 7% will get a false positive reading on their mammograms, giving us 70 women incorrectly told they have cancer. In total, we have 77 women with positive mammograms, 7 of whom actually have breast cancer. Only 9% of women with positive mammograms have breast cancer.
- vanderZwan 8y agoI know this is an example, but this assumes false positives and false negatives are equally likely, and I now am wondering if that is true in real life.
- janekm 8y agoIt doesn't: P(positive_mammogram | cancer) = 90% P(positive_mammogram | ~cancer) = 7%
- thaumasiotes 8y agoTo be a little clearer, the false negative rate is 10% and the false positive rate is 7%.
- tomrandle 8y agoI really rate this book so it’s great to see it getting some publicity! I think one misnomer is that to do stats right you just need to do the math right. But what analysis is done in the first place, the overall methodology and how that data is interpreted are more often where we go wrong. At Geckoboard we’ve been trying to raise awareness of some of these issues. Heres a poster we put together: https://www.geckoboard.com/learn/data-literacy/statistical-fallacies/ https://www.geckoboard.com/learn/data-literacy/statistical-f...
- stared 8y agoFor practical statistic done well, I recommend "Sex by Numbers" by David Spiegelhalter (https://www.amazon.com/Sex-Numbers-Wellcome-David-Spiegelhalter/dp/1781253293/ https://www.amazon.com/Sex-Numbers-Wellcome-David-Spiegelhal...). I am reading Sex by Numbers, which I enjoy a lot. It's a touchy subject with a lot of data of varying quality. A naive approach would be to take all of it. A dogmatic - to set an arbitrary (and subjective!) threshold, separating "good" from "bad" data. I love the way, in which it is done there, i.e. by grading sources by: 4: numbers that we can believe (e.g. births and deaths) 3: numbers that are reasonably accurate (e.g. well-designed & conducted surveys, e.g. the Natsal report) 2: numbers that could be out by quite a long way (e.g. non-uniform sampling, the Kinsey report) 1: numbers that are unreliable (e.g. surveys from newspapers, even with huge sample sizes) 0: numbers that have just been made up (e.g. "men think of sex every 7 seconds") Just have a peek at the first chapter, which is freely accessible, and is exactly on data reporting, data reliability and dealing with subjective questions. A lot of thought is given about knowing the possible biases (e.g. people who are less likely to respond, who would like to downplay or exaggerate some things) and consistency of measurements. So - in short: started reading to get curious facts about sex, ended up recommending to my data science students and mentees. (As the vast majority of problems starts with how you collects data, interpret it, and how well are you aware of its shortcomings).
- jdietrich 8y agoAlong similar lines, Dr Ben Goldacre's Bad Science is an excellent beginner's introduction to the scientific method. If you understand how quacks bamboozle unwitting journalists, you gain a key insight into what good science looks like. https://www.amazon.com/Bad-Science-Quacks-Pharma-Flacks/dp/0771035780 https://www.amazon.com/Bad-Science-Quacks-Pharma-Flacks/dp/0...
- oriettaxx 8y agoThis has always been a great book https://en.m.wikipedia.org/wiki/How_to_Lie_with_Statistics https://en.m.wikipedia.org/wiki/How_to_Lie_with_Statistics