9 ms·
Statistical Mistakes and How to Avoid Them
- lisper 10y agoThis is the insight that made statistics "click" for me many years ago: a statistical test answers one central question: what are the odds that the results you observed could have arisen by chance? If those odds are low, then you are justified in concluding that the results probably did not arise by chance, and so there must be some other explanation (usually, but not always, the causal hypothesis you are advancing). Once consequence of this is that it is crucial that you advance your hypothesis before you collect (or at least look at) the data because the odds of something arising by chance change depending on whether you predict or postdict the results. Also, the more data you have, the more likely you are to find something in there that looks like a signal but is in fact just a coincidence. Many a day-trading fortune has been lost to this one mistake.
- stdbrouw 10y ago> a statistical test answers one central question: what are the odds that the results you observed could have arisen by chance Well, no, that'd be very interesting but unfortunately what a statistical test really says is the probability of the results you observed (or more extreme) given chance. P(data|model) and not P(model|data).
- lisper 10y agoIt's actually P(data|null-hypothesis).
- wfunction 10y agoThe model is the null hypothesis here... I don't think the parent comment was wrong w.r.t. this.
- lisper 10y agoNo, there's a difference. The model is something like, "This drug attaches itself selectively to cancer cells and kills them." The null hypothesis is, "This drug has no effect." So you conduct a double-blind study, measure the effect of the drug on cancer cells, collect some data and compute that P(data|null-hypothesis) is 1%. It it not the case that there is a 99% chance that your model is correct and that the drug does in fact attach itself to cancer cells and kill them. Because there are other possible models, e.g.: the drug redirects your Chi and channels it into your chakras which kills the cancer cells. Statistics alone cannot tell you which of those two models is correct.
- fluxion 10y agoI think this is just an argument over semantics, the "null hypothesis" is a perfectly valid model using the definition of the statistics community [0] (i.e. a collection of probability distributions over some sample space). [0] https://en.wikipedia.org/wiki/Statistical_model https://en.wikipedia.org/wiki/Statistical_model
- lisper 10y agoThe word "model" is being used here in two mutually incompatible ways. Yes, the null hypothesis is a model, but it is not an explanatory model. A scientific hypothesis has to meet two tests to be considered a valid theory. It has to be consistent with the data, and it has to have explanatory power. The theory that cancer drugs work by aligning a patient's chi with their chakras is rejected not because it is inconsistent with the data (it's not) but because it lacks the explanatory power of alternative theories based on molecular biology. The null hypothesis never has explanatory power. The null hypothesis is always a statement of the form, "The explanatory hypothesis under test is wrong for some unknown reason." This is why rejecting the null hypothesis, i.e. showing that the data are (with high probability) inconsistent with the null hypothesis, is considered a positive result.
- nonbel 10y ago>"Statistics alone cannot tell you which of those two models is correct." Statistics can tell you whether a model is consistent with the data. But you need to deduce the null hypothesis from your model rather than use the default "no difference" (of course, sometimes no difference is deduced from a real model, but not often, in that case: great!). In fact, that is the proper use of statistics. I would guess >99.99% of current usage is incorrect (ie pseudoscience) and amounts to a waste of time at best. The usual usage turns scientific reasoning on its head, and has lead to a (literally for most people) unbelievable amount of trouble. This was pointed out most aptly by Paul Meehl long, long ago: http://www.fisme.science.uu.nl/staff/christianb/downloads/meehl1967.pdf http://www.fisme.science.uu.nl/staff/christianb/downloads/me...
- qwrusz 10y agoHow about you guys are both right, sort of. There are both Bayesian and Frequentist approaches in statistics! They represent very different methods to statistics, but they are also quite similar. My apologies, I couldn't find one link that gave a good description of Bayesian vs. Frequentist. Here are a couple links to get started: https://xkcd.com/1132/ https://xkcd.com/1132/ http://jakevdp.github.io/blog/2014/03/11/frequentism-and-bayesianism-a-practical-intro/ http://jakevdp.github.io/blog/2014/03/11/frequentism-and-bay... If anyone comes across a link that describes Bayesian and Frequentist clearly, please share it if you don't mind.
- lisper 10y agoThat cartoon is actually a pretty good illustration of why frequentists are wrong.
- qwrusz 10y agoDo you mean probably wrong? Either way I disagree. Both approaches have pros and cons depending on the type of analysis. At the very least, just the presence of competing approaches in the field has pushed statisticians to have more rigour and do way more double checking than they might have, out of fear the other side actively looking to poke holes. It's easy to lie with statistics and the only people who can call statisticians out on their BS is other statisticians...Need more of this.
- lisper 10y ago> Do you mean probably wrong? Nope. > Both approaches have pros and cons What are the pros of the frequentist approach?
- danieltillett 10y agoSimple maths that can be done without a computer. Useful in the past - not so useful today.
- rspeer 10y ago
- stdbrouw 10y agoYup, and the null hypothesis is a reduced model. Different words, same thing.
- lisper 10y agoI have no idea what you mean by "reduced" but they are not the same thing. The difference in their explanatory power. A (scientific) hypothesis explains things. A null hypothesis simply says, "This hypothesis is wrong" but doesn't say why. It's a crucial distinction.
- deleted 10y ago[deleted]
- kgwgk 10y ago> what are the odds that the results you observed could have arisen by chance? > the probability of the results you observed (or more extreme) given chance I think he said the same thing as you. He didn't talk about "the odds that the results you observed have arisen by chance" but the odds that the results you have observed (the or more extreme is implicit) could have arisen by chance (i.e. the probability of happening by chance alone, given that only chance was involved).
- fela 10y agoUnfortunately no. Very much no, even though it's widely believed that that is a good definition/intuition (and used in many places). It's the odds of having that results due to chance, if the null hypothesis is true[0]. That latter part might sound pedantic, but the whole point is that we don't know how likely the null hypothesis is. If I test wheather the sun has just died[1] and get a p-value of 0.01 it's still very likely that this result is due to change (surely more than 1%)! We need a prior probability (i.e. bayesian statistics) to calculate the probability that the result was due to chance, that is why that partial definition is incomplete and actually very misleading. This point is subtle, but very important to really understand p-values. Another way to look at it is: if we knew the probability that the result was due to chance we could also just take 1-p and have to probability of there actually being some effect, a probability that hypothesis testing cannot give us. There is one nice property that hypothesis testing does have (and why presumably it's so widely used): if the idea you are testing is wrong (which actually means "null hypothesis true") you will most likely (1-p) not find any positive results. This is good, this means that if the sun in fact did not die, and use 0.01 as your threshold, 99% of the experiments will conclude that there is no reason to believe the sun has died. So hypothesis testing does limit the number of false positive findings. The xkcd comic is a bit misleading it this regard, yes it does highlight the limitations of frequentist hypothesis testing, but the scenario depicted is a very unlikely one, in 99% of the cases there would have been a boring and reasonable "No, the sun hasn't died". For an incredibly interesting article about the difficulty of concluding anything definitive from scientific results I highly recommend "The Control Group is out of Control" at slatestarcodex[2]. [0] To be even more pedantic you would have to add "equal or more extreme", and "under a given model", but "if the null hypothesis is true" is by far the most important piece often missing. [1] https://xkcd.com/1132/ https://xkcd.com/1132/ [2] http://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ http://slatestarcodex.com/2014/04/28/the-control-group-is-ou...
- lisper 10y ago> It's the odds of having that results due to chance, if the null hypothesis is true[0]. Yes, that's right. I don't know why you think this is at odds with what I said. In fact, I clarified this myself a few hours ago in a sibling comment: https://news.ycombinator.com/item?id=13026907 https://news.ycombinator.com/item?id=13026907
- deleted 10y ago[deleted]
- amelius 10y agoI don't like how the article tries to push statistics on the reader. If a CS paper compares a pair of averages, then that gives certain information. If statistics can add to that, and make the results a little more precise, then that is nice. But by no means is it absolutely necessary. And statistics will not give a conclusive result either. I think that authors should use statistics when they see fit, and when it does not distract too much from the original subject of the paper.
- esfandia 10y agoIMO it's better to just plot the two distributions that you want to compare so that people can eyeball the difference (or run the statistical tests themselves should they wish to do so). For one thing not all distributions are Gaussian. And the t-test only answers one specific question (i.e., with one given p-value). Then there's people who misinterpret the result of the t-test. Or people who mess with the p-value till they get what they need.
- samps 10y agoNeedless to say, I disagree. It can be straight-up misleading to report means without including a more nuanced view of the distribution. You don't need to use a bunch of fancy statistics, but you do need to consider whether your results could have arisen by random chance. That's not a distraction; it's accurately reporting what you found. Here's one frightening example of spurious performance results in CS: https://www.cis.upenn.edu/~cis501/papers/producing-wrong-data.pdf https://www.cis.upenn.edu/~cis501/papers/producing-wrong-dat...
- amelius 10y ago> It can be straight-up misleading It is only misleading if the reader doesn't understand statistics. There is, imho, nothing wrong with putting all your focus on the subject matter, and skipping the statistics while being frank about it. Also, if you need statistics to show that your method is better than other methods, then perhaps your method is not really that much better.
- 10y ago
- frozenport 10y agoThe t-test assumes a normal distribution which, is rarely true, especially when the number of runs is under 100. A better test is the Mann-Whitney U test which is applicable for a wider category of distributions.
- platz 10y agoCentral limit theorem means there are lots of cases where normal distributions are directly applicable.
- ekianjo 10y agoThe central limit theorem applies to independent variables only. If you are not sure your variables are independent you cannot rely on that assumption.
- platz 10y agoGood to know!
- ronald_raygun 10y agoThat's not 100% true. There are lots of different theorems that are "central limit theorems", and that work across different cases. You can have CLT's with non-iid variables (either the aren't identically distributed or aren't independent). The math just becomes much harder, and you have to assume specific dependence structures. For example https://en.wikipedia.org/wiki/Martingale_central_limit_theorem https://en.wikipedia.org/wiki/Martingale_central_limit_theor...
- ekianjo 10y agoThat's a good article but pretty short. There would be a lot more ground to cover.
- skybrian 10y agoI'm not a statistician but even so I think this article makes assumptions that may not hold up for computer science. The first thing to do is plot your data. If it doesn't look like a bell curve, it's unlikely that common statistical calculations (which assume something close to gaussian) apply here. If you're doing benchmarking, another common model is a peak at a minimum value (when everything goes right) and a long tail, due to various events like cache misses that always slow things down, but don't happen in every test run. On a system with multiple programs running (a typical desktop), taking the mean is meaningless - this just adds noise due to activity unrelated to your program. You'd be better off taking the minimum, which with enough test runs should capture all the events that happen every time and none of the events that don't. The median or 95% percentile might also be useful if you're investigating events that don't happen every time. But if you want to know about cold start performance (for example), maybe the best thing to do would be to flush your caches before every test run, so the events you're interested in are events that happen every time.
- srean 10y ago> If it doesn't look like a bell curve, it's unlikely that common statistical calculations (which assume something close to gaussian) apply here. The key word in there is common. There is an entire industry of statistical techniques that do not require Gaussian assumption or for that matter any parametric assumption. I strongly feel it is time to retire the Gaussian distribution from the space it occupies. Discovering and studying Gaussian distribution and the bog standard central limit theorem should be considered one of mankind's crowning achievements. They deserve to be put on a pedestal to appreciate their elegance, but when rubber meets the road one has to open ones mind to look beyond. Appearance of the Gaussian distribution is rarely as normal as many expect/claim it to be (I blame the stats education machinery for this), nor was it invented by Gauss. In fact Gauss used it as a post-hoc justification for backing the least-squares method. His original motivation for least squares was simplicity and convenience, not the normal distribution or CLT or for that matter the Gauss-Markov theorem.
- thanatropism 10y agoThe Gaussian distribution is central in continuous-time models for different reasons: https://almostsure.wordpress.com/2010/04/13/levys-characterization-of-brownian-motion/ https://almostsure.wordpress.com/2010/04/13/levys-characteri... Basically any reasonable* stochastic continuous process is driven by a brownian motion. Also: discontinuous processes are more or less* the sum of a brownian motion and a poisson-type process. https://en.wikipedia.org/wiki/L%C3%A9vy_process#L.C3.A9vy.E2.80.93Khintchine_representation https://en.wikipedia.org/wiki/L%C3%A9vy_process#L.C3.A9vy.E2... (* Much details about filtrations, Banach spaces yadda yadda omitted)
- petters 10y agoOne common mistake is to take the average of run times (for identical runs). I think taking the minimum should be better under reasonable assumptions. Edit: I now see that this was mentioned elsewhere here. Good!
- fela 10y ago"it’s telling you that there’s at most an alpha chance that the difference arose from random chance. In 95 out of 100 parallel universes, your paper found a difference that actually exists. I’d take that bet." This is wrong. It’s telling you that there’s at most an alpha chance that a difference like that (or more) would have arisen from random chance if the quantities are actually equal. And if the quantities are equal 95 out of 100 parallel universes would not be able to reject the null hypothesis. Is he saying that he would take the xkcd bet[0] on the frequentist side? [0] https://xkcd.com/1132/ https://xkcd.com/1132/
- wodenokoto 10y agoIs there a good resource to learn the underpinnings of P values and T-tests? I feel like everybody says these are important, show a formula and then arguments ensue about what p=0.95 means, and nobody seems to know this.
- imh 10y agoI think any intro stats book should do the trick. As far as I know, the material in a first stats course is pretty homogeneous. I'm not a biostatistician, but I happen to like this book [0] for introductory stuff. Amazon says you can get it used for $26. [0] https://www.amazon.com/Principles-Biostatistics-CD-ROM-Marcello-Pagano/dp/0534229026 https://www.amazon.com/Principles-Biostatistics-CD-ROM-Marce...
- wodenokoto 10y agoI took intro to stats at a business school and switched to computational linguistics and honestly, I have only been met with the "It is something that you do" in regards to P-values. Ill try and look at an introductory book again and see if it satisfies my curiosity.
- imh 10y agoThe general motivation for a p value is that you can model what your data should look like under the assumption that your model is correct, but you can't really say what your data should look like under the assumption that your model in incorrect. There are just too many ways that it could be incorrect. As a concrete example, I might ask you for the distribution of the mean of N samples given that they come from the standard normal distribution (mean zero, variance 1). That's easy. The sample mean, which is itself a random variable, also is normally distributed with a mean of zero and a variance of 1/N. On the other hand, if I ask you about the mean, but the only info you have is that your data isn't from a standard normal, then it could be anything! There's no objective way to say how the sample mean is distributed, given that one crappy piece of info. The most basic thing you can do then, is to assume that your model is true and see if your data is plausible. If I have a hypothesis that I'm flipping a fair coin and I get all heads on 10 flips, I'm going to start doubting my hypothesis. The probability of all heads or all tails with a fair coin is only 1/512=0.002. P values formalize that notion. We call the hypothesis we can model our "null hypothesis", and see if we get data that makes sense with it. If your observations are some of the most unlikely ones according to your null model, let's start doubting the model. That's it. The benefit and trouble are both that we dodged the entire question of what an alternative to our model could be, and how the data looks under those alternatives. Ignoring that incredibly important question can give rise to a weird way of thinking, and opens the door to some conceptually mind bending mistakes, but it all comes from a simple interpretation of a p value. How unlikely is your data given your null hypothesis (given the model you're trying to test). Formally, this tends to be "what is the probability that some statistic is this unlikely or worse." Does that make any sense?
- glangdale 10y agoThe idea that you should 'plot the error bars' ahead of, well, looking at the data seems a bit premature. As many other comments have stated, looking at the data first is critical. It drives me up the wall: we have 1200dpi printers, retina displays, and so on, and yet somehow people feel the need to collapse everything they've done to these giant finger-painting quality bar charts. Statistical tests are well and good, but I'm amazed at the extent to which smart people will happily plug data which they have never actually seen into statistical metrics. So a mean might be derived from 9 reasonable results and a howlingly off factor-of-2 outlier, and you can dutifully plug this series into a bunch of standard tests and speak confidently about p-values.