20 ms·
This is the insight that made statistics "click" for me many years ago: a statistical test answers one central question: what are the odds that the results you
by lisper 10y ago
This is the insight that made statistics "click" for me many years ago: a statistical test answers one central question: what are the odds that the results you observed could have arisen by chance? If those odds are low, then you are justified in concluding that the results probably did not arise by chance, and so there must be some other explanation (usually, but not always, the causal hypothesis you are advancing).
Once consequence of this is that it is crucial that you advance your hypothesis before you collect (or at least look at) the data because the odds of something arising by chance change depending on whether you predict or postdict the results. Also, the more data you have, the more likely you are to find something in there that looks like a signal but is in fact just a coincidence. Many a day-trading fortune has been lost to this one mistake.
- stdbrouw 10y ago> a statistical test answers one central question: what are the odds that the results you observed could have arisen by chance Well, no, that'd be very interesting but unfortunately what a statistical test really says is the probability of the results you observed (or more extreme) given chance. P(data|model) and not P(model|data).
- lisper 10y agoIt's actually P(data|null-hypothesis).
- wfunction 10y agoThe model is the null hypothesis here... I don't think the parent comment was wrong w.r.t. this.
- lisper 10y agoNo, there's a difference. The model is something like, "This drug attaches itself selectively to cancer cells and kills them." The null hypothesis is, "This drug has no effect." So you conduct a double-blind study, measure the effect of the drug on cancer cells, collect some data and compute that P(data|null-hypothesis) is 1%. It it not the case that there is a 99% chance that your model is correct and that the drug does in fact attach itself to cancer cells and kill them. Because there are other possible models, e.g.: the drug redirects your Chi and channels it into your chakras which kills the cancer cells. Statistics alone cannot tell you which of those two models is correct.
- fluxion 10y agoI think this is just an argument over semantics, the "null hypothesis" is a perfectly valid model using the definition of the statistics community [0] (i.e. a collection of probability distributions over some sample space). [0] https://en.wikipedia.org/wiki/Statistical_model https://en.wikipedia.org/wiki/Statistical_model
- lisper 10y agoThe word "model" is being used here in two mutually incompatible ways. Yes, the null hypothesis is a model, but it is not an explanatory model. A scientific hypothesis has to meet two tests to be considered a valid theory. It has to be consistent with the data, and it has to have explanatory power. The theory that cancer drugs work by aligning a patient's chi with their chakras is rejected not because it is inconsistent with the data (it's not) but because it lacks the explanatory power of alternative theories based on molecular biology. The null hypothesis never has explanatory power. The null hypothesis is always a statement of the form, "The explanatory hypothesis under test is wrong for some unknown reason." This is why rejecting the null hypothesis, i.e. showing that the data are (with high probability) inconsistent with the null hypothesis, is considered a positive result.
- nonbel 10y ago>"Statistics alone cannot tell you which of those two models is correct." Statistics can tell you whether a model is consistent with the data. But you need to deduce the null hypothesis from your model rather than use the default "no difference" (of course, sometimes no difference is deduced from a real model, but not often, in that case: great!). In fact, that is the proper use of statistics. I would guess >99.99% of current usage is incorrect (ie pseudoscience) and amounts to a waste of time at best. The usual usage turns scientific reasoning on its head, and has lead to a (literally for most people) unbelievable amount of trouble. This was pointed out most aptly by Paul Meehl long, long ago: http://www.fisme.science.uu.nl/staff/christianb/downloads/meehl1967.pdf http://www.fisme.science.uu.nl/staff/christianb/downloads/me...
- 10y ago
- qwrusz 10y agoHow about you guys are both right, sort of. There are both Bayesian and Frequentist approaches in statistics! They represent very different methods to statistics, but they are also quite similar. My apologies, I couldn't find one link that gave a good description of Bayesian vs. Frequentist. Here are a couple links to get started: https://xkcd.com/1132/ https://xkcd.com/1132/ http://jakevdp.github.io/blog/2014/03/11/frequentism-and-bayesianism-a-practical-intro/ http://jakevdp.github.io/blog/2014/03/11/frequentism-and-bay... If anyone comes across a link that describes Bayesian and Frequentist clearly, please share it if you don't mind.
- lisper 10y agoThat cartoon is actually a pretty good illustration of why frequentists are wrong.
- qwrusz 10y agoDo you mean probably wrong? Either way I disagree. Both approaches have pros and cons depending on the type of analysis. At the very least, just the presence of competing approaches in the field has pushed statisticians to have more rigour and do way more double checking than they might have, out of fear the other side actively looking to poke holes. It's easy to lie with statistics and the only people who can call statisticians out on their BS is other statisticians...Need more of this.
- lisper 10y ago> Do you mean probably wrong? Nope. > Both approaches have pros and cons What are the pros of the frequentist approach?
- danieltillett 10y agoSimple maths that can be done without a computer. Useful in the past - not so useful today.
- rspeer 10y ago
- stdbrouw 10y agoYup, and the null hypothesis is a reduced model. Different words, same thing.
- lisper 10y agoI have no idea what you mean by "reduced" but they are not the same thing. The difference in their explanatory power. A (scientific) hypothesis explains things. A null hypothesis simply says, "This hypothesis is wrong" but doesn't say why. It's a crucial distinction.
- deleted 10y ago[deleted]
- kgwgk 10y ago> what are the odds that the results you observed could have arisen by chance? > the probability of the results you observed (or more extreme) given chance I think he said the same thing as you. He didn't talk about "the odds that the results you observed have arisen by chance" but the odds that the results you have observed (the or more extreme is implicit) could have arisen by chance (i.e. the probability of happening by chance alone, given that only chance was involved).
- fela 10y agoUnfortunately no. Very much no, even though it's widely believed that that is a good definition/intuition (and used in many places). It's the odds of having that results due to chance, if the null hypothesis is true[0]. That latter part might sound pedantic, but the whole point is that we don't know how likely the null hypothesis is. If I test wheather the sun has just died[1] and get a p-value of 0.01 it's still very likely that this result is due to change (surely more than 1%)! We need a prior probability (i.e. bayesian statistics) to calculate the probability that the result was due to chance, that is why that partial definition is incomplete and actually very misleading. This point is subtle, but very important to really understand p-values. Another way to look at it is: if we knew the probability that the result was due to chance we could also just take 1-p and have to probability of there actually being some effect, a probability that hypothesis testing cannot give us. There is one nice property that hypothesis testing does have (and why presumably it's so widely used): if the idea you are testing is wrong (which actually means "null hypothesis true") you will most likely (1-p) not find any positive results. This is good, this means that if the sun in fact did not die, and use 0.01 as your threshold, 99% of the experiments will conclude that there is no reason to believe the sun has died. So hypothesis testing does limit the number of false positive findings. The xkcd comic is a bit misleading it this regard, yes it does highlight the limitations of frequentist hypothesis testing, but the scenario depicted is a very unlikely one, in 99% of the cases there would have been a boring and reasonable "No, the sun hasn't died". For an incredibly interesting article about the difficulty of concluding anything definitive from scientific results I highly recommend "The Control Group is out of Control" at slatestarcodex[2]. [0] To be even more pedantic you would have to add "equal or more extreme", and "under a given model", but "if the null hypothesis is true" is by far the most important piece often missing. [1] https://xkcd.com/1132/ https://xkcd.com/1132/ [2] http://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ http://slatestarcodex.com/2014/04/28/the-control-group-is-ou...
- lisper 10y ago> It's the odds of having that results due to chance, if the null hypothesis is true[0]. Yes, that's right. I don't know why you think this is at odds with what I said. In fact, I clarified this myself a few hours ago in a sibling comment: https://news.ycombinator.com/item?id=13026907 https://news.ycombinator.com/item?id=13026907