2 ms·
My stats knowledge is fairly rusty, so I got a bit lost midway through, but here's a side question I've been wondering about, especially given that the linked M
by majormajor 14y ago
My stats knowledge is fairly rusty, so I got a bit lost midway through, but here's a side question I've been wondering about, especially given that the linked Miller article talks about the null hypothesis being that the two are equal: how does that null hypothesis fit into these tests? I.e., you have to choose a button, so does it even make sense when the hypothetical employee in that example at the start of the article says "we didn't get an answer"? A .18 p-value isn't great, but I imagine he'd still recommend the green one -- there's much less reason to think the other button was better.
Why not do a one-sided test where the null hypothesis is "A is at least as good as B" and the test hypothesis is "B is better than A"? It seems like you'd be gaining some more power to detect B being better, without losing much since A's safe to choose regardless of if they're about the same or if A's actually better?
- btilly 14y agoThe reason is that statistics is done with numbers. I do not know how to take a statement like "A is at least as good as B" and tell you the probability that after 15 coin flips, A is 3 ahead of B. I do know how to do that calculation under the assumption that they are exactly equal. Or under the assumption that A comes up 51% of the time, B 49% of the time. The point of the null hypothesis is that it is an absolute worst case. So whatever the real difference, it can't be harder to detect than the null hypothesis. Therefore if we make very few mistakes on the null hypothesis, then we're confident that we'll make very few mistakes, no matter what. And finally you're right that going with the green button is better than just tossing up your hands. But how to decide how much better is a surprisingly complicated question, and there is no simple answer about how to quantify it. (Remember, statistics has to be done with numbers, so quantifying the answer matters.)
- majormajor 14y agoWouldn't it just be a normal one-sided hypothesis test? The only thing that changes under H_0: A > B vs H_0: A = B is that you're only interested in the area under one side of the curve instead of both. The test statistic remains the same, you just get a different p value. It should let you get away with a lower sample size at the expense of possibly only being able to conclude "A's better" or "A's not better" instead of "A's better" or "A and B could be the same" or "B's better." (It's unclear to me what would be the problem with looking at the P values for both the > test and the != test, for a single test, and only "falling back" to the > test if you happen to be in a range where you can say "we can't say for sure that B's better, but we can be pretty sure that A isn't better," like how the different P values are presented here[1].) It still wouldn't be valid to do something like "oh the two-sided one was inconclusive but leaned in favor of A, so let's do a one-sided one to check if A's better after all," though. [1] http://www.ats.ucla.edu/stat/mult_pkg/faq/general/tail_tests.htm http://www.ats.ucla.edu/stat/mult_pkg/faq/general/tail_tests...
- btilly 14y agoIn this article I was going for the highest possible standard. The ins and outs of one-sided versus two-sided was not a topic that I wanted to explore. In the next article I plan to relax things a lot farther than just one-sided versus two-sided tests!
- majormajor 14y agoGotcha, thanks. And one- vs two-sided alone would hardly be a huge win, I was just curious since I didn't remember seeing it brought up at all in any of the prior articles I'd read.