3 ms·
It is great to understand this stuff. But people don't. And I am personally of the belief that p-values are popular exactly because they are so easy to misund
by btilly 14y ago
It is great to understand this stuff. But people don't. And I am personally of the belief that p-values are popular exactly because they are so easy to misunderstand as the answer to the question that we want to ask (what is the probability that we are right).
That said, if you need to use p-values in A/B testing, you might want to read http://elem.com/~btilly/ab-testing-multiple-looks/part1-rigorous.html http://elem.com/~btilly/ab-testing-multiple-looks/part1-rigo... to get a procedure that gives always valid p-values, and then http://elem.com/~btilly/ab-testing-multiple-looks/part2-limited-data.html http://elem.com/~btilly/ab-testing-multiple-looks/part2-limi... for a more practical alternative and the caveats. (I still intend to return to the series, but not for a bit more.)
- msellout 14y agoNo such thing as an always-valid p-value. Observations that were predictive yesterday may have no value tomorrow.
- btilly 14y agoThe usefulness of a statistical answer always, of course, depends on the validity of the statistical model used to generate said statistics. But that particular statistical model allows unlimited looks at the data, and unlimited stopping opportunities, without ruining the p-value.