5 ms·
I agree as well! Here is what probability theory teaches us. The proper role of data is to adjust our prior beliefs about probabilities to posterior beliefs t
by btilly 11y ago
I agree as well!
Here is what probability theory teaches us. The proper role of data is to adjust our prior beliefs about probabilities to posterior beliefs through Bayes' theorem. The challenge is how to best communicate this result to people who may have had a wide range of prior beliefs.
p-values capture a degree of surprise in the result. Naively, a surprising result should catch our attention and cause us to rethink things. This is not a valid statistical procedure, but it IS how we naively think. And the substitution of a complex question for a simpler one is exactly how our brains are set up to handle complex questions about our environment. (I'm currently working through Thinking Fast and Slow which has a lot to say about this.)
Simple Bayesian approaches take the opposite approach. You generally start with some relatively naive prior, and then treat the posterior as being the conclusion. Which is not very realistic if the real prior was something quite different.
Both approaches have a fundamental mistake. The mistake is that we are taking a data set and asking what it TELLS us about the world. When in probability theory the real role of data is how to UPDATE our views about the world.
This is why I have come to believe that for simple A/B testing, thinking about p-values is a mistake. The only three pieces of information that you need is how much data you are willing to collect, how much you have collected, and how big the performance difference is. Stop either when you have hit the maximum amount of data you're willing to throw at the test, or when the difference exceeds the square root of that maximum amount. This is about as good as any simple rule can do.
If you try to be clever with p-values you will generally wind up putting yourself at saving yourself some effort in return for a small risk per test of making very bad mistakes. Accepting small risk per test over many tests for a long time puts you at high odds of eventually making a catastrophic mistake. This is a very bad tradeoff.
I've personally seen a bad A/B test with a low p-value rolled out that produced a 15% loss in business for a company whose revenues were in the tens of millions annually. It. Was. Not. Pretty. (The problem was eventually found and fixed..a year later and after considerable turnover among the executive team.)
- danenania 11y agoThanks for the very interesting thoughts. Could you explain a bit how your rule of thumb works and why it's better than p-values? Why is the difference vs. the square root of the max available sample size a meaningful measure?
- btilly 11y agoThe idea is that you will decide when you've either expended as much energy as you are willing to, or when you're convinced that you won't make a different decision. There is a simple symmetry argument that shows that the odds of a random walk reaching sqrt(N) in one direction and then getting to the opposite direction by N observations is exactly the same as the odds of a random walk reaching 2 sqrt(N) by the time you reach N observations. So the sqrt(N) rule is a simplified version of, "Call off the test when you've reached 95% certainty that we have our final answer." You can easily make a million minor improvements. But I can't come up with a simpler rule meeting the business needs which is less likely to lead to dangerous misunderstandings. "Decide when you pass this threshold, or when you've put out as much effort as you're willing to." And the effort required to testing are easily understood as well. "The cost of a bad A/B test is at most 100 lost sales and remembering it is there for 2 months." (This would be for an organization that expects 10,000 sales in 2 months, and is willing to leave tests up for that long.) And if memory serves me (I'd have to redo the calculation) the rule of thumb in this case is that wins below 1% are called randomly, above 2% are usually called right, you're confident of getting the right answer above 5%, and even over many tests you are confident of never making a 10% mistake. That last rule of thumb is important. Small organizations can't benefit from A/B tests. Large ones are stupid not to use it heavily. And very few people have a good sense of whether it makes sense for them.
- danenania 11y agoThanks! That's very helpful.
- AstralStorm 11y agoDid you know that your rule of "do the tests until significance is reached" denies the use of chain probability and most tests based on it? You can show that in the limit, your rule will always reach any given threshold just by pure luck. You really need to set the number of trials or samples before performing them. Here's a more in depth explanation: http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html
- zamalek 11y agoThe way I see p-values is a mathematical way to share how certain you are that something is in the realm of reality (not truth). It is not a formal conclusion of the results, and is nothing more than a "spoiler" indicating what you might also conclude.
- btilly 11y agoThis point of view is not supported by the mathematics. If you think otherwise, then you do not understand the mathematics.
- yummyfajitas 11y agoSimple Bayesian approaches take the opposite approach. You generally start with some relatively naive prior, and then treat the posterior as being the conclusion. Which is not very realistic if the real prior was something quite different. I don't think this is a completely accurate portrayal of Bayesian stats. In Bayesian stats, there is no "real prior". Probability distributions are all subjective representations of belief. The prior is just what you believe prior to evidence, and the posterior is what you believe after you've taken evidence into account. That said, moving away from p-values and towards something more robust is something the A/B testing industry needs. (Obviously I have my own opinion of what that something should be, and it's a bit different from what you are advocating.) There are far too many consultancies and agencies p-hacking their way to positive results ("hey unsophisticated client - guess what I made your conversion rate go up 25%!") and I'd love to see every one of them die.
- btilly 11y agoThere are different approaches a Bayesian might take. The one that I described is certainly among them, though it is not the only one.
- achompas 11y agoI think the word "naive" is problematic here. Have you seen instances where Bayesians choose a prior that isn't at least somewhat informed by exploratory analysis?