4 ms·

Thanks for the very interesting thoughts. Could you explain a bit how your rule of thumb works and why it's better than p-values? Why is the difference vs. the

Thanks for the very interesting thoughts. Could you explain a bit how your rule of thumb works and why it's better than p-values? Why is the difference vs. the square root of the max available sample size a meaningful measure?

btilly 11y ago
The idea is that you will decide when you've either expended as much energy as you are willing to, or when you're convinced that you won't make a different decision. There is a simple symmetry argument that shows that the odds of a random walk reaching sqrt(N) in one direction and then getting to the opposite direction by N observations is exactly the same as the odds of a random walk reaching 2 sqrt(N) by the time you reach N observations. So the sqrt(N) rule is a simplified version of, "Call off the test when you've reached 95% certainty that we have our final answer." You can easily make a million minor improvements. But I can't come up with a simpler rule meeting the business needs which is less likely to lead to dangerous misunderstandings. "Decide when you pass this threshold, or when you've put out as much effort as you're willing to." And the effort required to testing are easily understood as well. "The cost of a bad A/B test is at most 100 lost sales and remembering it is there for 2 months." (This would be for an organization that expects 10,000 sales in 2 months, and is willing to leave tests up for that long.) And if memory serves me (I'd have to redo the calculation) the rule of thumb in this case is that wins below 1% are called randomly, above 2% are usually called right, you're confident of getting the right answer above 5%, and even over many tests you are confident of never making a 10% mistake. That last rule of thumb is important. Small organizations can't benefit from A/B tests. Large ones are stupid not to use it heavily. And very few people have a good sense of whether it makes sense for them.
- danenania 11y ago
  Thanks! That's very helpful.
- AstralStorm 11y ago
  Did you know that your rule of "do the tests until significance is reached" denies the use of chain probability and most tests based on it? You can show that in the limit, your rule will always reach any given threshold just by pure luck. You really need to set the number of trials or samples before performing them. Here's a more in depth explanation: http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html
  - btilly 11y ago
    The procedure I suggested starts with setting the maximum number of conversions before beginning the test. It is emphatically not the procedure that Evan Miller criticizes. In fact see http://www.evanmiller.org/sequential-ab-testing.html http://www.evanmiller.org/sequential-ab-testing.html for Evan Miller suggesting a similar procedure to mine, based on past conversations that I have had with him. The difference between that one and mine is that I focus on "make the best decision available, even though it may be only little better than a coin flip" while he tries to only decide when you can decide with confidence. In past conversations we have agreed that each other's procedures are reasonable, but they achieve different goals.

btilly 11y ago

The idea is that you will decide when you've either expended as much energy as you are willing to, or when you're convinced that you won't make a different decision. There is a simple symmetry argument that shows that the odds of a random walk reaching sqrt(N) in one direction and then getting to the opposite direction by N observations is exactly the same as the odds of a random walk reaching 2 sqrt(N) by the time you reach N observations. So the sqrt(N) rule is a simplified version of, "Call off the test when you've reached 95% certainty that we have our final answer." You can easily make a million minor improvements. But I can't come up with a simpler rule meeting the business needs which is less likely to lead to dangerous misunderstandings. "Decide when you pass this threshold, or when you've put out as much effort as you're willing to." And the effort required to testing are easily understood as well. "The cost of a bad A/B test is at most 100 lost sales and remembering it is there for 2 months." (This would be for an organization that expects 10,000 sales in 2 months, and is willing to leave tests up for that long.) And if memory serves me (I'd have to redo the calculation) the rule of thumb in this case is that wins below 1% are called randomly, above 2% are usually called right, you're confident of getting the right answer above 5%, and even over many tests you are confident of never making a 10% mistake. That last rule of thumb is important. Small organizations can't benefit from A/B tests. Large ones are stupid not to use it heavily. And very few people have a good sense of whether it makes sense for them.

danenania 11y ago
Thanks! That's very helpful.
AstralStorm 11y ago
Did you know that your rule of "do the tests until significance is reached" denies the use of chain probability and most tests based on it? You can show that in the limit, your rule will always reach any given threshold just by pure luck. You really need to set the number of trials or samples before performing them. Here's a more in depth explanation: http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html
- btilly 11y ago
  The procedure I suggested starts with setting the maximum number of conversions before beginning the test. It is emphatically not the procedure that Evan Miller criticizes. In fact see http://www.evanmiller.org/sequential-ab-testing.html http://www.evanmiller.org/sequential-ab-testing.html for Evan Miller suggesting a similar procedure to mine, based on past conversations that I have had with him. The difference between that one and mine is that I focus on "make the best decision available, even though it may be only little better than a coin flip" while he tries to only decide when you can decide with confidence. In past conversations we have agreed that each other's procedures are reasonable, but they achieve different goals.