3 ms·
A lot of math detail, but a naive reader reading it will still make the mistake outlined in http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.
by btilly 11y ago
A lot of math detail, but a naive reader reading it will still make the mistake outlined in http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html. And a less naive reader will still be left with no guidance on how to make real-world decisions if you weren't so lucky as to get a strong statistical result.
Here is a much, much simpler strategy that I can suggest for business users.
1. Figure out the longest that you can afford to run your test for.
2. Figure out the number of conversions N you expect in that time.
3. Start running your test, randomly splitting visitors into A and B.
4. Stop the test as soon as one version has sqrt(N) more conversions than the other. Else wait until you get to N conversions between them, and go with what is ahead.
Here are some comments on the procedure.
Stopping before N/2 total conversions is roughly the same certainty as stopping at 95% confidence. Stopping after that is an admission that you are crossing your fingers and going with an educated guess, and it is not feasible to collect enough data to get a better answer.
If one version has a conversion rate which is (1 + 2/sqrt(n)) better than the other, you pretty reliably choose right. The flip side of this is that you'll make a lot of mistakes if you get much below that threshold. If those potential errors are too big, then A/B testing is not going to work well for you because you don't have enough data for statistics.
- danieltillett 11y agobtilly where does this approach come from?
- btilly 11y agoMy scratchbook. :-) I keep meaning to write it up. Mathematically it is similar http://www.evanmiller.org/sequential-ab-testing.html http://www.evanmiller.org/sequential-ab-testing.html but with the difference that Evan produced a stopping rule that is a 95% confidence interval, while mine is stopping at 95% confidence that you won't come to a different conclusion by the end of the test. Otherwise the idea of the analysis is the same. (Note, under the null hypothesis reaching sqrt(N) then crossing 0 is exactly as hard as reading 2 sqrt(N) as can be seen by taking any possible sequence that does one, and swapping the values after first reaching sqrt(N) to get a sequence that does the other.)
- danieltillett 11y agoPlease do write it up as I would certainly be interested in reading it.