4 ms·
Meh, you make it sound too hard. Here is a perfectly adequate A/B testing regime. Assign visitors into buckets randomly. Track conversion counts only. Decla
by btilly 10y ago
Meh, you make it sound too hard. Here is a perfectly adequate A/B testing regime.
Assign visitors into buckets randomly. Track conversion counts only. Declare the winner of an A/B test whenever one gets 100 conversions ahead, or whichever is ahead after 10,000 conversions. Do this with every test. As long as there are no obvious interaction effects, you can run multiple tests in parallel.
If you wish, you can replace 10,000 with N where N is however many conversions you expect to get in a month. Replace 100 by the square root of N. Real differences below 2/sqrt(N) are basically coin flips. Even after many tests, you are very unlikely to have ever made an error as big as 5/sqrt(N).
If you want to do something more sophisticated than this, you need competent people. But this is worlds better than what most companies do. (And yes, it is better than what Optimizely does for you by default.)
- ken47 10y agoCan you explain the statistics behind this methodology?
- btilly 10y agoEssentially the same as http://www.evanmiller.org/sequential-ab-testing.html http://www.evanmiller.org/sequential-ab-testing.html except that he aims to not produce an answer most of the time by chance, while I aim to always produce an answer even if it is statistically a coin flip. (Because, after all, you need to actually choose a headline in the end...)
- robbiemitchell 10y agoYou're describing the math of A/B testing, but not the work of analyzing data to decide what experiment to do next, designing the experiment, building and integrating it, then assigning visitors to the buckets and so forth.
- btilly 10y agoThe technical end of building / integrating / assigning visitors and so on tends to be relatively simple if it is done by competent people who know how to not overcomplicate things. Coming up with ideas to test tends to be fairly easy if you've got a competent product person. Perhaps I'm biased. I've built multiple A/B testing systems for multiple companies, and know what to do. However it didn't seem hard for me the first time either.
- in_cahoots 10y agoI appreciate what you're trying to say here, but you've basically just described the prototypical way not to run an A/B test. Ending a test as soon as you reach significance ensures a high rate of false positives, and doesn't clearly tell you when to end the experiment if there is no effect. See also: http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html
- btilly 10y agoYou have just pattern-matched to a response without understanding. And you're wrong. See http://www.evanmiller.org/sequential-ab-testing.html http://www.evanmiller.org/sequential-ab-testing.html for a very similar methodology presented by the very person you are citing. The difference is that I am aiming to always produce an answer from the test, even if it is a statistical coin flip, while he aims to only produce an answer roughly 5% of the time if there is no difference.
- in_cahoots 10y agoMaybe I am not understanding your post, but aren't you just declaring a winner after N trials, even if that is not significant? That seems to be a critical distinction here.
- btilly 10y agoRight. We are trying to make a business decision, not conduct science. If you go with whichever is ahead, you'll reliably produce correct answers for wins at that you can't reliably measure. Yes, there will be mistakes, but the mistakes are guaranteed to be pretty small. If you insist on a higher standard, you'll learn which choices you were confident of..but now need to figure out what to do when the statistics didn't give you a final answer. I think that the first choice is more useful. Evan prefers to clearly distinguish your coin flips from solid results. But in the end you have to make a decision, and it isn't material what decision you make if the conversion rates are close.
- 10y ago
- deleted 10y ago[deleted]
- dhbradshaw 10y agoThis seems like a nice application of two related principles: 1) The harder it is to tell which choice is better, the less it matters which you choose, and 2) a decision has value even in the absence of proof that it's the very best decision because it frees up resources for the next decision, which may be more important.
- cle 10y agoDepending on the implementation of "assign visitors into buckets randomly", this should include a test to ensure the populations are correctly randomized. "Unsophisticated" users of these heuristics are prone to make terrible decisions if the treatment populations are not properly randomized (this can be due to systemic bias caused by e.g. caching, or treatment effects like survival bias).