5 ms·

In order to be useful, you probably want to see AB tests reaching a conclusion in under 30 days. I'd say for a conversion rate goal, this is going to be when yo

In order to be useful, you probably want to see AB tests reaching a conclusion in under 30 days. I'd say for a conversion rate goal, this is going to be when you have around 100k visitors a month. There are a few variables to consider - What goal would you like to AB test? Conversion rate is an end-of-funnel goal that needs a lot of traffic, you can use upper funnel goals like product views, add to bag etc to get quicker conclusions (not as accurate but often a good approximation) - The stats engine/AB testing tool you are using. More simple tools might conclude quicker but in my experience they can be so inaccurate they are counter productive. Usually a long time to conclude = reliable results. I've never used Google Optimize so I'm not sure where it stands. - How many people are being exposed to the AB test, for example is it all web traffic or just mobile? - How much of an affect the AB test has on behavior. A button color/text change will normally take long to conclude than a feature that's really helping your users. - How confident do you want to be before reaching a conclusion? I'd recommend looking for 95% confidence in uplift before concluding an AB test.

citrablue 8y ago
Google Optimize offers MAB testing, which is awesome. Personally, I don't see it as a "when do results become reliable" problem -- it's a "what is the cost of a false positive, false negative, or 'do nothing' decision?"
- roberttod 8y ago
  Multi Armed bandit tests need even more traffic, as there are more variations being tested. I think you have to be careful with false positives in AB testing - drawing conclusions too fast can nullify it's usefulness.
  - citrablue 8y ago
    The different between "traditional" A/B testing and Multi-Armed Bandit has nothing to do with the number of treatments. A/B is really shorthand for A/B/n. In an A/B test, the probability of selecting an "arm" (a treatment to show to the visitor) is equally distributed. Google Optimizer differs by adjusting the traffic distribution, sending more traffic to better performing options, what's usually called MAB approach. The difference is that MABs will more quickly converge on the "winning" variation, but are more likely to get stuck in a local minimum. EG, if an "worse" option performs better right off the bat, the MAB might send most traffic to it. This would take a while for the algorithm to "recover". The major advantages of the MAB approach are minimizing opportunity costs and the ability to capture seasonality because you can continuously run tests. Traditional split testing runs for a while, gets a result, and moves on. With a MAB, you can assign an "explore" budget that keeps tests running in the background to capture the seasonal/periodic change in conversions. That comes with a cost: every visitor who doesn't see your best page is lost revenue.

citrablue 8y ago

Google Optimize offers MAB testing, which is awesome. Personally, I don't see it as a "when do results become reliable" problem -- it's a "what is the cost of a false positive, false negative, or 'do nothing' decision?"

roberttod 8y ago
Multi Armed bandit tests need even more traffic, as there are more variations being tested. I think you have to be careful with false positives in AB testing - drawing conclusions too fast can nullify it's usefulness.
- citrablue 8y ago
  The different between "traditional" A/B testing and Multi-Armed Bandit has nothing to do with the number of treatments. A/B is really shorthand for A/B/n. In an A/B test, the probability of selecting an "arm" (a treatment to show to the visitor) is equally distributed. Google Optimizer differs by adjusting the traffic distribution, sending more traffic to better performing options, what's usually called MAB approach. The difference is that MABs will more quickly converge on the "winning" variation, but are more likely to get stuck in a local minimum. EG, if an "worse" option performs better right off the bat, the MAB might send most traffic to it. This would take a while for the algorithm to "recover". The major advantages of the MAB approach are minimizing opportunity costs and the ability to capture seasonality because you can continuously run tests. Traditional split testing runs for a while, gets a result, and moves on. With a MAB, you can assign an "explore" budget that keeps tests running in the background to capture the seasonal/periodic change in conversions. That comes with a cost: every visitor who doesn't see your best page is lost revenue.