3 ms·

With this multivariate testing, how many visits does it take to get statistically significant data?

• mikkel 13y ago
  Calculating the number of tests needed to reach statistical certainty is not straightforward at all. The equation for simple A/B tests relies on the change of conversion rate from a control. More info here: https://developer.amazon.com/sdk/ab-testing/reference/ab-math.html https://developer.amazon.com/sdk/ab-testing/reference/ab-mat... That's for simple A/B tests - this is not simple A/B tests, instead it is setting up a search space of possible variations of a page and comparing many to many to find which is best. Splitting the data based on visitor referral, country, and more make the sample size needed even larger. TLDR: This solution needs a decent amount of traffic to have reasonable results. The exact numbers are not definite. Source: I co-created xander.io which is basically this concept but more open source.
  • adityar 13y ago
    that does look interesting.
• adityar 13y ago
  Depends on the tests. However a point to note is that this is not classical testing followed by hard-coded values. It's more like a multi-armed bandit-type optimization.
  • wfunction 13y ago
    Yeah, it sounds like the perfect kind of Q-learning problem.