7 ms·

@darrenix, I see that your sample sizes are pretty low, and your actual conversions are incredibly low. Are you controlling for statistical significance, and ar

@darrenix, I see that your sample sizes are pretty low, and your actual conversions are incredibly low. Are you controlling for statistical significance, and are you accounting for repeated significance testing errors? [1] Genuinely curious, not trying to shoot down what looks like an interesting approach. [1] http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html

RA_Fisher 13y ago
It's possible to work with these sample sizes using survival analysis, but I doubt Bayes would find the difference even with a really well initialized prior. I don't think that traditional significance testing is the answer here, those require even larger sample sizes (b/c you're not limiting the space of the estimate). It's all about limiting the domain-space. You know conversion rates are never over 10% on your homepage, why would you not exclude that to give your statistical work more power? That's at least how I think about it.
- jdlshore 13y ago
  It sounds like your stat-fu is way better than mine. :-) Is there a layman-friendly article you can point to that explains what you're talking about in more detail?
- aj37 13y ago
  Can you elaborate? I'm not familiar with survival analysis for AB testing
darrennix 13y ago
The actual conversion numbers are higher because, as I noted in the article, I'm only reporting number for web forms in order to keep things simple. There's also email and phone inquiries to count so you can basically multiply by 3x. As for sample size, we're vigilant to make sure that we reach statistical significance. Typically our tests will encompass about 2,000 data points per group. Our job is made easier by the fact that we're looking for huge wins so when we see a 10% gain with a 99% confidence, that's still a "fail" scenario for us.

RA_Fisher 13y ago

It's possible to work with these sample sizes using survival analysis, but I doubt Bayes would find the difference even with a really well initialized prior. I don't think that traditional significance testing is the answer here, those require even larger sample sizes (b/c you're not limiting the space of the estimate). It's all about limiting the domain-space. You know conversion rates are never over 10% on your homepage, why would you not exclude that to give your statistical work more power? That's at least how I think about it.

jdlshore 13y ago
It sounds like your stat-fu is way better than mine. :-) Is there a layman-friendly article you can point to that explains what you're talking about in more detail?
aj37 13y ago
Can you elaborate? I'm not familiar with survival analysis for AB testing

darrennix 13y ago

The actual conversion numbers are higher because, as I noted in the article, I'm only reporting number for web forms in order to keep things simple. There's also email and phone inquiries to count so you can basically multiply by 3x. As for sample size, we're vigilant to make sure that we reach statistical significance. Typically our tests will encompass about 2,000 data points per group. Our job is made easier by the fact that we're looking for huge wins so when we see a 10% gain with a 99% confidence, that's still a "fail" scenario for us.