3 ms·

statistical significance is meaningless when split testing websites (you are guaranteed to archive significance if you run the test long enough Wait, w

by cbr 11y ago

statistical significance is meaningless when split testing websites (you are guaranteed to archive significance if you run the test long enough Wait, what? Not if you use the right test to evaluate your change. This article isn't showing that A/B testing is bunk, it's showing that Optimizely is using an inappropriate test. Do an a/b test with identical versions of you site. Spoiler: With 100% certainty, you'll see both significant results and a sizable effect size difference. The whole point of significance testing is that, done properly, it tells you how likely you would be to get these results by chance if the two sides of the test were actually identical. Running your A vs A experiment and evaluating it correctly you'll find results significant at the p < N rate N% of the time. And the longer you run it the lower your estimate of the effect size will be.

bildung 11y ago
I think we agree. I should have elaborated more. > Wait, what? Not if you use the right test to evaluate your change. Sure, but how often to people actually choose the right combination of test, testing tool and estimated effect size for their significance to have actual meaning? The problem with experiments like your typical website split test is the potentially infinite population. With a big enough N, the probability of reaching significance approaches 1. > The whole point of significance testing is that, done properly, it tells you how likely you would be to get these results by chance if the two sides of the test were actually identical. Yes, I know. But the magic lies in that "done properly". Have you ever approached a tool in the field that would fail my proposed test? I haven't.
- cbr 11y ago
  With a big enough N, the probability of reaching significance approaches 1. Well, not with an A vs A experiment. Then the probability of significance at the p < 0.05 level should stay 5%. On the other hand, in an A vs B experiment where B is very slightly different from A you're right that with a large enough population you become very likely to reach a p < 0.05 significance level, just with an extremely tiny effect size. The experiment has told you something about the difference between A and B, and you now have high certainty, but it's just such a small difference that you probably don't care. Have you ever approached a tool in the field that would fail my proposed test? Your proposed test being, run an A vs A experiment and a tool is faulty if it concludes there's a difference significant at the 0.05 level more than 5% of the time? All the tools I've used for A/B testing would do fine on this test. This includes a custom tool at a computational advertising startup I worked at, Google's internal experiment platform, and Google Analytics' Content Experiments. (If I didn't trust a tool to pass this test I'd just use it to get the counts for both sides of the experiment and do that stats on my own.)

by cbr 11y ago

bildung 11y ago

I think we agree. I should have elaborated more. > Wait, what? Not if you use the right test to evaluate your change. Sure, but how often to people actually choose the right combination of test, testing tool and estimated effect size for their significance to have actual meaning? The problem with experiments like your typical website split test is the potentially infinite population. With a big enough N, the probability of reaching significance approaches 1. > The whole point of significance testing is that, done properly, it tells you how likely you would be to get these results by chance if the two sides of the test were actually identical. Yes, I know. But the magic lies in that "done properly". Have you ever approached a tool in the field that would fail my proposed test? I haven't.

cbr 11y ago
With a big enough N, the probability of reaching significance approaches 1. Well, not with an A vs A experiment. Then the probability of significance at the p < 0.05 level should stay 5%. On the other hand, in an A vs B experiment where B is very slightly different from A you're right that with a large enough population you become very likely to reach a p < 0.05 significance level, just with an extremely tiny effect size. The experiment has told you something about the difference between A and B, and you now have high certainty, but it's just such a small difference that you probably don't care. Have you ever approached a tool in the field that would fail my proposed test? Your proposed test being, run an A vs A experiment and a tool is faulty if it concludes there's a difference significant at the 0.05 level more than 5% of the time? All the tools I've used for A/B testing would do fine on this test. This includes a custom tool at a computational advertising startup I worked at, Google's internal experiment platform, and Google Analytics' Content Experiments. (If I didn't trust a tool to pass this test I'd just use it to get the counts for both sides of the experiment and do that stats on my own.)