3 ms·

Essentially the article says (paraphrased) "Wow, look, if you A/B test two versions of the exact same design, you get slightly different conversion rates. Can s

by harrybr 16y ago

Essentially the article says (paraphrased) "Wow, look, if you A/B test two versions of the exact same design, you get slightly different conversion rates. Can split testing be trusted?" I don't understand why they find this surprising. Of course there's going to be some variation in the conversion rates. This is the reason why GWO reports statistical significance.

sokoloff 16y ago
While I don't disagree with your main point, there is another question just under the surface: Is the method by which you're distributing sessions into your tests somehow biased? I'm going to call A' the "new" (but identical) test, and A the original test. If you've been running A for a long time, and now add A', what is the chance that the visitor populations between A and A' will be different enough to drive statisically significant differences that are population-related rather than test-content-related? Put slightly differently: If your returning session conversion rate is higher than your first session conversion rate, you will need to take some pains to ensure that each of the tests is getting a fair shake at the traffic. In many cases, that means ending test A, and creating a new test a and a' such that neither a nor a' has an advantage. It's easy to ASSume that there is no meaningful test bias, while the reality is that it's quite easy to have test bias creep in.
- hartror 16y ago
  Another common bias is if your are running multiple split tests in parallel you need to assign each visitor to an option in each split test at RANDOM. If you don't you'll end up with 50% of the users going though all the same 50% of the options and the other 50% of users going through the other 50% of the options.
edanm 16y ago
sokoloff makes a great point. Something I learned from years as a QA lead: always ask that every test you do has a failure and a success case. Surprisingly often, the tests that show your program works fine aren't actually doing their job properly, so they also need to be checked. In other words, testing your implementation of A/B testing is a Good Thing. Even if you don't understand that statistics behind it, at least you can see the same variance occurs in A/A testing, which tells you to go read up some more on what you're doing wrong.

by harrybr 16y ago

sokoloff 16y ago

While I don't disagree with your main point, there is another question just under the surface: Is the method by which you're distributing sessions into your tests somehow biased? I'm going to call A' the "new" (but identical) test, and A the original test. If you've been running A for a long time, and now add A', what is the chance that the visitor populations between A and A' will be different enough to drive statisically significant differences that are population-related rather than test-content-related? Put slightly differently: If your returning session conversion rate is higher than your first session conversion rate, you will need to take some pains to ensure that each of the tests is getting a fair shake at the traffic. In many cases, that means ending test A, and creating a new test a and a' such that neither a nor a' has an advantage. It's easy to ASSume that there is no meaningful test bias, while the reality is that it's quite easy to have test bias creep in.

hartror 16y ago
Another common bias is if your are running multiple split tests in parallel you need to assign each visitor to an option in each split test at RANDOM. If you don't you'll end up with 50% of the users going though all the same 50% of the options and the other 50% of users going through the other 50% of the options.

edanm 16y ago

sokoloff makes a great point. Something I learned from years as a QA lead: always ask that every test you do has a failure and a success case. Surprisingly often, the tests that show your program works fine aren't actually doing their job properly, so they also need to be checked. In other words, testing your implementation of A/B testing is a Good Thing. Even if you don't understand that statistics behind it, at least you can see the same variance occurs in A/A testing, which tells you to go read up some more on what you're doing wrong.