5 ms·

No analysis of AB testing should ever miss this incredibly important fact about statistics. http://xkcd.com/882/ http://xkcd.com/882/ When you make confidence

by RyanZAG 14y ago

No analysis of AB testing should ever miss this incredibly important fact about statistics. http://xkcd.com/882/ http://xkcd.com/882/ When you make confidence tests on a continuous process, you will hit this 1 day in 100 on average for 100 days of testing at 99%. If you're doing multiple AB tests per day, well... EDIT: And another important issue: if you are testing on Valentines day and the pink color/A test is beating out the orange color/B test, then this result may only happen on that one day per year. This applies to hot news topics and numerous other fads as well. Also make sure your tests really are independent - if you do A/B testing off of fixed IP address hashing and have a small user base, by random chance you may have your top customer always being in 'group A' and giving group A a higher than expected base average - an extra bonus click for every group A test may skew your statistical tests as they assume independence.

tel 14y ago
Thus why 95% alpha is really important to notice since you should probably be applying Bonferroni corrections (at least).
thenomad 14y ago
Counter point to your first - do I, as the Practical A/B testing Guy, really care if I get one in 100 tests wrong? If I'm running 100 A/B tests a year, 50% of which produce a 5% improvement in my conversion rate, who cares if I get one of them wrong and drop 5% instead? I'll still have improved my conversions by 1100% over the course of the year. (Obviously, I'm not taking into account local maxima here - but any statistical misstep will only temporarily roll your testing down the hill, and subsequent tests will push back up to the local maximum anyway.)
- btilly 14y ago
  In reality no, because nobody will notice that the company is about 10% smaller than it could have been, and you're doing good stuff. However the cost of that 10% change in your company probably exceeds the cost of running 100 tests. Note that in the real world, after a while you exhaust the easy 5% wins. Then you wind up chasing a lot of 1% and 2% marginal wins, and those tests take a lot longer to run. But even so, if your company is pulling down, say a million per month, a 2% drop in business from getting a 1% test wrong probably exceeds the costs of all of the tests you run per year. So unless extending tests is prohibitively expensive, or there are significant opportunity tests from not being able to run the tests you want, you should go to higher confidence.
  - thenomad 14y ago
    Sorry, I'm not quite following you here. In my experience, the primary limiter of how many A/B tests you can run is available traffic. Let's assume that I'm running the 2% tests you mention. I can either run 60 2% tests at 95% confidence, or (approximately) 30 2% tests at 99.9% confidence, if the math posted elsewhere in this thread is right, over a year, say - that's how much traffic I have. Let's once again assume a third of those pan out. I'm still not seeing why I'd prefer 10 2% wins (from the 99.9% approach) to 20 2% wins plus 3 2% losses (from the 95% approach). Yes, there are more errors, but overall I end up with a 40% improvement as opposed to a 24% improvement. Can you explain?
    - btilly 14y ago
      You can run multiple A/B tests in parallel, as long as the tests themselves do not interact. And coming up with and implementing likely ideas is a challenge. Now do you understand?
      - thenomad 14y ago
        On the first point - I may be being dense here. Do you mean a) segmenting traffic, and running A vs B on one segment, and A vs C on another, b) A/B testing, say, headline and CTA on the same page at the same time, or c) testing different parts of the funnel at the same time? On the second point - this may be an industry difference, but I've never really had any problem coming up with things to A/B test on a landing page, for example. Just off the top of my head, you could test: - Headlines - at least 5 of them. - Headline and subhead vs no subhead. - Font face. - Color scheme (overall) - Images - probably want to test 6-10 of them. - Call To Action (CTA) text. - Call To Action button vs link - CTA placement - Multiple vs single CTAs. - Long copy vs short. - Layout. Image on left, right or top? More than one image? Etc. On the average LP I could come up with 10 possible layouts before pausing for breath. - Testimonials. PLacement, which ones, how long, picture vs audio vs video vs text. - Video vs image-and-text. - Ugly vs good-looking. - Text highlighting, bolding etc - yellow highlighter style vs bold and italic vs nothing. - Other social proofing elements - which media icons to use, where to place them, etc. That's at least 50 A/B tests right there, on a single LP. And all of those elements have been shown in one test or another to affect conversion rates.

by RyanZAG 14y ago

tel 14y ago

Thus why 95% alpha is really important to notice since you should probably be applying Bonferroni corrections (at least).

thenomad 14y ago

Counter point to your first - do I, as the Practical A/B testing Guy, really care if I get one in 100 tests wrong? If I'm running 100 A/B tests a year, 50% of which produce a 5% improvement in my conversion rate, who cares if I get one of them wrong and drop 5% instead? I'll still have improved my conversions by 1100% over the course of the year. (Obviously, I'm not taking into account local maxima here - but any statistical misstep will only temporarily roll your testing down the hill, and subsequent tests will push back up to the local maximum anyway.)

btilly 14y ago
In reality no, because nobody will notice that the company is about 10% smaller than it could have been, and you're doing good stuff. However the cost of that 10% change in your company probably exceeds the cost of running 100 tests. Note that in the real world, after a while you exhaust the easy 5% wins. Then you wind up chasing a lot of 1% and 2% marginal wins, and those tests take a lot longer to run. But even so, if your company is pulling down, say a million per month, a 2% drop in business from getting a 1% test wrong probably exceeds the costs of all of the tests you run per year. So unless extending tests is prohibitively expensive, or there are significant opportunity tests from not being able to run the tests you want, you should go to higher confidence.
- thenomad 14y ago
  Sorry, I'm not quite following you here. In my experience, the primary limiter of how many A/B tests you can run is available traffic. Let's assume that I'm running the 2% tests you mention. I can either run 60 2% tests at 95% confidence, or (approximately) 30 2% tests at 99.9% confidence, if the math posted elsewhere in this thread is right, over a year, say - that's how much traffic I have. Let's once again assume a third of those pan out. I'm still not seeing why I'd prefer 10 2% wins (from the 99.9% approach) to 20 2% wins plus 3 2% losses (from the 95% approach). Yes, there are more errors, but overall I end up with a 40% improvement as opposed to a 24% improvement. Can you explain?
  - btilly 14y ago
    You can run multiple A/B tests in parallel, as long as the tests themselves do not interact. And coming up with and implementing likely ideas is a challenge. Now do you understand?
    - thenomad 14y ago
      On the first point - I may be being dense here. Do you mean a) segmenting traffic, and running A vs B on one segment, and A vs C on another, b) A/B testing, say, headline and CTA on the same page at the same time, or c) testing different parts of the funnel at the same time? On the second point - this may be an industry difference, but I've never really had any problem coming up with things to A/B test on a landing page, for example. Just off the top of my head, you could test: - Headlines - at least 5 of them. - Headline and subhead vs no subhead. - Font face. - Color scheme (overall) - Images - probably want to test 6-10 of them. - Call To Action (CTA) text. - Call To Action button vs link - CTA placement - Multiple vs single CTAs. - Long copy vs short. - Layout. Image on left, right or top? More than one image? Etc. On the average LP I could come up with 10 possible layouts before pausing for breath. - Testimonials. PLacement, which ones, how long, picture vs audio vs video vs text. - Video vs image-and-text. - Ugly vs good-looking. - Text highlighting, bolding etc - yellow highlighter style vs bold and italic vs nothing. - Other social proofing elements - which media icons to use, where to place them, etc. That's at least 50 A/B tests right there, on a single LP. And all of those elements have been shown in one test or another to affect conversion rates.