4 ms·

Top 7 split testing blunders you must avoid

26 points by paraschopra 15y ago

jaysonelliot 15y ago
I'm on board with every statement except the first one, "Test whispers instead of screams." The author suggests that you should only test the big things that could be "grand slams," like a "radically different design." There are two major problems with that statement. First, a radically different design is going to have so many differences that you'll never be able to pinpoint which element made the difference. If you're seeing a 10% lift, does that mean you might have gotten rid of something that was working great, and you could have had a 30% lift by just fixing the actual problem? What if the radical new design actually lower sales? Now you have to find out which of the forty-seven changes caused it, or just go back to the old design altogether. That's a lot of unnecessary and expensive work. The second major problem is that by going for "grand slams," you're basically throwing spaghetti against the wall to see what sticks. Instead, take the time to be analytical and form real testable hypotheses about what could make an impact. Jared Spool tells a story in the book "Web Form Design" about a checkout process that started with a simple two-field input form asking users to register for an account or login before paying for their purchase. After looking at user data carefully, they found that almost all of their dropoffs were happening right there. They tried moving the login fields to the end of the checkout process instead of the beginning, and saw a 45% conversion increase, over $1.5 million in the first month. A radical redesign wouldn't have fixed the problem - they needed to do user path analysis and give the issue careful thought before deciding where to make the change. It's often a single usability issue that's causing the problems, and you can't find those by changing everything on a page and hoping for the best.
- paraschopra 15y ago
  I agree. This is a guest post on our blog but my opinion is that one should do a lot (and lot) of small A/B tests coupled with a couple of big, large scale tests. Interestingly, most of our A/B testing case studies have been about small changes leading to large conversion rate changes: http://visualwebsiteoptimizer.com/case-studies.php http://visualwebsiteoptimizer.com/case-studies.php Although "grand slam" tests are essential so that you don't get trapped to local minima.
- Reeveso 15y ago
  Thanks for your comments Jayson. Funnily enough, I actually agree with your comments, depending on the type of tester you are. There are basically 2 ways to test radical changes. 1) Like you said. Do it little by little. 2) Do research FIRST and find out what people want, and then test 2-3 radical changes. Once you find the best "radical" change... you test each individual component. So basically, it's your personal preference. It also depends on the amount of traffic you get. Smaller websites are basically FORCED to do "screams" while sites getting thousands of visitors per day can afford to do whispers. The bottom line is... each method has pro's and con's and works great depending on the circumstance. I know I've personally done both for myself and clients :) Oh and as a "P.S"... you're definitely NOT throwing spaghetti against the wall to see what sticks. When doing radical changes, you're not just slapping up any random idea. Each individual component should be put in there based on previous testing data and best practices. Then... you go from there and test each individual component. Hope that explanation helps! Jeremy
- TimJRobinson 15y ago
  Very true, though I think the point the author was making is that it's better to start big and work your way down (IIRC 37 signals said something similar when they did an overhaul of their site a few months ago). It often works out better to do one completely different layout or style of page and A/B test between them. Then once you have a winner for that start testing all the details with a multivariate test.
tnorthcutt 15y ago
(I left a similar comment on the blog as well) “Let’s just say you get a 10% bump. Well… if you then roll that out into the other 5 products, you can count on a 10% bump on each of those as well… giving you a total of 50% in total increased conversions spread out throughout your site.” That’s incorrect. A 10% increase in conversions across all products on your site gives you a total increase of 10%, not 50%. Perhaps #8 on this list should be “incorrectly interpreting results”.
- Reeveso 15y ago
  Please read the sentence again. I made sure to say... "a TOTAL of 50% in TOTAL increased conversions SPREAD OUT throughout your site". I do understand what you're trying to say, though. It's simply a matter of the way you're thinking about it and depends on the context.
micheljansen 15y ago
I think a more important blunder is failing to understand statistics. "I’ve ran plenty of tests where one variation was the control, hitting 95% confidence, after just a handful of conversions ... Because you need to run your tests long enough to get accurate results! You need to get past what I call “random coin syndrome”." Statistics don't work that way. A result is either significant (e.g. you can either draw conclusions with a 95% confidence rate), or it is not. If you reach 95% confidence after 14 conversions (10 vs 4), it either means your questions are too broad or you failed to control for external influences.
- Jabbles 15y ago
  The flaw is more fundamental than that. By testing the confidence each time, you run the risk of vastly overestimating the true significance of your tests. A previous discussion of this exact problem: http://news.ycombinator.com/item?id=2368825 http://news.ycombinator.com/item?id=2368825

26 points by paraschopra 15y ago

I'm on board with every statement except the first one, "Test whispers instead of screams." The author suggests that you should only test the big things that could be "grand slams," like a "radically different design." There are two major problems with that statement. First, a radically different design is going to have so many differences that you'll never be able to pinpoint which element made the difference. If you're seeing a 10% lift, does that mean you might have gotten rid of something that was working great, and you could have had a 30% lift by just fixing the actual problem? What if the radical new design actually lower sales? Now you have to find out which of the forty-seven changes caused it, or just go back to the old design altogether. That's a lot of unnecessary and expensive work. The second major problem is that by going for "grand slams," you're basically throwing spaghetti against the wall to see what sticks. Instead, take the time to be analytical and form real testable hypotheses about what could make an impact. Jared Spool tells a story in the book "Web Form Design" about a checkout process that started with a simple two-field input form asking users to register for an account or login before paying for their purchase. After looking at user data carefully, they found that almost all of their dropoffs were happening right there. They tried moving the login fields to the end of the checkout process instead of the beginning, and saw a 45% conversion increase, over $1.5 million in the first month. A radical redesign wouldn't have fixed the problem - they needed to do user path analysis and give the issue careful thought before deciding where to make the change. It's often a single usability issue that's causing the problems, and you can't find those by changing everything on a page and hoping for the best.

paraschopra 15y ago
I agree. This is a guest post on our blog but my opinion is that one should do a lot (and lot) of small A/B tests coupled with a couple of big, large scale tests. Interestingly, most of our A/B testing case studies have been about small changes leading to large conversion rate changes: http://visualwebsiteoptimizer.com/case-studies.php http://visualwebsiteoptimizer.com/case-studies.php Although "grand slam" tests are essential so that you don't get trapped to local minima.
Reeveso 15y ago
Thanks for your comments Jayson. Funnily enough, I actually agree with your comments, depending on the type of tester you are. There are basically 2 ways to test radical changes. 1) Like you said. Do it little by little. 2) Do research FIRST and find out what people want, and then test 2-3 radical changes. Once you find the best "radical" change... you test each individual component. So basically, it's your personal preference. It also depends on the amount of traffic you get. Smaller websites are basically FORCED to do "screams" while sites getting thousands of visitors per day can afford to do whispers. The bottom line is... each method has pro's and con's and works great depending on the circumstance. I know I've personally done both for myself and clients :) Oh and as a "P.S"... you're definitely NOT throwing spaghetti against the wall to see what sticks. When doing radical changes, you're not just slapping up any random idea. Each individual component should be put in there based on previous testing data and best practices. Then... you go from there and test each individual component. Hope that explanation helps! Jeremy
TimJRobinson 15y ago
Very true, though I think the point the author was making is that it's better to start big and work your way down (IIRC 37 signals said something similar when they did an overhaul of their site a few months ago). It often works out better to do one completely different layout or style of page and A/B test between them. Then once you have a winner for that start testing all the details with a multivariate test.

tnorthcutt 15y ago

(I left a similar comment on the blog as well) “Let’s just say you get a 10% bump. Well… if you then roll that out into the other 5 products, you can count on a 10% bump on each of those as well… giving you a total of 50% in total increased conversions spread out throughout your site.” That’s incorrect. A 10% increase in conversions across all products on your site gives you a total increase of 10%, not 50%. Perhaps #8 on this list should be “incorrectly interpreting results”.

Reeveso 15y ago
Please read the sentence again. I made sure to say... "a TOTAL of 50% in TOTAL increased conversions SPREAD OUT throughout your site". I do understand what you're trying to say, though. It's simply a matter of the way you're thinking about it and depends on the context.

micheljansen 15y ago

I think a more important blunder is failing to understand statistics. "I’ve ran plenty of tests where one variation was the control, hitting 95% confidence, after just a handful of conversions ... Because you need to run your tests long enough to get accurate results! You need to get past what I call “random coin syndrome”." Statistics don't work that way. A result is either significant (e.g. you can either draw conclusions with a 95% confidence rate), or it is not. If you reach 95% confidence after 14 conversions (10 vs 4), it either means your questions are too broad or you failed to control for external influences.

Jabbles 15y ago
The flaw is more fundamental than that. By testing the confidence each time, you run the risk of vastly overestimating the true significance of your tests. A previous discussion of this exact problem: http://news.ycombinator.com/item?id=2368825 http://news.ycombinator.com/item?id=2368825