4 ms·

I'm on board with every statement except the first one, "Test whispers instead of screams." The author suggests that you should only test the big things that c

I'm on board with every statement except the first one, "Test whispers instead of screams." The author suggests that you should only test the big things that could be "grand slams," like a "radically different design." There are two major problems with that statement. First, a radically different design is going to have so many differences that you'll never be able to pinpoint which element made the difference. If you're seeing a 10% lift, does that mean you might have gotten rid of something that was working great, and you could have had a 30% lift by just fixing the actual problem? What if the radical new design actually lower sales? Now you have to find out which of the forty-seven changes caused it, or just go back to the old design altogether. That's a lot of unnecessary and expensive work. The second major problem is that by going for "grand slams," you're basically throwing spaghetti against the wall to see what sticks. Instead, take the time to be analytical and form real testable hypotheses about what could make an impact. Jared Spool tells a story in the book "Web Form Design" about a checkout process that started with a simple two-field input form asking users to register for an account or login before paying for their purchase. After looking at user data carefully, they found that almost all of their dropoffs were happening right there. They tried moving the login fields to the end of the checkout process instead of the beginning, and saw a 45% conversion increase, over $1.5 million in the first month. A radical redesign wouldn't have fixed the problem - they needed to do user path analysis and give the issue careful thought before deciding where to make the change. It's often a single usability issue that's causing the problems, and you can't find those by changing everything on a page and hoping for the best.

paraschopra 15y ago
I agree. This is a guest post on our blog but my opinion is that one should do a lot (and lot) of small A/B tests coupled with a couple of big, large scale tests. Interestingly, most of our A/B testing case studies have been about small changes leading to large conversion rate changes: http://visualwebsiteoptimizer.com/case-studies.php http://visualwebsiteoptimizer.com/case-studies.php Although "grand slam" tests are essential so that you don't get trapped to local minima.
Reeveso 15y ago
Thanks for your comments Jayson. Funnily enough, I actually agree with your comments, depending on the type of tester you are. There are basically 2 ways to test radical changes. 1) Like you said. Do it little by little. 2) Do research FIRST and find out what people want, and then test 2-3 radical changes. Once you find the best "radical" change... you test each individual component. So basically, it's your personal preference. It also depends on the amount of traffic you get. Smaller websites are basically FORCED to do "screams" while sites getting thousands of visitors per day can afford to do whispers. The bottom line is... each method has pro's and con's and works great depending on the circumstance. I know I've personally done both for myself and clients :) Oh and as a "P.S"... you're definitely NOT throwing spaghetti against the wall to see what sticks. When doing radical changes, you're not just slapping up any random idea. Each individual component should be put in there based on previous testing data and best practices. Then... you go from there and test each individual component. Hope that explanation helps! Jeremy
TimJRobinson 15y ago
Very true, though I think the point the author was making is that it's better to start big and work your way down (IIRC 37 signals said something similar when they did an overhaul of their site a few months ago). It often works out better to do one completely different layout or style of page and A/B test between them. Then once you have a winner for that start testing all the details with a multivariate test.

paraschopra 15y ago

I agree. This is a guest post on our blog but my opinion is that one should do a lot (and lot) of small A/B tests coupled with a couple of big, large scale tests. Interestingly, most of our A/B testing case studies have been about small changes leading to large conversion rate changes: http://visualwebsiteoptimizer.com/case-studies.php http://visualwebsiteoptimizer.com/case-studies.php Although "grand slam" tests are essential so that you don't get trapped to local minima.

Reeveso 15y ago

Thanks for your comments Jayson. Funnily enough, I actually agree with your comments, depending on the type of tester you are. There are basically 2 ways to test radical changes. 1) Like you said. Do it little by little. 2) Do research FIRST and find out what people want, and then test 2-3 radical changes. Once you find the best "radical" change... you test each individual component. So basically, it's your personal preference. It also depends on the amount of traffic you get. Smaller websites are basically FORCED to do "screams" while sites getting thousands of visitors per day can afford to do whispers. The bottom line is... each method has pro's and con's and works great depending on the circumstance. I know I've personally done both for myself and clients :) Oh and as a "P.S"... you're definitely NOT throwing spaghetti against the wall to see what sticks. When doing radical changes, you're not just slapping up any random idea. Each individual component should be put in there based on previous testing data and best practices. Then... you go from there and test each individual component. Hope that explanation helps! Jeremy

TimJRobinson 15y ago

Very true, though I think the point the author was making is that it's better to start big and work your way down (IIRC 37 signals said something similar when they did an overhaul of their site a few months ago). It often works out better to do one completely different layout or style of page and A/B test between them. Then once you have a winner for that start testing all the details with a multivariate test.