4 ms·
The whole point of the article is "don't A/B test (or do anything else fancy) until you run out of obvious problems."
by derefr 14y ago
The whole point of the article is "don't A/B test (or do anything else fancy) until you run out of obvious problems."
- noelwelsh 14y agoSorry, let me be clearer. What I may believe to be an obvious problem may not actually be a problem for customers. What may be an obvious problem for customers may not be an obvious problem for me. Intuition is often wrong.
- derefr 14y agoThere's no impassible wall between you and your customers. Watch a few of them use your product. If 80% of them stumble somewhere--that's an obvious problem. It didn't take Chi-square analysis for Microsoft to notice that nobody could figure out how to start a program in Windows 95; they just sat down and watched a few people, and it was suddenly very clear that they needed a big arrow pointing to the start button.
- noelwelsh 14y agoAgreed! Generating hypotheses often involves qualitative techniques. Validation must be quantitative.
- gav 14y agoIt's more that A/B testing will find the local maxima; you might get a 0.1% improvement to conversion by making the "Add to cart" a better color, but you could get a 1% improvement by streamlining the checkout flow. A lot of this comes from having experience, you know the right place to start looking for the problem.
- noelwelsh 14y agoThis argument comes up quite regularly, but it is based on a misunderstanding of A/B testing. People see blog posts about changing button colours and think that is all there is to A/B testing. I've seen far more success with much more radical changes. I work in the A/B testing industry (see my startup Myna: http://mynaweb.com http://mynaweb.com) and this experience is reflected by others in the industry. E.g. see this video by Dan Siroker from our Optimizely (a competitor of ours): http://www.youtube.com/watch?v=lDfoVxHud7Y&feature=youtu.be http://www.youtube.com/watch?v=lDfoVxHud7Y&feature=youtu... He directly addresses this issue. There is no reason you can't test the example you give of streamlining the checkout flow. So long as you have sufficient resources you can test anything you can change and measure -- landing pages, websites, business models...
- gav 14y agoI think maybe my point wasn't clear enough, I agree with you. I was trying to point out that with experience you know where to start testing so you can focus on fixing the pain points that will have the biggest impact. A/B testing is an important tool to aid optimization, but knowing where to focus your efforts is key. I have over a decade's worth of experience improving e-commerce conversion so generally my first step isn't introducing testing, but fixing the obvious things I know will have an impact because they always have an impact, e.g. fixing a confusing checkout; improving search and navigation; embarking on a data cleanup and improving product content, etc..
- PeterisP 14y agoWell, but that's exactly what A/B testing should be used for - instead of optimizing to a local optimum, you use it to verify if the 'new and improved' checkout or search or navigation is really an improvement, or is insignificant, or happens to make things worse. "The obvious things I know will have an impact" sometimes don't have an impact and sometimes have a negative impact. Experience makes the chances better, but nowhere near perfect.