3 ms·

It's more that A/B testing will find the local maxima; you might get a 0.1% improvement to conversion by making the "Add to cart" a better color, but you could

by gav 14y ago

It's more that A/B testing will find the local maxima; you might get a 0.1% improvement to conversion by making the "Add to cart" a better color, but you could get a 1% improvement by streamlining the checkout flow. A lot of this comes from having experience, you know the right place to start looking for the problem.

noelwelsh 14y ago
This argument comes up quite regularly, but it is based on a misunderstanding of A/B testing. People see blog posts about changing button colours and think that is all there is to A/B testing. I've seen far more success with much more radical changes. I work in the A/B testing industry (see my startup Myna: http://mynaweb.com http://mynaweb.com) and this experience is reflected by others in the industry. E.g. see this video by Dan Siroker from our Optimizely (a competitor of ours): http://www.youtube.com/watch?v=lDfoVxHud7Y&feature=youtu.be http://www.youtube.com/watch?v=lDfoVxHud7Y&feature=youtu... He directly addresses this issue. There is no reason you can't test the example you give of streamlining the checkout flow. So long as you have sufficient resources you can test anything you can change and measure -- landing pages, websites, business models...
- gav 14y ago
  I think maybe my point wasn't clear enough, I agree with you. I was trying to point out that with experience you know where to start testing so you can focus on fixing the pain points that will have the biggest impact. A/B testing is an important tool to aid optimization, but knowing where to focus your efforts is key. I have over a decade's worth of experience improving e-commerce conversion so generally my first step isn't introducing testing, but fixing the obvious things I know will have an impact because they always have an impact, e.g. fixing a confusing checkout; improving search and navigation; embarking on a data cleanup and improving product content, etc..
  - PeterisP 14y ago
    Well, but that's exactly what A/B testing should be used for - instead of optimizing to a local optimum, you use it to verify if the 'new and improved' checkout or search or navigation is really an improvement, or is insignificant, or happens to make things worse. "The obvious things I know will have an impact" sometimes don't have an impact and sometimes have a negative impact. Experience makes the chances better, but nowhere near perfect.

by gav 14y ago

noelwelsh 14y ago

This argument comes up quite regularly, but it is based on a misunderstanding of A/B testing. People see blog posts about changing button colours and think that is all there is to A/B testing. I've seen far more success with much more radical changes. I work in the A/B testing industry (see my startup Myna: http://mynaweb.com http://mynaweb.com) and this experience is reflected by others in the industry. E.g. see this video by Dan Siroker from our Optimizely (a competitor of ours): http://www.youtube.com/watch?v=lDfoVxHud7Y&feature=youtu.be http://www.youtube.com/watch?v=lDfoVxHud7Y&feature=youtu... He directly addresses this issue. There is no reason you can't test the example you give of streamlining the checkout flow. So long as you have sufficient resources you can test anything you can change and measure -- landing pages, websites, business models...

gav 14y ago
I think maybe my point wasn't clear enough, I agree with you. I was trying to point out that with experience you know where to start testing so you can focus on fixing the pain points that will have the biggest impact. A/B testing is an important tool to aid optimization, but knowing where to focus your efforts is key. I have over a decade's worth of experience improving e-commerce conversion so generally my first step isn't introducing testing, but fixing the obvious things I know will have an impact because they always have an impact, e.g. fixing a confusing checkout; improving search and navigation; embarking on a data cleanup and improving product content, etc..
- PeterisP 14y ago
  Well, but that's exactly what A/B testing should be used for - instead of optimizing to a local optimum, you use it to verify if the 'new and improved' checkout or search or navigation is really an improvement, or is insignificant, or happens to make things worse. "The obvious things I know will have an impact" sometimes don't have an impact and sometimes have a negative impact. Experience makes the chances better, but nowhere near perfect.