4 ms·

I enjoyed this article and I agree with the main point it makes. I want to quibble about a smaller point, which is that intuition is not always a reliable guide

I enjoyed this article and I agree with the main point it makes. I want to quibble about a smaller point, which is that intuition is not always a reliable guide. The author gives a few examples where they got stuck in and fixed the "obvious" problems. My experience is that what is a problem is not always obvious. Furthermore, while the big wins might be obvious, you'll miss many small wins if you're not carefully tracking stats. These are the main lessons of A/B testing.

derefr 14y ago
The whole point of the article is "don't A/B test (or do anything else fancy) until you run out of obvious problems."
- noelwelsh 14y ago
  Sorry, let me be clearer. What I may believe to be an obvious problem may not actually be a problem for customers. What may be an obvious problem for customers may not be an obvious problem for me. Intuition is often wrong.
  - derefr 14y ago
    There's no impassible wall between you and your customers. Watch a few of them use your product. If 80% of them stumble somewhere--that's an obvious problem. It didn't take Chi-square analysis for Microsoft to notice that nobody could figure out how to start a program in Windows 95; they just sat down and watched a few people, and it was suddenly very clear that they needed a big arrow pointing to the start button.
    - noelwelsh 14y ago
      Agreed! Generating hypotheses often involves qualitative techniques. Validation must be quantitative.
- gav 14y ago
  It's more that A/B testing will find the local maxima; you might get a 0.1% improvement to conversion by making the "Add to cart" a better color, but you could get a 1% improvement by streamlining the checkout flow. A lot of this comes from having experience, you know the right place to start looking for the problem.
  - noelwelsh 14y ago
    This argument comes up quite regularly, but it is based on a misunderstanding of A/B testing. People see blog posts about changing button colours and think that is all there is to A/B testing. I've seen far more success with much more radical changes. I work in the A/B testing industry (see my startup Myna: http://mynaweb.com http://mynaweb.com) and this experience is reflected by others in the industry. E.g. see this video by Dan Siroker from our Optimizely (a competitor of ours): http://www.youtube.com/watch?v=lDfoVxHud7Y&feature=youtu.be http://www.youtube.com/watch?v=lDfoVxHud7Y&feature=youtu... He directly addresses this issue. There is no reason you can't test the example you give of streamlining the checkout flow. So long as you have sufficient resources you can test anything you can change and measure -- landing pages, websites, business models...
    - gav 14y ago
      I think maybe my point wasn't clear enough, I agree with you. I was trying to point out that with experience you know where to start testing so you can focus on fixing the pain points that will have the biggest impact. A/B testing is an important tool to aid optimization, but knowing where to focus your efforts is key. I have over a decade's worth of experience improving e-commerce conversion so generally my first step isn't introducing testing, but fixing the obvious things I know will have an impact because they always have an impact, e.g. fixing a confusing checkout; improving search and navigation; embarking on a data cleanup and improving product content, etc..
      - PeterisP 14y ago
        Well, but that's exactly what A/B testing should be used for - instead of optimizing to a local optimum, you use it to verify if the 'new and improved' checkout or search or navigation is really an improvement, or is insignificant, or happens to make things worse. "The obvious things I know will have an impact" sometimes don't have an impact and sometimes have a negative impact. Experience makes the chances better, but nowhere near perfect.
tel 14y ago
The problem, I feel, is impedance. You should be A/B testing constantly and using it to both find obvious problems, complement qualitative testing, and inform decisions about next steps. To give your full decision-making power over to an algorithm is stupid. To ignore quantitative power when it's available to you in your decision-making is stupid.
sopooneo 14y ago
Yes. I have seen nothing cause more wasted time than failing to carefully determine what the question (i.e. problem) is. It takes discussion and thought and creativity and getting it right is more important than any other aspect of a software project.
deleted 14y ago
[deleted]

derefr 14y ago

The whole point of the article is "don't A/B test (or do anything else fancy) until you run out of obvious problems."

noelwelsh 14y ago
Sorry, let me be clearer. What I may believe to be an obvious problem may not actually be a problem for customers. What may be an obvious problem for customers may not be an obvious problem for me. Intuition is often wrong.
- derefr 14y ago
  There's no impassible wall between you and your customers. Watch a few of them use your product. If 80% of them stumble somewhere--that's an obvious problem. It didn't take Chi-square analysis for Microsoft to notice that nobody could figure out how to start a program in Windows 95; they just sat down and watched a few people, and it was suddenly very clear that they needed a big arrow pointing to the start button.
  - noelwelsh 14y ago
    Agreed! Generating hypotheses often involves qualitative techniques. Validation must be quantitative.
gav 14y ago
It's more that A/B testing will find the local maxima; you might get a 0.1% improvement to conversion by making the "Add to cart" a better color, but you could get a 1% improvement by streamlining the checkout flow. A lot of this comes from having experience, you know the right place to start looking for the problem.
- noelwelsh 14y ago
  This argument comes up quite regularly, but it is based on a misunderstanding of A/B testing. People see blog posts about changing button colours and think that is all there is to A/B testing. I've seen far more success with much more radical changes. I work in the A/B testing industry (see my startup Myna: http://mynaweb.com http://mynaweb.com) and this experience is reflected by others in the industry. E.g. see this video by Dan Siroker from our Optimizely (a competitor of ours): http://www.youtube.com/watch?v=lDfoVxHud7Y&feature=youtu.be http://www.youtube.com/watch?v=lDfoVxHud7Y&feature=youtu... He directly addresses this issue. There is no reason you can't test the example you give of streamlining the checkout flow. So long as you have sufficient resources you can test anything you can change and measure -- landing pages, websites, business models...
  - gav 14y ago
    I think maybe my point wasn't clear enough, I agree with you. I was trying to point out that with experience you know where to start testing so you can focus on fixing the pain points that will have the biggest impact. A/B testing is an important tool to aid optimization, but knowing where to focus your efforts is key. I have over a decade's worth of experience improving e-commerce conversion so generally my first step isn't introducing testing, but fixing the obvious things I know will have an impact because they always have an impact, e.g. fixing a confusing checkout; improving search and navigation; embarking on a data cleanup and improving product content, etc..
    - PeterisP 14y ago
      Well, but that's exactly what A/B testing should be used for - instead of optimizing to a local optimum, you use it to verify if the 'new and improved' checkout or search or navigation is really an improvement, or is insignificant, or happens to make things worse. "The obvious things I know will have an impact" sometimes don't have an impact and sometimes have a negative impact. Experience makes the chances better, but nowhere near perfect.

tel 14y ago

The problem, I feel, is impedance. You should be A/B testing constantly and using it to both find obvious problems, complement qualitative testing, and inform decisions about next steps. To give your full decision-making power over to an algorithm is stupid. To ignore quantitative power when it's available to you in your decision-making is stupid.

sopooneo 14y ago

Yes. I have seen nothing cause more wasted time than failing to carefully determine what the question (i.e. problem) is. It takes discussion and thought and creativity and getting it right is more important than any other aspect of a software project.

deleted 14y ago

[deleted]