4 ms·

I've always thought the problem with A/B testing is that it's mostly hill-climbing optimization which is likely to hit and get stuck at a local optimum.

by varelse 13y ago

noelwelsh 13y ago
You could A/B test two completely different business models if you had the resources to do so. Your imagination and your bank account are the meaningful limits on what you can test, not the technology.
- _delirium 13y ago
  That works if you've narrowed it down (via some other process) to two choices of designs. A common argument is that A/B testing should replace unscientific, subjective design processes with data-driven design. But doing that basically reduces to applying randomized-optimization methods (genetic algorithms or whatever your favorite is) to everything: you need a way of mutating, testing, and repeating that explores the entire design space, or at least some relevant portion. And that tends to be difficult to implement in a way that works in complex domains where you haven't formalized all the elements. A/B testing does work well for: 1) choosing between a small number of pre-selected options; or 2) exploring local variations in the neighborhood of an option. #1 is a very traditional kind of experimental study where participants are randomly assigned to two buckets, and #2 is a setting where randomized optimization tends to work well.
  - RyanZAG 13y ago
    Agreed completely. To put it more simply: don't use AB testing to actually create your site by throwing mud on the wall. First design a good site, and then use AB testing to slowly tweak it to the closest best choices. Just to add: unless you are testing google.com or some other top 10 site, #2 is not going to work well as each bucket will be too small to complete the AB test properly. eg. 3/100 users in bucket 92 converting is no proof that it is better than bucket 47 with 2/100 users converting.
  - noelwelsh 13y ago
    I'm agreed you can't test everything -- but that's a limitation imposed by scarce resources (developer time, money, experimental subjects), not by the technology per se. And I'm agreed that trying to enumerate the space of possible designs is a mostly futile effort for most businesses. However, if we take a broader view you can see a market as an experiment. Different companies in a competitive space (e.g. MP3 players before the iPod's dominance, A/B testing software now) choose particular feature sets, price points, and so on. Some are successful, and some aren't. It's difficult to learn from these experiments what contributes to success, as so many factors are mingled together and there is no systematic exploration of the space. As such, a common response is to assign success to the intangible -- "vision" or "genius" -- but I think this is taking the easy way out. Larger companies certainly use the market as a system o judge companies to acquire, and then often get burned when they fail to understand and maintain what brought about the acquired company's success.
    - _delirium 13y ago
      I agree with the first point, but maybe it's my AI background: "that's a limitation imposed by scarce resources" is a pretty major limitation! If I had infinite time on infinitely fast CPUs with infinite human input, genetic programming algorithms writing code could completely replace human programmers and software architects. The reason they can't is because of a lack of infinite resources.
zaguios 13y ago
He addresses exactly that point with False Premise #2. It really depends on how you use A/B testing, if you only do it to test small incremental changes then yes, you're correct, but there's nothing stopping you from testing two totally different features/designs with A/B testing.
- varelse 13y ago
  To which I'd respond that from personal experience, on a large corporate project, it's difficult to get authorization to do just that. And even so, testing just two different designs is rolling the dice rather than exploring feature/design space in search of a new optimum. As examples, I personally hate what's happened to the android clients for GMail and Google Maps. The former seems a victim of incremental change while the latter seems the consequence of a radical redesign that hit me out of the blue in the middle of trying to drive somewhere when I was already late.
  - CharlesW 13y ago
    "And even so, testing just two different designs is rolling the dice rather than exploring feature/design space in search of a new optimum." A/B testing doesn't force you to innovate through radical design changes — you can use it just as easily for incremental improvements.
ojilles 13y ago
Only if you keep testing variations on the same thing. There's nothing about A/B testing that says (or drives you toward) doing that, per se.
wikwocket 13y ago
A/B testing doesn't necessarily mean "test two subtle shades of the same thing." For a great example, see the Highrise A/B test: http://37signals.com/svn/posts/2991-behind-the-scenes-ab-testing-part-3-final http://37signals.com/svn/posts/2991-behind-the-scenes-ab-tes... This is a good example that disproves many of the false premises mentioned in the OP, as the clear winner of the A/B tests was a bold dynamic design with lots of character, that was completely different from the original designs.
- anywhichway 13y ago
  Great example to bring into the conversation. One complexity to consider is that the long form versus the smiling person may have retained completely different user demographic. To me, the smiling person may represents an easy, idiot proof service with customer support, but may be lacking more advanced technical features, which is fine if you are set up to cater to that demo. Technical services like Github may suffer from increased sign ups from the wrong demo if it means their forums are swamped with beginner level questions that they aren't equipped to hand hold.

I've always thought the problem with A/B testing is that it's mostly hill-climbing optimization which is likely to hit and get stuck at a local optimum.

by varelse 13y ago

I've always thought the problem with A/B testing is that it's mostly hill-climbing optimization which is likely to hit and get stuck at a local optimum.

noelwelsh 13y ago

You could A/B test two completely different business models if you had the resources to do so. Your imagination and your bank account are the meaningful limits on what you can test, not the technology.

_delirium 13y ago
That works if you've narrowed it down (via some other process) to two choices of designs. A common argument is that A/B testing should replace unscientific, subjective design processes with data-driven design. But doing that basically reduces to applying randomized-optimization methods (genetic algorithms or whatever your favorite is) to everything: you need a way of mutating, testing, and repeating that explores the entire design space, or at least some relevant portion. And that tends to be difficult to implement in a way that works in complex domains where you haven't formalized all the elements. A/B testing does work well for: 1) choosing between a small number of pre-selected options; or 2) exploring local variations in the neighborhood of an option. #1 is a very traditional kind of experimental study where participants are randomly assigned to two buckets, and #2 is a setting where randomized optimization tends to work well.
- RyanZAG 13y ago
  Agreed completely. To put it more simply: don't use AB testing to actually create your site by throwing mud on the wall. First design a good site, and then use AB testing to slowly tweak it to the closest best choices. Just to add: unless you are testing google.com or some other top 10 site, #2 is not going to work well as each bucket will be too small to complete the AB test properly. eg. 3/100 users in bucket 92 converting is no proof that it is better than bucket 47 with 2/100 users converting.
- noelwelsh 13y ago
  I'm agreed you can't test everything -- but that's a limitation imposed by scarce resources (developer time, money, experimental subjects), not by the technology per se. And I'm agreed that trying to enumerate the space of possible designs is a mostly futile effort for most businesses. However, if we take a broader view you can see a market as an experiment. Different companies in a competitive space (e.g. MP3 players before the iPod's dominance, A/B testing software now) choose particular feature sets, price points, and so on. Some are successful, and some aren't. It's difficult to learn from these experiments what contributes to success, as so many factors are mingled together and there is no systematic exploration of the space. As such, a common response is to assign success to the intangible -- "vision" or "genius" -- but I think this is taking the easy way out. Larger companies certainly use the market as a system o judge companies to acquire, and then often get burned when they fail to understand and maintain what brought about the acquired company's success.
  - _delirium 13y ago
    I agree with the first point, but maybe it's my AI background: "that's a limitation imposed by scarce resources" is a pretty major limitation! If I had infinite time on infinitely fast CPUs with infinite human input, genetic programming algorithms writing code could completely replace human programmers and software architects. The reason they can't is because of a lack of infinite resources.

zaguios 13y ago

He addresses exactly that point with False Premise #2. It really depends on how you use A/B testing, if you only do it to test small incremental changes then yes, you're correct, but there's nothing stopping you from testing two totally different features/designs with A/B testing.

varelse 13y ago
To which I'd respond that from personal experience, on a large corporate project, it's difficult to get authorization to do just that. And even so, testing just two different designs is rolling the dice rather than exploring feature/design space in search of a new optimum. As examples, I personally hate what's happened to the android clients for GMail and Google Maps. The former seems a victim of incremental change while the latter seems the consequence of a radical redesign that hit me out of the blue in the middle of trying to drive somewhere when I was already late.
- CharlesW 13y ago
  "And even so, testing just two different designs is rolling the dice rather than exploring feature/design space in search of a new optimum." A/B testing doesn't force you to innovate through radical design changes — you can use it just as easily for incremental improvements.

ojilles 13y ago

Only if you keep testing variations on the same thing. There's nothing about A/B testing that says (or drives you toward) doing that, per se.

wikwocket 13y ago

A/B testing doesn't necessarily mean "test two subtle shades of the same thing." For a great example, see the Highrise A/B test: http://37signals.com/svn/posts/2991-behind-the-scenes-ab-testing-part-3-final http://37signals.com/svn/posts/2991-behind-the-scenes-ab-tes... This is a good example that disproves many of the false premises mentioned in the OP, as the clear winner of the A/B tests was a bold dynamic design with lots of character, that was completely different from the original designs.

anywhichway 13y ago
Great example to bring into the conversation. One complexity to consider is that the long form versus the smiling person may have retained completely different user demographic. To me, the smiling person may represents an easy, idiot proof service with customer support, but may be lacking more advanced technical features, which is fine if you are set up to cater to that demo. Technical services like Github may suffer from increased sign ups from the wrong demo if it means their forums are swamped with beginner level questions that they aren't equipped to hand hold.