4 ms·

You could A/B test two completely different business models if you had the resources to do so. Your imagination and your bank account are the meaningful limits

You could A/B test two completely different business models if you had the resources to do so. Your imagination and your bank account are the meaningful limits on what you can test, not the technology.

_delirium 13y ago
That works if you've narrowed it down (via some other process) to two choices of designs. A common argument is that A/B testing should replace unscientific, subjective design processes with data-driven design. But doing that basically reduces to applying randomized-optimization methods (genetic algorithms or whatever your favorite is) to everything: you need a way of mutating, testing, and repeating that explores the entire design space, or at least some relevant portion. And that tends to be difficult to implement in a way that works in complex domains where you haven't formalized all the elements. A/B testing does work well for: 1) choosing between a small number of pre-selected options; or 2) exploring local variations in the neighborhood of an option. #1 is a very traditional kind of experimental study where participants are randomly assigned to two buckets, and #2 is a setting where randomized optimization tends to work well.
- RyanZAG 13y ago
  Agreed completely. To put it more simply: don't use AB testing to actually create your site by throwing mud on the wall. First design a good site, and then use AB testing to slowly tweak it to the closest best choices. Just to add: unless you are testing google.com or some other top 10 site, #2 is not going to work well as each bucket will be too small to complete the AB test properly. eg. 3/100 users in bucket 92 converting is no proof that it is better than bucket 47 with 2/100 users converting.
- noelwelsh 13y ago
  I'm agreed you can't test everything -- but that's a limitation imposed by scarce resources (developer time, money, experimental subjects), not by the technology per se. And I'm agreed that trying to enumerate the space of possible designs is a mostly futile effort for most businesses. However, if we take a broader view you can see a market as an experiment. Different companies in a competitive space (e.g. MP3 players before the iPod's dominance, A/B testing software now) choose particular feature sets, price points, and so on. Some are successful, and some aren't. It's difficult to learn from these experiments what contributes to success, as so many factors are mingled together and there is no systematic exploration of the space. As such, a common response is to assign success to the intangible -- "vision" or "genius" -- but I think this is taking the easy way out. Larger companies certainly use the market as a system o judge companies to acquire, and then often get burned when they fail to understand and maintain what brought about the acquired company's success.
  - _delirium 13y ago
    I agree with the first point, but maybe it's my AI background: "that's a limitation imposed by scarce resources" is a pretty major limitation! If I had infinite time on infinitely fast CPUs with infinite human input, genetic programming algorithms writing code could completely replace human programmers and software architects. The reason they can't is because of a lack of infinite resources.

_delirium 13y ago

That works if you've narrowed it down (via some other process) to two choices of designs. A common argument is that A/B testing should replace unscientific, subjective design processes with data-driven design. But doing that basically reduces to applying randomized-optimization methods (genetic algorithms or whatever your favorite is) to everything: you need a way of mutating, testing, and repeating that explores the entire design space, or at least some relevant portion. And that tends to be difficult to implement in a way that works in complex domains where you haven't formalized all the elements. A/B testing does work well for: 1) choosing between a small number of pre-selected options; or 2) exploring local variations in the neighborhood of an option. #1 is a very traditional kind of experimental study where participants are randomly assigned to two buckets, and #2 is a setting where randomized optimization tends to work well.

RyanZAG 13y ago
Agreed completely. To put it more simply: don't use AB testing to actually create your site by throwing mud on the wall. First design a good site, and then use AB testing to slowly tweak it to the closest best choices. Just to add: unless you are testing google.com or some other top 10 site, #2 is not going to work well as each bucket will be too small to complete the AB test properly. eg. 3/100 users in bucket 92 converting is no proof that it is better than bucket 47 with 2/100 users converting.
noelwelsh 13y ago
I'm agreed you can't test everything -- but that's a limitation imposed by scarce resources (developer time, money, experimental subjects), not by the technology per se. And I'm agreed that trying to enumerate the space of possible designs is a mostly futile effort for most businesses. However, if we take a broader view you can see a market as an experiment. Different companies in a competitive space (e.g. MP3 players before the iPod's dominance, A/B testing software now) choose particular feature sets, price points, and so on. Some are successful, and some aren't. It's difficult to learn from these experiments what contributes to success, as so many factors are mingled together and there is no systematic exploration of the space. As such, a common response is to assign success to the intangible -- "vision" or "genius" -- but I think this is taking the easy way out. Larger companies certainly use the market as a system o judge companies to acquire, and then often get burned when they fail to understand and maintain what brought about the acquired company's success.
- _delirium 13y ago
  I agree with the first point, but maybe it's my AI background: "that's a limitation imposed by scarce resources" is a pretty major limitation! If I had infinite time on infinitely fast CPUs with infinite human input, genetic programming algorithms writing code could completely replace human programmers and software architects. The reason they can't is because of a lack of infinite resources.