4 ms·

I enjoyed this, but the title and concept are a bit misleading. I was expecting something where he made iterative tweaks to the creative (i.e. the sign) and di

I enjoyed this, but the title and concept are a bit misleading. I was expecting something where he made iterative tweaks to the creative (i.e. the sign) and did continual comparisons of the success rate to see how different changes affected the result. All this guy did was take a homeless guy's sign and add a really cool marketing idea to it, which increased the homeless guy's take. Yes, the results of his changes were awesome, but it really had nothing to do with "A/B testing" per se.

Splines 17y ago
Would A/B testing encompass changes as large as what happened here? Having never done A/B testing, it sounds like an iterative process. Would continuous A/B testing ever converge on a solution like the one arrived here? I guess what I'm asking is this: Is A/B testing an evolutionary approach to developing a product? Are "revolutionary" leaps possible if you strictly follow the A/B testing methodology? (My guess: probably not, which implies that most people don't follow A/B exclusively, which sounds like common sense).
- inerte 17y ago
  Yes, but not encouraged. The literature, or most common applications of A/B testing, is about little improvements. Phrases being changed, picture alignment, that kinda of thing. Of course, in theory, A/B as the name implies is about testing two different approaches, so again, in theory, any stuff completely different can be seen as an experiment for A/B. The problem is that humans have been trying different ways to do things for a loooooooooong time, and the whole point of trying to teach and learn A/B is to test things, measure, and walk towards your goal. So yes, you can "A/B test" two completely different things, but it will be so. much. harder to actually know what it worked. For example, you're A/B testing a bicycle and a Ferrari Testarossa. D'uh... obviously they'll work differently, but what caused? Price, color, speed, brand? That's why when A/B testing is mentioned it's about small, isolable, repeatable, incremental and evolutionary approach. Because it's a technique with this purpose in mind.
ja27 17y ago
I can't find it now, but I read a great article about different signs that panhandlers "fly". One guy flipped his sign depending on the age of the closest likely target. The old person side asked for help and had "God Bless You". The young person side had "I just need a beer". But the best I've seen was "I bet you can't hit me with a quarter".

Splines 17y ago

Would A/B testing encompass changes as large as what happened here? Having never done A/B testing, it sounds like an iterative process. Would continuous A/B testing ever converge on a solution like the one arrived here? I guess what I'm asking is this: Is A/B testing an evolutionary approach to developing a product? Are "revolutionary" leaps possible if you strictly follow the A/B testing methodology? (My guess: probably not, which implies that most people don't follow A/B exclusively, which sounds like common sense).

inerte 17y ago
Yes, but not encouraged. The literature, or most common applications of A/B testing, is about little improvements. Phrases being changed, picture alignment, that kinda of thing. Of course, in theory, A/B as the name implies is about testing two different approaches, so again, in theory, any stuff completely different can be seen as an experiment for A/B. The problem is that humans have been trying different ways to do things for a loooooooooong time, and the whole point of trying to teach and learn A/B is to test things, measure, and walk towards your goal. So yes, you can "A/B test" two completely different things, but it will be so. much. harder to actually know what it worked. For example, you're A/B testing a bicycle and a Ferrari Testarossa. D'uh... obviously they'll work differently, but what caused? Price, color, speed, brand? That's why when A/B testing is mentioned it's about small, isolable, repeatable, incremental and evolutionary approach. Because it's a technique with this purpose in mind.

ja27 17y ago

I can't find it now, but I read a great article about different signs that panhandlers "fly". One guy flipped his sign depending on the age of the closest likely target. The old person side asked for help and had "God Bless You". The young person side had "I just need a beer". But the best I've seen was "I bet you can't hit me with a quarter".