5 ms·

Marketing Lessons from A/B Testing a Homeless Man's Sign

136 points by aresant 17y ago

edo 17y ago
You will love this video about a businessman helping a homeless man by changing his sign. http://www.youtube.com/watch?v=zNLmrv7-6OY http://www.youtube.com/watch?v=zNLmrv7-6OY On another note: Hilarious idea to try and A/B test on a sign of a homeless man ;-).
brianimmel 17y ago
I often get too wrapped up in the idea of A/B testing as a digital concept. After reading I'm starting to think of all the "everyday" items that can be optimized with a little testing. That said, I am now researching a way to add hand sanitizer to my website.
jsdalton 17y ago
I enjoyed this, but the title and concept are a bit misleading. I was expecting something where he made iterative tweaks to the creative (i.e. the sign) and did continual comparisons of the success rate to see how different changes affected the result. All this guy did was take a homeless guy's sign and add a really cool marketing idea to it, which increased the homeless guy's take. Yes, the results of his changes were awesome, but it really had nothing to do with "A/B testing" per se.
- Splines 17y ago
  Would A/B testing encompass changes as large as what happened here? Having never done A/B testing, it sounds like an iterative process. Would continuous A/B testing ever converge on a solution like the one arrived here? I guess what I'm asking is this: Is A/B testing an evolutionary approach to developing a product? Are "revolutionary" leaps possible if you strictly follow the A/B testing methodology? (My guess: probably not, which implies that most people don't follow A/B exclusively, which sounds like common sense).
  - inerte 17y ago
    Yes, but not encouraged. The literature, or most common applications of A/B testing, is about little improvements. Phrases being changed, picture alignment, that kinda of thing. Of course, in theory, A/B as the name implies is about testing two different approaches, so again, in theory, any stuff completely different can be seen as an experiment for A/B. The problem is that humans have been trying different ways to do things for a loooooooooong time, and the whole point of trying to teach and learn A/B is to test things, measure, and walk towards your goal. So yes, you can "A/B test" two completely different things, but it will be so. much. harder to actually know what it worked. For example, you're A/B testing a bicycle and a Ferrari Testarossa. D'uh... obviously they'll work differently, but what caused? Price, color, speed, brand? That's why when A/B testing is mentioned it's about small, isolable, repeatable, incremental and evolutionary approach. Because it's a technique with this purpose in mind.
- ja27 17y ago
  I can't find it now, but I read a great article about different signs that panhandlers "fly". One guy flipped his sign depending on the age of the closest likely target. The old person side asked for help and had "God Bless You". The young person side had "I just need a beer". But the best I've seen was "I bet you can't hit me with a quarter".
edw519 17y ago
Nice story, but without any practical application to A/B Testing. Focus was changed from "This is about me" to "This is about you." No need for A/B testing - this will almost always result in significant improvement.
- charlesju 17y ago
  I agree that the blog post could have contained more information, perhaps some pretty graphs on the conversions per hour with sign 1 vs sign 2. Maybe taking 2 different signs, one with a red cup, one with a blue cup and A/B testing. But the underlying thesis is still valid. Testing multiple scenarios returns positive changes, and further showing an interesting application that I have never thought about -- helping the homeless with it.
- enjo 17y ago
  If you're truly data-driven, the yes there is an absolute need for A/B testing. The thing about behavioral psychology is that behavior can be quite different given different variables. I'm constantly surprised when I test things that I'm absolutely sure will work... that don't.
dylanz 17y ago
The hand sanitizer is an added cost that should definitely be factored into the test. Also, this isn't a straight A/B, but an A/Incentivized-B.
cryptnoob 17y ago
I was disappointed not to be informed of the experimental method. Did he normalize the data for each time period? (Dollars/Person within 20 feet), or is it unnormalized and we could just be looking at a situation where more people came later. How long were the time periods of each trial? How much better did he do at each trial, in terms of percentages? Also, since he apparently has the data, I'm dieing to know how much one of these guys does make per day.
andrewcooke 17y ago
am i the only person that finds this creepy? (yes, i understand that in some sense he's helping the homeless guy; no i cannot yet fully articulate why this bothers me).
- jonbishop 17y ago
  "Before you get all 'ugh, you are mocking a homeless man' on me, realize this... This experiment improved this bums ROI" I got the same feeling. It's because he doesn't treat the guy like a person; he never calls the homeless man by his name, Keith, but keeps referring to him as a bum.
  - physcab 17y ago
    I didn't get that feeling at all. It seemed like he referred to him as the "homeless man". Either way, I felt he focused more on the problem of A/B testing than making any denigrating comments about the homeless.
    - andrewcooke 17y ago
      if he was "making [...] denigrating comments about the homeless" then it would be obviously wrong. that's really not what we are discussing here - it's certainly not what i felt (i would have simply flamed away if that were the case). in general, saying "it's not [some extreme caricature of a position]" isn't much use in arguing against anything, and doesn't really help anyone much...
  - potatolicious 17y ago
    I feel like he did more harm than good. Panhandling, unless extraordinarily effective and coupled with a lack of substance abuse problems (all too common with the homeless) will not pull this man out of poverty for good, or even in the medium-term. Giving this man more money, but without coupling that with professional assistance, may do him a lot more harm than good. There's a reason why real academic studies have to pass through an ethics review, and I take issue with how cavalier this guy is about his study. If the author of the blog has experience working with the homeless, then ignore all I've said above. But otherwise, he's operating way outside his field of expertise involving people and situations he doesn't understand, and complexities he does not seem to be aware of. This is dangerous.
aresant 17y ago
What I took away from this: - always be testing, even (especially?) offline - repackaging your offer can = drastic result - incentives work Agree that lack of full data was disappointing.
hhjj 17y ago
One reason to do A/B testing is to collect data/facts. That article contains only opinions ie : "The biggest difference, is that we are now introducing a bribe" Trust data, not opinions.
kasterma 17y ago
"The first thing we always do is make slight modifications to what we already have." rather contrasts with "The next big difference is that we changed colors and went from cardboard to white to spark the interest of people walking by instead of automatically having negative associations that they have with cardboard and homeless people." The second can (and in the context of the article maybe should) be tested. The small change is to make exactly the same text on the different material.
roboneal 17y ago
This is hardly A-B testing...this is more barely readable sign with hidden cup to VASTLY improved sign with prominent red cup and BRIBE!
jasonlbaptiste 17y ago
when did max change his domain name?
00joe 17y ago
Forget the whole a/b testing thing. The real value of this blog post is that it shows a great way to utilize personal skills to help someone else. That's the real genius, out of the box thinking here.
rythie 17y ago
So basically: A) Homeless man begging B) Useful service with a variable price

136 points by aresant 17y ago

edo 17y ago

You will love this video about a businessman helping a homeless man by changing his sign. http://www.youtube.com/watch?v=zNLmrv7-6OY http://www.youtube.com/watch?v=zNLmrv7-6OY On another note: Hilarious idea to try and A/B test on a sign of a homeless man ;-).

brianimmel 17y ago

I often get too wrapped up in the idea of A/B testing as a digital concept. After reading I'm starting to think of all the "everyday" items that can be optimized with a little testing. That said, I am now researching a way to add hand sanitizer to my website.

jsdalton 17y ago

I enjoyed this, but the title and concept are a bit misleading. I was expecting something where he made iterative tweaks to the creative (i.e. the sign) and did continual comparisons of the success rate to see how different changes affected the result. All this guy did was take a homeless guy's sign and add a really cool marketing idea to it, which increased the homeless guy's take. Yes, the results of his changes were awesome, but it really had nothing to do with "A/B testing" per se.

Splines 17y ago
Would A/B testing encompass changes as large as what happened here? Having never done A/B testing, it sounds like an iterative process. Would continuous A/B testing ever converge on a solution like the one arrived here? I guess what I'm asking is this: Is A/B testing an evolutionary approach to developing a product? Are "revolutionary" leaps possible if you strictly follow the A/B testing methodology? (My guess: probably not, which implies that most people don't follow A/B exclusively, which sounds like common sense).
- inerte 17y ago
  Yes, but not encouraged. The literature, or most common applications of A/B testing, is about little improvements. Phrases being changed, picture alignment, that kinda of thing. Of course, in theory, A/B as the name implies is about testing two different approaches, so again, in theory, any stuff completely different can be seen as an experiment for A/B. The problem is that humans have been trying different ways to do things for a loooooooooong time, and the whole point of trying to teach and learn A/B is to test things, measure, and walk towards your goal. So yes, you can "A/B test" two completely different things, but it will be so. much. harder to actually know what it worked. For example, you're A/B testing a bicycle and a Ferrari Testarossa. D'uh... obviously they'll work differently, but what caused? Price, color, speed, brand? That's why when A/B testing is mentioned it's about small, isolable, repeatable, incremental and evolutionary approach. Because it's a technique with this purpose in mind.
ja27 17y ago
I can't find it now, but I read a great article about different signs that panhandlers "fly". One guy flipped his sign depending on the age of the closest likely target. The old person side asked for help and had "God Bless You". The young person side had "I just need a beer". But the best I've seen was "I bet you can't hit me with a quarter".

edw519 17y ago

Nice story, but without any practical application to A/B Testing. Focus was changed from "This is about me" to "This is about you." No need for A/B testing - this will almost always result in significant improvement.

charlesju 17y ago
I agree that the blog post could have contained more information, perhaps some pretty graphs on the conversions per hour with sign 1 vs sign 2. Maybe taking 2 different signs, one with a red cup, one with a blue cup and A/B testing. But the underlying thesis is still valid. Testing multiple scenarios returns positive changes, and further showing an interesting application that I have never thought about -- helping the homeless with it.
enjo 17y ago
If you're truly data-driven, the yes there is an absolute need for A/B testing. The thing about behavioral psychology is that behavior can be quite different given different variables. I'm constantly surprised when I test things that I'm absolutely sure will work... that don't.

dylanz 17y ago

The hand sanitizer is an added cost that should definitely be factored into the test. Also, this isn't a straight A/B, but an A/Incentivized-B.

cryptnoob 17y ago

I was disappointed not to be informed of the experimental method. Did he normalize the data for each time period? (Dollars/Person within 20 feet), or is it unnormalized and we could just be looking at a situation where more people came later. How long were the time periods of each trial? How much better did he do at each trial, in terms of percentages? Also, since he apparently has the data, I'm dieing to know how much one of these guys does make per day.

andrewcooke 17y ago

am i the only person that finds this creepy? (yes, i understand that in some sense he's helping the homeless guy; no i cannot yet fully articulate why this bothers me).

jonbishop 17y ago
"Before you get all 'ugh, you are mocking a homeless man' on me, realize this... This experiment improved this bums ROI" I got the same feeling. It's because he doesn't treat the guy like a person; he never calls the homeless man by his name, Keith, but keeps referring to him as a bum.
- physcab 17y ago
  I didn't get that feeling at all. It seemed like he referred to him as the "homeless man". Either way, I felt he focused more on the problem of A/B testing than making any denigrating comments about the homeless.
  - andrewcooke 17y ago
    if he was "making [...] denigrating comments about the homeless" then it would be obviously wrong. that's really not what we are discussing here - it's certainly not what i felt (i would have simply flamed away if that were the case). in general, saying "it's not [some extreme caricature of a position]" isn't much use in arguing against anything, and doesn't really help anyone much...
- potatolicious 17y ago
  I feel like he did more harm than good. Panhandling, unless extraordinarily effective and coupled with a lack of substance abuse problems (all too common with the homeless) will not pull this man out of poverty for good, or even in the medium-term. Giving this man more money, but without coupling that with professional assistance, may do him a lot more harm than good. There's a reason why real academic studies have to pass through an ethics review, and I take issue with how cavalier this guy is about his study. If the author of the blog has experience working with the homeless, then ignore all I've said above. But otherwise, he's operating way outside his field of expertise involving people and situations he doesn't understand, and complexities he does not seem to be aware of. This is dangerous.

aresant 17y ago

What I took away from this: - always be testing, even (especially?) offline - repackaging your offer can = drastic result - incentives work Agree that lack of full data was disappointing.

hhjj 17y ago

One reason to do A/B testing is to collect data/facts. That article contains only opinions ie : "The biggest difference, is that we are now introducing a bribe" Trust data, not opinions.

kasterma 17y ago

"The first thing we always do is make slight modifications to what we already have." rather contrasts with "The next big difference is that we changed colors and went from cardboard to white to spark the interest of people walking by instead of automatically having negative associations that they have with cardboard and homeless people." The second can (and in the context of the article maybe should) be tested. The small change is to make exactly the same text on the different material.

roboneal 17y ago

This is hardly A-B testing...this is more barely readable sign with hidden cup to VASTLY improved sign with prominent red cup and BRIBE!

jasonlbaptiste 17y ago

when did max change his domain name?

00joe 17y ago

Forget the whole a/b testing thing. The real value of this blog post is that it shows a great way to utilize personal skills to help someone else. That's the real genius, out of the box thinking here.

rythie 17y ago

So basically: A) Homeless man begging B) Useful service with a variable price