7 ms·

How Obama Raised $60 Million by Running a Simple Experiment

211 points by dsiroker 16y ago

run4yourlives 16y ago
The best thing about this article is that it is a giant ad. Informative, educational and useful, but an ad nonetheless. Wonderful marketing and great article as well.
- rodericksilva 16y ago
  It was educational and informative. It wasn't until you mentioned it as being an ad that I went back to see who actually wrote the article. You increased it's effectiveness by focusing on it...and now even further. ;)
  - run4yourlives 16y ago
    I don't mind that to be honest, because it's a product that I may end up using. It provides a real value, just like the ad did. :-)
- gallerytungsten 16y ago
  Indeed. I thought it was particularly clever how he apparently used the campaign data, but ran it through his product to get the "experiment results" screen shots.
  - natrius 16y ago
    I assumed that was Google Website Optimizer or whatever they actually used back in 2008. If not, they should make that clearer.
- zazi 16y ago
  Most company blogs are giant ads. What matters though is this article still provides value despite being an ad. I agree that this is wonderful marketing.
- danielnicollet 16y ago
  This post is an ad and at the same time an effective demonstration of the usefulness of advanced multivariate A/B tests. This dual purpose content/ad is very relevant to this community because we care about usability and good UIs. So while being a self-promoting piece it is also a well written tutorial of sorts.
Shamiq 16y ago
Glad they were able to actually write a short case-study on such a high worth (I'm assuming) case. That's real transparency.
- sudont 16y ago
  http://www.designing-obama.com/ http://www.designing-obama.com/ => I snagged the PDF when it was still free. EDIT: Still free. See responses. VSA Partners on the Logo: Part One: http://www.youtube.com/watch?v=etEP1Bhgui0 http://www.youtube.com/watch?v=etEP1Bhgui0 Part Two: http://www.youtube.com/watch?v=ukIMW833EPE http://www.youtube.com/watch?v=ukIMW833EPE There's a ton of info out there on Obama's campaign, it's really one of the best branding and marketing success stories.
  - arfrank 16y ago
    The PDF is still available here: http://digital.designing-obama.com/designing_obama_ed1.pdf http://digital.designing-obama.com/designing_obama_ed1.pdf
  - bjonathan 16y ago
    The PDF is still free: http://digital.designing-obama.com/ http://digital.designing-obama.com/ You can also read "Barack Obama Social Media Toolkit by Edelman" : http://www.scribd.com/full/10807015?access_key=key-nc75wovez6hgum15t6 http://www.scribd.com/full/10807015?access_key=key-nc75wovez...
maxklein 16y ago
"That would have been a huge mistake since it turns out that all of the videos did worse than all of the images." It annoys me no end when websites have videos explaining their product and no equivalent text. I hate watching videos about products. Always make the video optional and secondary.
- robryan 16y ago
  I like when an image can turn into a video when clicked on, that way it acts as another screen for explanation but also a video if I want to watch it.
- ryanwaggoner 16y ago
  Yes, always. Assuming your audience is maxklein. Otherwise, best to test.
  - maxklein 16y ago
    Obamas audience was massive. Millions and millions of people. I believe that's representative of the general taste of people.
    - ryanwaggoner 16y ago
      True, but extrapolating this single test with a particular audience looking for a particular type of information and being presented with these particular images and videos out to all websites is probably not the wisest thing.
      - maxklein 16y ago
        Of course, that's true, but we can't test EVERYTHING. If a very large test has been made with a very general population, then I think one should just make the assumption that will work similarly on your audience. If we start testing everything, we'll just be stuck testing the whole time! There are an infinite number of things to test, if there is a large and credible study showing one particular thing to be true, and no "anti" study, then I will just accept it and not retest it. Rather, I test things unique to my product or presentation. Running a test, particularly a multivariate test is not an easy or quick task. It requires a lot to do it properly, and if I had to do hundreds of them, I'd be wasting a lot of time. Most times as a small company, one should not be focusing on if a button is red or green, but on if you are at all in the right business. Such small-scale and detailed testing is really a way to stick your head in the sand, when larger optimisations or corrections could lead to a much much bigger jump.
ddlatham 16y ago
He seems to assume that the contribution and volunteer rate of the added marginal people (those who signed up after seeing the best performing media / button, but would not have signed up with the default media / button) is as good as the rest of the people (those who would have signed up regardless).
- kenjackson 16y ago
  He doesn't mention it, but that's easy to determine. Simply see what the average contribution of people who contributed during the A/B testing was per image/button combination. If there not all about the same then there may be some effect (negative or positive) that the added marginal people brought. If they're about the same, I think its a decent assumption to believe that it will be the same going forward. And of course, you can simply validate it by seeing if it is the same going forward.
  - dsiroker 16y ago
    Simply see what the average contribution of people who contributed during the A/B testing was per image/button combination We didn't measure the average contribution for each variation for this particular experiment but in future splash page tests we measured eventual $ per pageview for each variation and it was very closely correlated to % signup rate.
waterside81 16y ago
Kind of a side note, a complete noob when it comes to A/B, I tried out Optimizely after reading this and I've gotta say, this thing is put together well. I think I created 6 variations for our home page in about 2 minutes, saved it, and started watching results flow in. Amazing execution.
- WillyF 16y ago
  I completely agree. I just started using Optimizely today, and the ability to make a small change instantly across all pages on your site is extremely valuable.
Semiapies 16y ago
"Sending email to people who signed up on our splash page and asking them to volunteer typically converted 10% of them into volunteers." Is it wrong for me to be a little startled by that conversion rate? I wouldn't have expected anything that high.
- kgermino 16y ago
  Is it wrong for me to be a little startled by that conversion rate? I wouldn't have expected anything that high. It depends on how much of a commitment they're asking from volunteers. If you've already signed up on the website it's not much more to volunteer to pass out fliers at your office. If they want you to work a benefit or something than ya I'd agree it's a little high. Keep in mind too that these are people who have already actively signed on to support the campaign.
  - Semiapies 16y ago
    Good points, though the best-testing design was the one that didn't phrase signing up as a commitment, but as an opportunity to "Learn More", hence my surprise.
    - kgermino 16y ago
      That's a good point and when I was reading the article I actually predicted that the "Learn More" button would win because it was the least committal but I never drew the connection between that and volunteer signups. A few theories: 1) Maybe the feeling of not committing by pressing the button made people more comfortable signing up to be a volunteer, figuring that it probably wouldn't be too much of a commitment 2) Maybe the content on the Learn More page was just so good that the only thing that really mattered was getting more people onto that page and they would be comfortable 3) Perhaps the type of person more likely to click after seeing the picture of the family (Stay at home moms perhaps) was more likely to have time/desire to volunteer. As I think about it, assuming that one of these ideas is right which is probably not the case, I would guess that it's #3. I don't know the demographics of the people who volunteered but around me it seems like it would be the moms who go to the site, and they would probably be more likely to be skewed toward clicking with the family and non-committal button, while also being more likely to have the time to volunteer. Disclaimer: This is pure, assumption based, uninformed speculation of the worst kind.
mmaunder 16y ago
$400/month for 200K visits? The pricing seems to be based on how much they'll contribute to your bottom line rather than what the market will bear, what competitors are charging, whether they should focus on growth rather than early profit, etc. This might be a great opportunity for someone to clone the business and charge 25% of their prices, which will still be $250/month for 1 million visits.
- patio11 16y ago
  You think the opportunity to be the cheap, crappy competitor with an audience of technically inclined poor people is attractive? I think that is terrible positioning. If you charged $50, that would still be a whole lot of ramen, and you'd lose your customers to the first still cheaper, still crappier alternative or to free options. (There exist compelling free options for A/B testing if one is technically inclined.) $400 is not a lot of money when you probably demonstrate increases in revenue. (It is a fraction of BCCs gains from A/B testing. Nationwide political campaigns, Fortune 500 companies, and real estate firms all have slightly more money to spend than BCC does.)
guelo 16y ago
The "Learn More" button and this whole page are spammy. I remember being annoyed by this splash page, I just wanted to get to the main site. When users saw the "Sign Up" or "Join Us" buttons they probably realized it was an email phishing form and then searched for the hidden "Continue to Website" link. The "Learn More" button otoh seems to be what you are looking for, an escape, but when the button complained about the missing email some users probably went ahead and entered it even though they didn't want to because it seemed like the only way into the site.
- kinofcain 16y ago
  Precisely. It doesn't take very many A/B tests on lead-gen forms to realize you can get much better conversion rates if you're willing to trick your users into clicking on stuff. "Learn More" isn't a better title for that button, it's a deliberately misleading title for that button. You can have a button labelled "Get a free puppy" and a button labelled "Spam me every 24 hours and sell my e-mail address to the highest bidder". Guess which one will convert better; Guess which one is honest.
lkrubner 16y ago
The experiment showed that video was a poor choice for Obama. Images worked better for him. However, I do not think anyone should generalize from this example. Obama was supported by specific demographics. The poor and working classes favored Obama to a disproportionate degree. These are the people most likely to still be on dial-up Internet connections. There are the people with the least access to broad band. Therefore, these are the people for whom video would be a poor choice.
- colinplamondon 16y ago
  A majority of folks earning 200k+/year went for Obama, as well.
  - lkrubner 16y ago
    How many people in the USA make over $200k a year? And how many people in the USA can be classified as poor or working class? If you make $200K a year, then you are in the upper 1% of income in the USA. When I said "poor and working classes" I was thinking of the bottom 2 quintiles of income distribution. I think this is a fairly common reference. Certainly the usage is common in articles that write about the demographics of elections. You realize there are more votes in the lower 40% than in the top 1%?
- lkrubner 16y ago
  I wonder why this was downvoted? What part of this do people disagree with? I made 3 main points: 1.) Obama had disproportionate support from the poor and working classes. This is easy to verify, there were many news organizations that covered this. Here is one example: http://geocommons.com/overlays/6095 http://geocommons.com/overlays/6095 2.) The poor and working classes have less access to broadband Internet. Is anyone seriously going to dispute this? 3.) Therefore video was a poor choice for Obama, since many of his supporters lacked broadband Internet access. My friends, instead of downvoting, why don't you explain what you are thinking? Does someone disagree with my thesis? I feel like I'm stating something fairly obvious here.
  - 16y ago
jonathanjaeger 16y ago
For any of those interested in seeing the Mixergy interview with Dan Siroker (founder of Optimizely), check out this link: http://mixergy.com/dan-siroker-optimizely-interview/ http://mixergy.com/dan-siroker-optimizely-interview/

211 points by dsiroker 16y ago

The best thing about this article is that it is a giant ad. Informative, educational and useful, but an ad nonetheless. Wonderful marketing and great article as well.

rodericksilva 16y ago
It was educational and informative. It wasn't until you mentioned it as being an ad that I went back to see who actually wrote the article. You increased it's effectiveness by focusing on it...and now even further. ;)
- run4yourlives 16y ago
  I don't mind that to be honest, because it's a product that I may end up using. It provides a real value, just like the ad did. :-)
gallerytungsten 16y ago
Indeed. I thought it was particularly clever how he apparently used the campaign data, but ran it through his product to get the "experiment results" screen shots.
- natrius 16y ago
  I assumed that was Google Website Optimizer or whatever they actually used back in 2008. If not, they should make that clearer.
zazi 16y ago
Most company blogs are giant ads. What matters though is this article still provides value despite being an ad. I agree that this is wonderful marketing.
danielnicollet 16y ago
This post is an ad and at the same time an effective demonstration of the usefulness of advanced multivariate A/B tests. This dual purpose content/ad is very relevant to this community because we care about usability and good UIs. So while being a self-promoting piece it is also a well written tutorial of sorts.

Shamiq 16y ago

Glad they were able to actually write a short case-study on such a high worth (I'm assuming) case. That's real transparency.

sudont 16y ago
http://www.designing-obama.com/ http://www.designing-obama.com/ => I snagged the PDF when it was still free. EDIT: Still free. See responses. VSA Partners on the Logo: Part One: http://www.youtube.com/watch?v=etEP1Bhgui0 http://www.youtube.com/watch?v=etEP1Bhgui0 Part Two: http://www.youtube.com/watch?v=ukIMW833EPE http://www.youtube.com/watch?v=ukIMW833EPE There's a ton of info out there on Obama's campaign, it's really one of the best branding and marketing success stories.
- arfrank 16y ago
  The PDF is still available here: http://digital.designing-obama.com/designing_obama_ed1.pdf http://digital.designing-obama.com/designing_obama_ed1.pdf
- bjonathan 16y ago
  The PDF is still free: http://digital.designing-obama.com/ http://digital.designing-obama.com/ You can also read "Barack Obama Social Media Toolkit by Edelman" : http://www.scribd.com/full/10807015?access_key=key-nc75wovez6hgum15t6 http://www.scribd.com/full/10807015?access_key=key-nc75wovez...

maxklein 16y ago

"That would have been a huge mistake since it turns out that all of the videos did worse than all of the images." It annoys me no end when websites have videos explaining their product and no equivalent text. I hate watching videos about products. Always make the video optional and secondary.

robryan 16y ago
I like when an image can turn into a video when clicked on, that way it acts as another screen for explanation but also a video if I want to watch it.
ryanwaggoner 16y ago
Yes, always. Assuming your audience is maxklein. Otherwise, best to test.
- maxklein 16y ago
  Obamas audience was massive. Millions and millions of people. I believe that's representative of the general taste of people.
  - ryanwaggoner 16y ago
    True, but extrapolating this single test with a particular audience looking for a particular type of information and being presented with these particular images and videos out to all websites is probably not the wisest thing.
    - maxklein 16y ago
      Of course, that's true, but we can't test EVERYTHING. If a very large test has been made with a very general population, then I think one should just make the assumption that will work similarly on your audience. If we start testing everything, we'll just be stuck testing the whole time! There are an infinite number of things to test, if there is a large and credible study showing one particular thing to be true, and no "anti" study, then I will just accept it and not retest it. Rather, I test things unique to my product or presentation. Running a test, particularly a multivariate test is not an easy or quick task. It requires a lot to do it properly, and if I had to do hundreds of them, I'd be wasting a lot of time. Most times as a small company, one should not be focusing on if a button is red or green, but on if you are at all in the right business. Such small-scale and detailed testing is really a way to stick your head in the sand, when larger optimisations or corrections could lead to a much much bigger jump.

ddlatham 16y ago

He seems to assume that the contribution and volunteer rate of the added marginal people (those who signed up after seeing the best performing media / button, but would not have signed up with the default media / button) is as good as the rest of the people (those who would have signed up regardless).

kenjackson 16y ago
He doesn't mention it, but that's easy to determine. Simply see what the average contribution of people who contributed during the A/B testing was per image/button combination. If there not all about the same then there may be some effect (negative or positive) that the added marginal people brought. If they're about the same, I think its a decent assumption to believe that it will be the same going forward. And of course, you can simply validate it by seeing if it is the same going forward.
- dsiroker 16y ago
  Simply see what the average contribution of people who contributed during the A/B testing was per image/button combination We didn't measure the average contribution for each variation for this particular experiment but in future splash page tests we measured eventual $ per pageview for each variation and it was very closely correlated to % signup rate.

waterside81 16y ago

Kind of a side note, a complete noob when it comes to A/B, I tried out Optimizely after reading this and I've gotta say, this thing is put together well. I think I created 6 variations for our home page in about 2 minutes, saved it, and started watching results flow in. Amazing execution.

WillyF 16y ago
I completely agree. I just started using Optimizely today, and the ability to make a small change instantly across all pages on your site is extremely valuable.

Semiapies 16y ago

"Sending email to people who signed up on our splash page and asking them to volunteer typically converted 10% of them into volunteers." Is it wrong for me to be a little startled by that conversion rate? I wouldn't have expected anything that high.

kgermino 16y ago
Is it wrong for me to be a little startled by that conversion rate? I wouldn't have expected anything that high. It depends on how much of a commitment they're asking from volunteers. If you've already signed up on the website it's not much more to volunteer to pass out fliers at your office. If they want you to work a benefit or something than ya I'd agree it's a little high. Keep in mind too that these are people who have already actively signed on to support the campaign.
- Semiapies 16y ago
  Good points, though the best-testing design was the one that didn't phrase signing up as a commitment, but as an opportunity to "Learn More", hence my surprise.
  - kgermino 16y ago
    That's a good point and when I was reading the article I actually predicted that the "Learn More" button would win because it was the least committal but I never drew the connection between that and volunteer signups. A few theories: 1) Maybe the feeling of not committing by pressing the button made people more comfortable signing up to be a volunteer, figuring that it probably wouldn't be too much of a commitment 2) Maybe the content on the Learn More page was just so good that the only thing that really mattered was getting more people onto that page and they would be comfortable 3) Perhaps the type of person more likely to click after seeing the picture of the family (Stay at home moms perhaps) was more likely to have time/desire to volunteer. As I think about it, assuming that one of these ideas is right which is probably not the case, I would guess that it's #3. I don't know the demographics of the people who volunteered but around me it seems like it would be the moms who go to the site, and they would probably be more likely to be skewed toward clicking with the family and non-committal button, while also being more likely to have the time to volunteer. Disclaimer: This is pure, assumption based, uninformed speculation of the worst kind.

mmaunder 16y ago

$400/month for 200K visits? The pricing seems to be based on how much they'll contribute to your bottom line rather than what the market will bear, what competitors are charging, whether they should focus on growth rather than early profit, etc. This might be a great opportunity for someone to clone the business and charge 25% of their prices, which will still be $250/month for 1 million visits.

patio11 16y ago
You think the opportunity to be the cheap, crappy competitor with an audience of technically inclined poor people is attractive? I think that is terrible positioning. If you charged $50, that would still be a whole lot of ramen, and you'd lose your customers to the first still cheaper, still crappier alternative or to free options. (There exist compelling free options for A/B testing if one is technically inclined.) $400 is not a lot of money when you probably demonstrate increases in revenue. (It is a fraction of BCCs gains from A/B testing. Nationwide political campaigns, Fortune 500 companies, and real estate firms all have slightly more money to spend than BCC does.)

guelo 16y ago

The "Learn More" button and this whole page are spammy. I remember being annoyed by this splash page, I just wanted to get to the main site. When users saw the "Sign Up" or "Join Us" buttons they probably realized it was an email phishing form and then searched for the hidden "Continue to Website" link. The "Learn More" button otoh seems to be what you are looking for, an escape, but when the button complained about the missing email some users probably went ahead and entered it even though they didn't want to because it seemed like the only way into the site.

kinofcain 16y ago
Precisely. It doesn't take very many A/B tests on lead-gen forms to realize you can get much better conversion rates if you're willing to trick your users into clicking on stuff. "Learn More" isn't a better title for that button, it's a deliberately misleading title for that button. You can have a button labelled "Get a free puppy" and a button labelled "Spam me every 24 hours and sell my e-mail address to the highest bidder". Guess which one will convert better; Guess which one is honest.

lkrubner 16y ago

The experiment showed that video was a poor choice for Obama. Images worked better for him. However, I do not think anyone should generalize from this example. Obama was supported by specific demographics. The poor and working classes favored Obama to a disproportionate degree. These are the people most likely to still be on dial-up Internet connections. There are the people with the least access to broad band. Therefore, these are the people for whom video would be a poor choice.

colinplamondon 16y ago
A majority of folks earning 200k+/year went for Obama, as well.
- lkrubner 16y ago
  How many people in the USA make over $200k a year? And how many people in the USA can be classified as poor or working class? If you make $200K a year, then you are in the upper 1% of income in the USA. When I said "poor and working classes" I was thinking of the bottom 2 quintiles of income distribution. I think this is a fairly common reference. Certainly the usage is common in articles that write about the demographics of elections. You realize there are more votes in the lower 40% than in the top 1%?
lkrubner 16y ago
I wonder why this was downvoted? What part of this do people disagree with? I made 3 main points: 1.) Obama had disproportionate support from the poor and working classes. This is easy to verify, there were many news organizations that covered this. Here is one example: http://geocommons.com/overlays/6095 http://geocommons.com/overlays/6095 2.) The poor and working classes have less access to broadband Internet. Is anyone seriously going to dispute this? 3.) Therefore video was a poor choice for Obama, since many of his supporters lacked broadband Internet access. My friends, instead of downvoting, why don't you explain what you are thinking? Does someone disagree with my thesis? I feel like I'm stating something fairly obvious here.
- 16y ago

jonathanjaeger 16y ago

For any of those interested in seeing the Mixergy interview with Dan Siroker (founder of Optimizely), check out this link: http://mixergy.com/dan-siroker-optimizely-interview/ http://mixergy.com/dan-siroker-optimizely-interview/