5 ms·

Microsoft attributes $80-$100 million dollars in revenue to a shade of blue

29 points by tbgvi 17y ago

mtarnovan 17y ago
"A specific color of blue (#0044CC) drove $80-$100 million dollars a year increase over the light blue the design team tried first." I call bullshit on this statement. There is no way they could have measured this reliably unless the only thing they changed for an entire year was the color of the links.
- nopassrecover 17y ago
  Unless they A/B tested the colour change.
  - mtarnovan 17y ago
    That's my point - you can't A/B test something while making other changes too.
    - nopassrecover 17y ago
      You could if those changes were replicated in both the colour A test and the colour B test?
- nandemo 17y ago
  Why would they have to change it for an entire year? Maybe it's enough to test it for 1 day, or 10 minutes for all I know. If it improved clicking rate and revenue by 9%(+-1%) and after 1 year they've got $1 billion of revenue, then you could say that 8~10% of that billion likely came from Magic Blue.
  - mtarnovan 17y ago
    You want to sample for 10 minutes or 1 day and extrapolate for 1 year ? The results would be meaningless. Even if you sample 1 continuous month you could have badly flawed results (for example you might have sampled only early adopters, which might behave differently; and that's just one example of the many, many factors that can make signal indistinguishable from noise). Now I'm sure that the guys working on Bing A/B testing know what they are doing, but I just can't believe a change in link color would bring a "$80-$100 million dollars a year increase" (from a total of ~2000 million $/year revenue which I extrapolate from this http://www.microsoft.com/msft/earnings/fy10/earn_rel_q2_10.mspx#income http://www.microsoft.com/msft/earnings/fy10/earn_rel_q2_10.m... but which includes all revenue of the Microsoft Online Service Business - I don't know how much of that is generated by Bing)
tbgvi 17y ago
These are notes from a presentation by Paul Ray. It's an hour long so I thought these would be better to digest. The $80 million color blue comment is down by "Data Driven Process". If you're interested in the full video presentation you can check it out here http://live.visitmix.com/MIX10/Sessions/CL06 http://live.visitmix.com/MIX10/Sessions/CL06
patio11 17y ago
This is the frightening reality of what happens when you have a large business where everything scales multiplicatively and then you make a trivial change which is net-beneficial. People want this to be impossible because it strikes you as intuitively wrong that five minutes of work could be worth eight figures. Reality doesn't care what you think, though. Relatedly, and more prosaicly, if you average one successful A/B test with a 5% lift per month, and you do nothing else, your business will grow at about 80% per year. Multiplicatively effective businesses are a wonderful place to be in life. Then I look over at Zynga at all where you can do multiplicative effectiveness within the context of a viral loop, which is just so effective I'm almost certain it goes against my religion somehow.
- fierarul 17y ago
  Don't take this the wrong way but on every comment I've read from you, there is "A/B" :-) Not sure how Zynga's effectivness might be against your religion, but "A/B" is part of you somehow: religiously or not.
  - scorciapino 17y ago
    test
  - MaysonL 17y ago
    A/B = the core of the scientific method.
    - fierarul 17y ago
      Not so sure about that... It looks to me that A/B leads quite easily to a local optimum. But it's better than no method at all.
- axod 17y ago
  That all sounds very deep for something I think it a lot simpler. 1. They tried some weird colors. Didn't work well 2. They thought "screw it" and copied the colors Google use. (Give or take) 3. Users feel more 'at home' and maybe even assume the results are from Google. You can't replicate that situation. You can't copy the colors Google uses again. I guess you can make Bing even more like Google, but only up to a point.
phoboslab 17y ago
Huh? The article makes no sense at all. I mean WTF?: "But something was troubling me with this number. In color theory the thing doesn’t work like that. In order something to be complete red + green + blue should be 255. But in this case we have (0 Red + 68 green +204blue = 272) And 272 <> 255."
- varikin 17y ago
  That also bothered me. Why should the math matter?
- gjm11 17y ago
  I think you meant for that to be a comment not on this item but on another one, pointing to some guy's rambling deranged blog entry about this allegedly very profitable colour change, full of numerology and nonsense. And yes, indeed, that one makes no sense at all. This one, on the other hand, seems reasonably coherent.
- radley 17y ago
  Each color goes from 0 to 255. It's not a sum.
dfranke 17y ago
They should name that color "Hooloovoo".
axod 17y ago
>> "Guess what? Microsoft also tested multiple versions of blue for links in their search results. A specific color of blue (#0044CC) drove $80-$100 million dollars a year increase over the light blue the design team tried first." You can't change a color, see things increase, and claim it's all down to a change of color. Also as bing "increased", Live "decreased". Has Bing been successful? It makes for a fun PR story, but it's complete BS. Whatever happened to science? How much money did Live search make? How much is Bing making? Have they been able to add any traffic? Some hard data would be interesting. One point to note is that they essentially copied the colors straight from Google.
- lmkg 17y ago
  It's called A/B testing. You serve people one of two versions of a page at random, and test which performs better. With enough data, you can be pretty damn certain that the single change caused the results. It is, in fact, hard science. There are obviously a couple of caveats. First, you need to run the two tests in parallel as you allude to, and second the separation between the samples needs to be completely random. I would assume that Microsoft can keep these caveats in mind.
  - unblogworthy 17y ago
    It think they'll need to do some A/B/A testing to be sure. Switch back to the original shade only for those that experienced the new shade. Maybe they'll see another multi-million dollar increase? That way you could further prove that it's indeed the color that made the difference and not simply the fact that the page was changed at all. Added: The Hawthorne effect is a form of reactivity whereby subjects improve an aspect of their behavior being experimentally measured simply in response to the fact that they are being studied, not in response to any particular experimental manipulation. http://en.wikipedia.org/wiki/Hawthorne_effect http://en.wikipedia.org/wiki/Hawthorne_effect
    - bemmu 17y ago
      Personally I like to A/B test with new users. But I guess on an open website like Bing (instead of one with logins) you won't be able to really know if someone has visited it before.
samd 17y ago
So they found the missing shade of blue!
jefarmstrong 17y ago
So the article says that #0044CC is the $80 million color. But if you check bing.com they're using #0033CC. So what's up? Maybe it's a trick to get everyone to switch to the obviously inferior #0044CC color while bing laughs all the way to the bank with the real killer color!! Bing - #0033CC Google - #2200CC DuckDuckGo - #3068DD Facebook - #3B5998 (what where THEY thinking? ;-)
- teamonkey 17y ago
  The point isn't that there's a magic colour, it's that they A/B split tested the new colour against the old one and saw an increase in the number of people clicking. It doesn't mean the process stops right there.
  - jefarmstrong 17y ago
    Yes, I was being sarcastic. :-)
    - teamonkey 17y ago
      I know, but I was being pedantic :)
      - OmarIsmail 17y ago
        and I'm being tautalogical :)
deleted 17y ago
[deleted]
motters 17y ago
Think I'll stick with DuckDuckGo.
Jim72 17y ago
Interesting that the url of the search result is also green, like Google. I just opened the two sites side-by-side and their results use mostly the same color scheme: Blue Title Black description Green URL I would definitely say MS went with a the "familiarity factor."
jafl5272 17y ago
Oh, good. They found the hyper-intelligent shade of blue from Hitchhiker's Guide to the Galaxy. (Actually, Google does the same kind of color testing.)
vais 17y ago
In case anyone's interested in pedantic details, the color appears to be #0033CC (not #0044CC) and it can best be described as STRONG VIOLET (according to the Color Curious iPhone app).

29 points by tbgvi 17y ago

"A specific color of blue (#0044CC) drove $80-$100 million dollars a year increase over the light blue the design team tried first." I call bullshit on this statement. There is no way they could have measured this reliably unless the only thing they changed for an entire year was the color of the links.

nopassrecover 17y ago
Unless they A/B tested the colour change.
- mtarnovan 17y ago
  That's my point - you can't A/B test something while making other changes too.
  - nopassrecover 17y ago
    You could if those changes were replicated in both the colour A test and the colour B test?
nandemo 17y ago
Why would they have to change it for an entire year? Maybe it's enough to test it for 1 day, or 10 minutes for all I know. If it improved clicking rate and revenue by 9%(+-1%) and after 1 year they've got $1 billion of revenue, then you could say that 8~10% of that billion likely came from Magic Blue.
- mtarnovan 17y ago
  You want to sample for 10 minutes or 1 day and extrapolate for 1 year ? The results would be meaningless. Even if you sample 1 continuous month you could have badly flawed results (for example you might have sampled only early adopters, which might behave differently; and that's just one example of the many, many factors that can make signal indistinguishable from noise). Now I'm sure that the guys working on Bing A/B testing know what they are doing, but I just can't believe a change in link color would bring a "$80-$100 million dollars a year increase" (from a total of ~2000 million $/year revenue which I extrapolate from this http://www.microsoft.com/msft/earnings/fy10/earn_rel_q2_10.mspx#income http://www.microsoft.com/msft/earnings/fy10/earn_rel_q2_10.m... but which includes all revenue of the Microsoft Online Service Business - I don't know how much of that is generated by Bing)

tbgvi 17y ago

These are notes from a presentation by Paul Ray. It's an hour long so I thought these would be better to digest. The $80 million color blue comment is down by "Data Driven Process". If you're interested in the full video presentation you can check it out here http://live.visitmix.com/MIX10/Sessions/CL06 http://live.visitmix.com/MIX10/Sessions/CL06

patio11 17y ago

This is the frightening reality of what happens when you have a large business where everything scales multiplicatively and then you make a trivial change which is net-beneficial. People want this to be impossible because it strikes you as intuitively wrong that five minutes of work could be worth eight figures. Reality doesn't care what you think, though. Relatedly, and more prosaicly, if you average one successful A/B test with a 5% lift per month, and you do nothing else, your business will grow at about 80% per year. Multiplicatively effective businesses are a wonderful place to be in life. Then I look over at Zynga at all where you can do multiplicative effectiveness within the context of a viral loop, which is just so effective I'm almost certain it goes against my religion somehow.

fierarul 17y ago
Don't take this the wrong way but on every comment I've read from you, there is "A/B" :-) Not sure how Zynga's effectivness might be against your religion, but "A/B" is part of you somehow: religiously or not.
- scorciapino 17y ago
  test
- MaysonL 17y ago
  A/B = the core of the scientific method.
  - fierarul 17y ago
    Not so sure about that... It looks to me that A/B leads quite easily to a local optimum. But it's better than no method at all.
axod 17y ago
That all sounds very deep for something I think it a lot simpler. 1. They tried some weird colors. Didn't work well 2. They thought "screw it" and copied the colors Google use. (Give or take) 3. Users feel more 'at home' and maybe even assume the results are from Google. You can't replicate that situation. You can't copy the colors Google uses again. I guess you can make Bing even more like Google, but only up to a point.

phoboslab 17y ago

Huh? The article makes no sense at all. I mean WTF?: "But something was troubling me with this number. In color theory the thing doesn’t work like that. In order something to be complete red + green + blue should be 255. But in this case we have (0 Red + 68 green +204blue = 272) And 272 <> 255."

varikin 17y ago
That also bothered me. Why should the math matter?
gjm11 17y ago
I think you meant for that to be a comment not on this item but on another one, pointing to some guy's rambling deranged blog entry about this allegedly very profitable colour change, full of numerology and nonsense. And yes, indeed, that one makes no sense at all. This one, on the other hand, seems reasonably coherent.
radley 17y ago
Each color goes from 0 to 255. It's not a sum.

dfranke 17y ago

They should name that color "Hooloovoo".

axod 17y ago

>> "Guess what? Microsoft also tested multiple versions of blue for links in their search results. A specific color of blue (#0044CC) drove $80-$100 million dollars a year increase over the light blue the design team tried first." You can't change a color, see things increase, and claim it's all down to a change of color. Also as bing "increased", Live "decreased". Has Bing been successful? It makes for a fun PR story, but it's complete BS. Whatever happened to science? How much money did Live search make? How much is Bing making? Have they been able to add any traffic? Some hard data would be interesting. One point to note is that they essentially copied the colors straight from Google.

lmkg 17y ago
It's called A/B testing. You serve people one of two versions of a page at random, and test which performs better. With enough data, you can be pretty damn certain that the single change caused the results. It is, in fact, hard science. There are obviously a couple of caveats. First, you need to run the two tests in parallel as you allude to, and second the separation between the samples needs to be completely random. I would assume that Microsoft can keep these caveats in mind.
- unblogworthy 17y ago
  It think they'll need to do some A/B/A testing to be sure. Switch back to the original shade only for those that experienced the new shade. Maybe they'll see another multi-million dollar increase? That way you could further prove that it's indeed the color that made the difference and not simply the fact that the page was changed at all. Added: The Hawthorne effect is a form of reactivity whereby subjects improve an aspect of their behavior being experimentally measured simply in response to the fact that they are being studied, not in response to any particular experimental manipulation. http://en.wikipedia.org/wiki/Hawthorne_effect http://en.wikipedia.org/wiki/Hawthorne_effect
  - bemmu 17y ago
    Personally I like to A/B test with new users. But I guess on an open website like Bing (instead of one with logins) you won't be able to really know if someone has visited it before.

samd 17y ago

So they found the missing shade of blue!

jefarmstrong 17y ago

So the article says that #0044CC is the $80 million color. But if you check bing.com they're using #0033CC. So what's up? Maybe it's a trick to get everyone to switch to the obviously inferior #0044CC color while bing laughs all the way to the bank with the real killer color!! Bing - #0033CC Google - #2200CC DuckDuckGo - #3068DD Facebook - #3B5998 (what where THEY thinking? ;-)

teamonkey 17y ago
The point isn't that there's a magic colour, it's that they A/B split tested the new colour against the old one and saw an increase in the number of people clicking. It doesn't mean the process stops right there.
- jefarmstrong 17y ago
  Yes, I was being sarcastic. :-)
  - teamonkey 17y ago
    I know, but I was being pedantic :)
    - OmarIsmail 17y ago
      and I'm being tautalogical :)

deleted 17y ago

[deleted]

motters 17y ago

Think I'll stick with DuckDuckGo.

Jim72 17y ago

Interesting that the url of the search result is also green, like Google. I just opened the two sites side-by-side and their results use mostly the same color scheme: Blue Title Black description Green URL I would definitely say MS went with a the "familiarity factor."

jafl5272 17y ago

Oh, good. They found the hyper-intelligent shade of blue from Hitchhiker's Guide to the Galaxy. (Actually, Google does the same kind of color testing.)

vais 17y ago

In case anyone's interested in pedantic details, the color appears to be #0033CC (not #0044CC) and it can best be described as STRONG VIOLET (according to the Color Curious iPhone app).