4 ms·

"A specific color of blue (#0044CC) drove $80-$100 million dollars a year increase over the light blue the design team tried first." I call bullshit on this st

"A specific color of blue (#0044CC) drove $80-$100 million dollars a year increase over the light blue the design team tried first." I call bullshit on this statement. There is no way they could have measured this reliably unless the only thing they changed for an entire year was the color of the links.

nopassrecover 17y ago
Unless they A/B tested the colour change.
- mtarnovan 17y ago
  That's my point - you can't A/B test something while making other changes too.
  - nopassrecover 17y ago
    You could if those changes were replicated in both the colour A test and the colour B test?
nandemo 17y ago
Why would they have to change it for an entire year? Maybe it's enough to test it for 1 day, or 10 minutes for all I know. If it improved clicking rate and revenue by 9%(+-1%) and after 1 year they've got $1 billion of revenue, then you could say that 8~10% of that billion likely came from Magic Blue.
- mtarnovan 17y ago
  You want to sample for 10 minutes or 1 day and extrapolate for 1 year ? The results would be meaningless. Even if you sample 1 continuous month you could have badly flawed results (for example you might have sampled only early adopters, which might behave differently; and that's just one example of the many, many factors that can make signal indistinguishable from noise). Now I'm sure that the guys working on Bing A/B testing know what they are doing, but I just can't believe a change in link color would bring a "$80-$100 million dollars a year increase" (from a total of ~2000 million $/year revenue which I extrapolate from this http://www.microsoft.com/msft/earnings/fy10/earn_rel_q2_10.mspx#income http://www.microsoft.com/msft/earnings/fy10/earn_rel_q2_10.m... but which includes all revenue of the Microsoft Online Service Business - I don't know how much of that is generated by Bing)

4 ms·

"A specific color of blue (#0044CC) drove $80-$100 million dollars a year increase over the light blue the design team tried first." I call bullshit on this st

by mtarnovan 17y ago

nopassrecover 17y ago
Unless they A/B tested the colour change.
- mtarnovan 17y ago
  That's my point - you can't A/B test something while making other changes too.
  - nopassrecover 17y ago
    You could if those changes were replicated in both the colour A test and the colour B test?
nandemo 17y ago
Why would they have to change it for an entire year? Maybe it's enough to test it for 1 day, or 10 minutes for all I know. If it improved clicking rate and revenue by 9%(+-1%) and after 1 year they've got $1 billion of revenue, then you could say that 8~10% of that billion likely came from Magic Blue.
- mtarnovan 17y ago
  You want to sample for 10 minutes or 1 day and extrapolate for 1 year ? The results would be meaningless. Even if you sample 1 continuous month you could have badly flawed results (for example you might have sampled only early adopters, which might behave differently; and that's just one example of the many, many factors that can make signal indistinguishable from noise). Now I'm sure that the guys working on Bing A/B testing know what they are doing, but I just can't believe a change in link color would bring a "$80-$100 million dollars a year increase" (from a total of ~2000 million $/year revenue which I extrapolate from this http://www.microsoft.com/msft/earnings/fy10/earn_rel_q2_10.mspx#income http://www.microsoft.com/msft/earnings/fy10/earn_rel_q2_10.m... but which includes all revenue of the Microsoft Online Service Business - I don't know how much of that is generated by Bing)