3 ms·

It's called A/B testing. You serve people one of two versions of a page at random, and test which performs better. With enough data, you can be pretty damn cert

by lmkg 17y ago

It's called A/B testing. You serve people one of two versions of a page at random, and test which performs better. With enough data, you can be pretty damn certain that the single change caused the results. It is, in fact, hard science. There are obviously a couple of caveats. First, you need to run the two tests in parallel as you allude to, and second the separation between the samples needs to be completely random. I would assume that Microsoft can keep these caveats in mind.

unblogworthy 17y ago
It think they'll need to do some A/B/A testing to be sure. Switch back to the original shade only for those that experienced the new shade. Maybe they'll see another multi-million dollar increase? That way you could further prove that it's indeed the color that made the difference and not simply the fact that the page was changed at all. Added: The Hawthorne effect is a form of reactivity whereby subjects improve an aspect of their behavior being experimentally measured simply in response to the fact that they are being studied, not in response to any particular experimental manipulation. http://en.wikipedia.org/wiki/Hawthorne_effect http://en.wikipedia.org/wiki/Hawthorne_effect
- bemmu 17y ago
  Personally I like to A/B test with new users. But I guess on an open website like Bing (instead of one with logins) you won't be able to really know if someone has visited it before.
axod 17y ago
It's not that simple. Suppose you ran an A/B test like you say. User 1 goes to website, it's site A. Dislikes it. Nothing happens. User 2 goes to website, it's site B. Loves it. Tells friends. Now how do you track that 'virality' element? How do you know if User 1 was the cause of increase traffic, or user 2? How do you know if I tell my friend "Hey checkout bing it has this awesome blue color", and how do you know if I was an A user or a B user? I think there's just too many untrackable elements in a case like this. And in any case, the story still sounds like there was no A/B testing involved. Just "we changed a color", and later, stuff happened.

by lmkg 17y ago

unblogworthy 17y ago

It think they'll need to do some A/B/A testing to be sure. Switch back to the original shade only for those that experienced the new shade. Maybe they'll see another multi-million dollar increase? That way you could further prove that it's indeed the color that made the difference and not simply the fact that the page was changed at all. Added: The Hawthorne effect is a form of reactivity whereby subjects improve an aspect of their behavior being experimentally measured simply in response to the fact that they are being studied, not in response to any particular experimental manipulation. http://en.wikipedia.org/wiki/Hawthorne_effect http://en.wikipedia.org/wiki/Hawthorne_effect

bemmu 17y ago
Personally I like to A/B test with new users. But I guess on an open website like Bing (instead of one with logins) you won't be able to really know if someone has visited it before.

axod 17y ago

It's not that simple. Suppose you ran an A/B test like you say. User 1 goes to website, it's site A. Dislikes it. Nothing happens. User 2 goes to website, it's site B. Loves it. Tells friends. Now how do you track that 'virality' element? How do you know if User 1 was the cause of increase traffic, or user 2? How do you know if I tell my friend "Hey checkout bing it has this awesome blue color", and how do you know if I was an A user or a B user? I think there's just too many untrackable elements in a case like this. And in any case, the story still sounds like there was no A/B testing involved. Just "we changed a color", and later, stuff happened.