4 ms·
There are a couple important biases where just changing something results in a positive result. * http://en.wikipedia.org/wiki/Novelty_effect http://en.wikiped
by sunir 15y ago
There are a couple important biases where just changing something results in a positive result.
* http://en.wikipedia.org/wiki/Novelty_effect http://en.wikipedia.org/wiki/Novelty_effect (everyone loves seeing something new!)
* http://en.wikipedia.org/wiki/Hawthorne_effect http://en.wikipedia.org/wiki/Hawthorne_effect (being a subject of an experiment is motivational)
There are a couple ways to deal with these biases. One is to try a large number of different designs, so the bias is spread across a large number of options. Thus the comparative winner is more likely to itself be intrinsically better, at least amongst the new designs. Note that this still negatively punishes the control.
The other way is to wait longer until the new design is no longer 'new'. In practice, that is impractical! Really, all you can do is pick the winner and keep monitoring it after the fact to see if there is a regression in the result.
I know others have said this, but you can harness these biases. Simply change the design frequently for no reason other than to stay 'fresh'. It keeps people interested and it shows you're alive and kicking.
- chc 15y agoThe Hawthorne effect probably wouldn't apply since it requires the subject to know he's the subject of an experiment. The novelty of the new designs might be a factor, but since we're measuring new sign-ups, it seems kind of unlikely that a lot of people were familiar enough with the design of the page to realize it was new.
- jcampbell1 15y agoIt is highly likely that people know they are looking at a new design. People don't just adopt a CRM on a whim. For my site, (foreign language learning), the customer visits an average of 6 times before buying. New designs drop the visit count temporarily, a clear sign the novelty effect is strong.