4 ms·

Great list Ben! In my experience of analyzing and commenting on A/B test results of our (Visual Website Optimizer) customers, one of the most important effects

Great list Ben! In my experience of analyzing and commenting on A/B test results of our (Visual Website Optimizer) customers, one of the most important effects I have observed is the newness effect of variations. Visitors are sometimes inclined to respond positively to a new variation just because it's a change from plain, old boring control. Even though the newness effect fades in after a couple of days, but it makes our customers prone to celebrating early the success of their test. That is why we always recommending calculating a preset number of days for which to run the test before even looking at the result (we have a setting for that). And as a general rule of thumb, we ask our customers to at least wait for 7 days before concluding anything from the results.

btilly 14y ago
The newness effect can be a really confusing PITA. Yet another strategy to use for it for long-running tests is cohort analysis - analyze what people's actions were starting x days after entering the test. That said, if you turn it around, deliberately finding ways to use the newness effect to your advantage can be very rewarding.
- paraschopra 14y ago
  >That said, if you turn it around, deliberately finding ways to use the newness effect to your advantage can be very rewarding. I'm curious to hear more about this. How do you make use of the newness effect? Have you practiced this in the wild?
  - btilly 14y ago
    As I suggested in http://bentilly.blogspot.com/2012/09/ab-testing-vs-mab-algorithms-its.html http://bentilly.blogspot.com/2012/09/ab-testing-vs-mab-algor... you can use a MAB algorithm with a forgetting factor for this. I've actually done it with a more sophisticated system, but the idea is similar - boost whatever you have evidence is performing better right now. (You should put in logic to avoid time-based fluctuations in performance throwing you off.) The one that I built had minor teething pains, but it is now working quite well. And it is convenient to be able to every so often drop in new versions, and remove ones which have proven unable to perform up to snuff even with a newness effect, and just trust that it will Do The Right Thing.

btilly 14y ago

The newness effect can be a really confusing PITA. Yet another strategy to use for it for long-running tests is cohort analysis - analyze what people's actions were starting x days after entering the test. That said, if you turn it around, deliberately finding ways to use the newness effect to your advantage can be very rewarding.

paraschopra 14y ago
>That said, if you turn it around, deliberately finding ways to use the newness effect to your advantage can be very rewarding. I'm curious to hear more about this. How do you make use of the newness effect? Have you practiced this in the wild?
- btilly 14y ago
  As I suggested in http://bentilly.blogspot.com/2012/09/ab-testing-vs-mab-algorithms-its.html http://bentilly.blogspot.com/2012/09/ab-testing-vs-mab-algor... you can use a MAB algorithm with a forgetting factor for this. I've actually done it with a more sophisticated system, but the idea is similar - boost whatever you have evidence is performing better right now. (You should put in logic to avoid time-based fluctuations in performance throwing you off.) The one that I built had minor teething pains, but it is now working quite well. And it is convenient to be able to every so often drop in new versions, and remove ones which have proven unable to perform up to snuff even with a newness effect, and just trust that it will Do The Right Thing.