A crucial element left out of the discussion with A/B testing is the importance of user segmenting. You'll see drastic swings by the day of the week, male / fe

A crucial element left out of the discussion with A/B testing is the importance of user segmenting. You'll see drastic swings by the day of the week, male / female users, traffic sources, entrance paths, browser, region, etc. A/B tests can be terribly misleading without awareness of such factors. A simple illustration of this is eMail response rates which show that Sunday's are far and away the best day to get people's attention (highest aggregate open / click rate by 20% over worst day of week). But inversly we've found through the experience of managing thousands of tests that while opens & clicks are high, purchase conversion rates are on the low end of the scale. Just food for thought - it's absolutely positively rolling up your sleeves and deep diving into the data and segmenting tests or at least being aware of external influences before you take your results to the bank. http://www.mailermailer.com/resources/metrics/daily-rates.rwp http://www.mailermailer.com/resources/metrics/daily-rates.rw... http://conversionvoodoo.com/blog/2010/07/personalizing-your-email-subjects-can-drop-your-conversion-rate/ http://conversionvoodoo.com/blog/2010/07/personalizing-your-...