Even though most A/B testing tools will tell you a variation has "won" in a short period of time, I highly recommend running tests for longer periods of time (f

Even though most A/B testing tools will tell you a variation has "won" in a short period of time, I highly recommend running tests for longer periods of time (few days minimum, ideally a week). - Consumer/user behavior is different day to day, and often follows trends (like weariness to commit or purchase prior to holidays) - Even with statistical signifance, I've often seen the winner flip on a daily basis because of traffic shifts. - The tool doesn't know if your traffic sources are consistent (but you should!) These aren't concrete by any means, but neither is a short-ran multivariate test. If your winner will be up for over 16 hours, give it some more time to "bake" than that :)