Though simple A/B tests are better than not doing any tests at all, the true power of testing is realized when you are doing targeted tests. In most of the solu

Though simple A/B tests are better than not doing any tests at all, the true power of testing is realized when you are doing targeted tests. In most of the solutions currently in the market, you can only do blanket tests: that is, all visitors see the same test. Much better approach would be to do tests for a segment of visitors because it is quite likely that your organic traffic would have different optima than your direct traffic. Similarly, repeat visitors would respond differently than first time visitors. Also, websites usually have multiple goals. For example, while your A/B test may tell you that you have reached optima as far as clicks on the banner is concerned, you won't know if you have compromised on other goals such as visitor engagement, newsletter signups, etc. The point here is that you must measure performance on multiple goals to see the correct tradeoff. My team has been developing a powerful testing solution called Wingify (http://www.wingify.com/ http://www.wingify.com/). We have limited number of private beta accounts available. If anyone is interested, mail me at paras@wingify.com Oh, and BTW, there is a screencast of using Wingify for setting up an experiment as well. You might find it helpful for an introduction to A/B testing - http://www.wingify.com/video-demos/create_experiment/ http://www.wingify.com/video-demos/create_experiment/