Perhaps multi armed bandit algorithms can help here. They automatically balance testing to see which version is better with using the best version as much as po

Perhaps multi armed bandit algorithms can help here. They automatically balance testing to see which version is better with using the best version as much as possible. A multi armed bandit algorithm is a gambler with a number of levers at his disposal. He chooses which lever to pull and then receives a reward. In this case lever 1 is "show page version A" and lever 2 is "show page version B". The algorithms work so that they balance discovering which lever is best with pulling the best lever. Here's an example of a very simple algorithm. Record the average profit for page A and page B in two variables. Now with probability p (for example p=95%) choose the page with the highest average profit so far. With probability 1-p pick one at random. A more advanced algorithm could vary p over time so that it starts at 0% increases towards 100%. http://en.wikipedia.org/wiki/Multi-armed_bandit http://en.wikipedia.org/wiki/Multi-armed_bandit