3 ms·

There isn't any downside. But if the test costs are negligible and you can afford to run a test for a week longer, it is always great to do so. I have seen too

There isn't any downside. But if the test costs are negligible and you can afford to run a test for a week longer, it is always great to do so. I have seen too many tests where confidence level after touching >95% came back to 70% or so once extended the test. An even better way is to do a follow up test where you do an A/B test where both variations are red. And if you see enough variance in that test, then I don't think you should take results seriously. When you are testing it is always better to try proving a hypothesis wrong rather than trying to prove it right. EDIT: clarified some parts.

jules 16y ago
Perhaps multi armed bandit algorithms can help here. They automatically balance testing to see which version is better with using the best version as much as possible. A multi armed bandit algorithm is a gambler with a number of levers at his disposal. He chooses which lever to pull and then receives a reward. In this case lever 1 is "show page version A" and lever 2 is "show page version B". The algorithms work so that they balance discovering which lever is best with pulling the best lever. Here's an example of a very simple algorithm. Record the average profit for page A and page B in two variables. Now with probability p (for example p=95%) choose the page with the highest average profit so far. With probability 1-p pick one at random. A more advanced algorithm could vary p over time so that it starts at 0% increases towards 100%. http://en.wikipedia.org/wiki/Multi-armed_bandit http://en.wikipedia.org/wiki/Multi-armed_bandit