4 ms·

I cited 95% confidence, which was calculated by Visual Website Optimizer. I wonder how they calculate their confidence level.

• bermanoid 16y ago
  Never used Visual Website Optimizer, but chances are, whatever they do leaves you vulnerable to the flaw mentioned in the article that swalberg linked. Basically, every time you peek at the results of a split test to see if the confidence level is high enough, you make your split test more prone to error. There are ways to adjust for this which allow you to continuously monitor the test and stop if it hits a threshold, but it's much easier just to set a known number of samples to take and accept the confidence level that comes out then.
  • Datasta 16y ago
    Actually, I had planned to test 2,000 views from the outset. It went a little over because I didn't stop it fast enough. I do agree that you are vulnerable to influencing results if you stop the test as soon as you hit your confidence level