3 ms·

I wish I knew more about statistics. Punching just the two variants that made it to the end into a Chi square calculator gave me a 90% confidence, not the 95% h

I wish I knew more about statistics. Punching just the two variants that made it to the end into a Chi square calculator gave me a 90% confidence, not the 95% he cited. I also wonder about stopping the variation early, and even the conditions under which he stopped the test. http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html talks about pitfalls when "testing until you reach confidence". Are there any better references for these types of questions?

Datasta 16y ago
I cited 95% confidence, which was calculated by Visual Website Optimizer. I wonder how they calculate their confidence level.
- bermanoid 16y ago
  Never used Visual Website Optimizer, but chances are, whatever they do leaves you vulnerable to the flaw mentioned in the article that swalberg linked. Basically, every time you peek at the results of a split test to see if the confidence level is high enough, you make your split test more prone to error. There are ways to adjust for this which allow you to continuously monitor the test and stop if it hits a threshold, but it's much easier just to set a known number of samples to take and accept the confidence level that comes out then.
  - Datasta 16y ago
    Actually, I had planned to test 2,000 views from the outset. It went a little over because I didn't stop it fast enough. I do agree that you are vulnerable to influencing results if you stop the test as soon as you hit your confidence level
paraschopra 16y ago
Founder of Visual Website Optimizer here. Couple of articles you can refer on math we use: http://visualwebsiteoptimizer.com/split-testing-blog/what-you-really-need-to-know-about-mathematics-of-ab-split-testing/ http://visualwebsiteoptimizer.com/split-testing-blog/what-yo... http://visualwebsiteoptimizer.com/split-testing-blog/ab-testing-significance-calculator-spreadsheet-in-excel/ http://visualwebsiteoptimizer.com/split-testing-blog/ab-test...

Datasta 16y ago

I cited 95% confidence, which was calculated by Visual Website Optimizer. I wonder how they calculate their confidence level.

bermanoid 16y ago
Never used Visual Website Optimizer, but chances are, whatever they do leaves you vulnerable to the flaw mentioned in the article that swalberg linked. Basically, every time you peek at the results of a split test to see if the confidence level is high enough, you make your split test more prone to error. There are ways to adjust for this which allow you to continuously monitor the test and stop if it hits a threshold, but it's much easier just to set a known number of samples to take and accept the confidence level that comes out then.
- Datasta 16y ago
  Actually, I had planned to test 2,000 views from the outset. It went a little over because I didn't stop it fast enough. I do agree that you are vulnerable to influencing results if you stop the test as soon as you hit your confidence level

paraschopra 16y ago

Founder of Visual Website Optimizer here. Couple of articles you can refer on math we use: http://visualwebsiteoptimizer.com/split-testing-blog/what-you-really-need-to-know-about-mathematics-of-ab-split-testing/ http://visualwebsiteoptimizer.com/split-testing-blog/what-yo... http://visualwebsiteoptimizer.com/split-testing-blog/ab-testing-significance-calculator-spreadsheet-in-excel/ http://visualwebsiteoptimizer.com/split-testing-blog/ab-test...