3 ms·
How many impressions were you running if you were able to measure a 2% difference?
by tobr 8y ago
How many impressions were you running if you were able to measure a 2% difference?
- chudi 8y agoIf you want to be completely sure, a really small amount, lets say your baseline is 20% to detect a 2% difference you just need 2 x 6,347 views for each test Check this tools[0] and this article [1] [0] https://www.evanmiller.org/ab-testing/sample-size.html#!20;80;5;2;0 https://www.evanmiller.org/ab-testing/sample-size.html#!20;8... [1] https://www.evanmiller.org/how-not-to-run-an-ab-test.html https://www.evanmiller.org/how-not-to-run-an-ab-test.html
- jermaustin1 8y agoWhen we implemented split-testing-by-default baked into the software itself, we saw maybe 1000 visitors a week from various sources. Our free conversion rate was at about 3% and once we were "fully optimized" we were closer to 10-15%, and our paid conversions were about 10-30% of the free users depending on the month (December and January always sucked). Most of these numbers were measured after at least a week of leaving the experiments running. We never tested more than experiment at a time per visitor, but our framework had the ability to run multiple tests simultaneously, while cookie-ing a visitor to a single test.