4 ms·

Curious - for someone statistically ignorant like me, can you provide some detail on how we can use Evan's tools for split testing improvements? I guess the qu

• aresant 13y ago
  The short answer is that they don't provide much depth. I've seen Optimizely call something a "winner" with 95% confidence after 48hrs. The triangular method we use w/the off-the-shelfers is something like: a) Optimizely base stats b) Convergence point analysis (useful to correct for day-of-week / unique traffic swings) c) Chi-Squared Testing which provides a range so that you can actually assess the risk of a high-confidence test. eg look at the example in evan's tool which shows 8.5%-22% and 13.%-28.9%. This means that Sample 1 could be as HIGH as 22.1% conversion and Sample 2 could be as LOW as 13.3%. If this was rated as a high-confidence test that Sample 2 was rated higher than Sample 1 you could be potentially risking a signifigant conversion decrease if you went with Sample 2. EG needs more data and don't just buy into the "this one is better"