5 ms·

Evan these are incredible tools, thank you for contributing another brick in the wall for those of us that bleed A/B testing :) I would love to see Optimizely

by aresant 13y ago

Evan these are incredible tools, thank you for contributing another brick in the wall for those of us that bleed A/B testing :) I would love to see Optimizely and VWO embrace similarly non-ambiguous and functional reporting as a default. EG - just introducing Chi-Squared testing into a discussion with clients or teams that think that they're A/B testing properly by following Optimizely's graphs usually turns the discussion on it's head - "you mean there's a RANGE? well how can we be certain?" etc. Great work, thank you!

archildress 13y ago
Curious - for someone statistically ignorant like me, can you provide some detail on how we can use Evan's tools for split testing improvements? I guess the question I'm boiling this down to is... Why are graphs and comparisons of results that Optimizely or VWO produces not good enough?
- aresant 13y ago
  The short answer is that they don't provide much depth. I've seen Optimizely call something a "winner" with 95% confidence after 48hrs. The triangular method we use w/the off-the-shelfers is something like: a) Optimizely base stats b) Convergence point analysis (useful to correct for day-of-week / unique traffic swings) c) Chi-Squared Testing which provides a range so that you can actually assess the risk of a high-confidence test. eg look at the example in evan's tool which shows 8.5%-22% and 13.%-28.9%. This means that Sample 1 could be as HIGH as 22.1% conversion and Sample 2 could be as LOW as 13.3%. If this was rated as a high-confidence test that Sample 2 was rated higher than Sample 1 you could be potentially risking a signifigant conversion decrease if you went with Sample 2. EG needs more data and don't just buy into the "this one is better"
timr 13y ago
All of the tools used to have this sort of presentation. GWO, in particular, once had a very nice visualization of confidence-range overlaps (the current product blows). I suspect they removed them for the same reason I tend to avoid the subject when discussing testing with non-technical people: nearly everyone is numerically illiterate, and looking for the "easy" answer. They want a used-car salesman, not a mathematics professor (i.e. "no more of this 'confidence' gobbeldygook -- give me the bottom line"). Sad, but in my experience, nearly universally true.

by aresant 13y ago

archildress 13y ago

Curious - for someone statistically ignorant like me, can you provide some detail on how we can use Evan's tools for split testing improvements? I guess the question I'm boiling this down to is... Why are graphs and comparisons of results that Optimizely or VWO produces not good enough?

aresant 13y ago
The short answer is that they don't provide much depth. I've seen Optimizely call something a "winner" with 95% confidence after 48hrs. The triangular method we use w/the off-the-shelfers is something like: a) Optimizely base stats b) Convergence point analysis (useful to correct for day-of-week / unique traffic swings) c) Chi-Squared Testing which provides a range so that you can actually assess the risk of a high-confidence test. eg look at the example in evan's tool which shows 8.5%-22% and 13.%-28.9%. This means that Sample 1 could be as HIGH as 22.1% conversion and Sample 2 could be as LOW as 13.3%. If this was rated as a high-confidence test that Sample 2 was rated higher than Sample 1 you could be potentially risking a signifigant conversion decrease if you went with Sample 2. EG needs more data and don't just buy into the "this one is better"

timr 13y ago

All of the tools used to have this sort of presentation. GWO, in particular, once had a very nice visualization of confidence-range overlaps (the current product blows). I suspect they removed them for the same reason I tend to avoid the subject when discussing testing with non-technical people: nearly everyone is numerically illiterate, and looking for the "easy" answer. They want a used-car salesman, not a mathematics professor (i.e. "no more of this 'confidence' gobbeldygook -- give me the bottom line"). Sad, but in my experience, nearly universally true.