2 ms·
Agreed. You're absolutely right that it's an indirect estimate of how many people you have in the test and doesn't add any extra information. The only thing I'
by kamens 14y ago
Agreed. You're absolutely right that it's an indirect estimate of how many people you have in the test and doesn't add any extra information.
The only thing I'm suggesting is that looking at a graph that is wildly fluctuating up and down can be more helpful and easily accessible than asking someone who isn't an expert in stats (I raise my hand) to look at a number of participants and immediately understand its effect on this specific experiment's variability.
No, we are tracking the number of people that the target event happened to -- but we've also tracked the number of times it happens, and as mentioned in the article, we are aware that these specific metrics are extremely outlier-prone and are looking into ways of improving this. All advice welcome, your tips are much appreciated.
- btilly 14y agoIf you're using a chi-square or its better relative, the g-test, then outliers do not matter. If you're using ANOVA or Student's T-test, don't do that. They assume that you're sampling from normal distributions and you are not. What you should do is truncate extreme outliers, then use a z-test. Start at http://elem.com/~btilly/effective-ab-testing/index.html#aslide94 http://elem.com/~btilly/effective-ab-testing/index.html#asli... and use the arrow keys to go through about 2 sections to for details on that. If you have any questions, shoot them to btilly@gmail.com.