4 ms·
Amen to that. This is an issue we've been very aware of recently, and we're discussing various possibilities for mitigating. The point about using historical g
by kamens 14y ago
Amen to that. This is an issue we've been very aware of recently, and we're discussing various possibilities for mitigating.
The point about using historical graphs instead of single significance numbers reduces, but does not solve, the likelihood of making a mistake due to one of these chance results.
- btilly 14y agoHistorical graphs do not actually help you at all. The conditional probability of seeing a particular historical graph is fixed once you state the current number of observations and significance number. In particular the historical graph gives you no information about the true underlying probability that you did not already have if you know the number of observations and the current significance number.
- kamens 14y agoSay your current significance number currently claims 99% significance in favor of alternative A. A historical graph can show you that 12 hours ago, there was a stat sig in favor of alternative B (we've seen this plenty of times). Some metrics take a while to stabilize, and the more metrics you have, the more likely you are to run into these particular situations (as you mention above). A graph helps you understand recent variability in the metric.
- btilly 14y agoRecent variability shouldn't matter. Assuming you don't have have some external factor driving variability (like an email program), the only statistically important fact that past variability tells you you is an indirect estimate of how many people you have in the test. But that is a number that you already have direct access to. If this this does not seem true for you, then you need to review how you are doing your stats. Because something sounds fishy. Perhaps, for example, you are plugging in the number of times the target event happened instead of the number of people that the target event happened to? Then observations are correlated, which throws off your statistical tests? There are a lot of ways to do the stats wrong, and I like to comment that if you do, then you'll come up with wrong answers - and believe them.
- kamens 14y agoAgreed. You're absolutely right that it's an indirect estimate of how many people you have in the test and doesn't add any extra information. The only thing I'm suggesting is that looking at a graph that is wildly fluctuating up and down can be more helpful and easily accessible than asking someone who isn't an expert in stats (I raise my hand) to look at a number of participants and immediately understand its effect on this specific experiment's variability. No, we are tracking the number of people that the target event happened to -- but we've also tracked the number of times it happens, and as mentioned in the article, we are aware that these specific metrics are extremely outlier-prone and are looking into ways of improving this. All advice welcome, your tips are much appreciated.
- btilly 14y agoIf you're using a chi-square or its better relative, the g-test, then outliers do not matter. If you're using ANOVA or Student's T-test, don't do that. They assume that you're sampling from normal distributions and you are not. What you should do is truncate extreme outliers, then use a z-test. Start at http://elem.com/~btilly/effective-ab-testing/index.html#aslide94 http://elem.com/~btilly/effective-ab-testing/index.html#asli... and use the arrow keys to go through about 2 sections to for details on that. If you have any questions, shoot them to btilly@gmail.com.