3 ms·
It's very easy to use the wrong statistical method or to do things with your data that you shouldn't, which causes you to misinterpret the output of statistical
by carbocation 16y ago
It's very easy to use the wrong statistical method or to do things with your data that you shouldn't, which causes you to misinterpret the output of statistical methods.
Which is more likely: (a) that this was the 1 in 1/(1-0.998) times that the correct method incorrectly rejected the null; (b) that there is actually a cryptic difference between A and B in the AB test; or (c) that the wrong method was used, or the right method was misused, or something along those lines?
I would say that those are arranged in increasing order of likelihood.
- ZeroMinx 16y agoAgreed. And that's why I liked this post -- it's highlighting the fact that it's easy to misread statistics.
- carbocation 16y agoAgreed. Also, sorry if it came across like I was disagreeing with your post; I meant to supplement it.
- StavrosK 16y agoThere wasn't much to go wrong... The A/B testing system renders one template in two ways depending on whether the user is in the test or control group, and the template was the same in both cases. If we had made a mistake, the results wouldn't normalise when we got more data, they would have just stayed there.
- carbocation 16y agoI'm saying that the mistake was overinterpreting the statistics without having a good handle on what they actually meant (part of option 'c' above). This comes off as ad hominem, but it's the most common error mode so I'd use this as a motivation to learn more stats and understand why the test was probably not telling you what you thought it was.