3 ms·

I like this. Not very many people have studied probability, whereas a lot (relatively) have heard the good effects of A/B testing. I think A/B testing is a ve

I like this. Not very many people have studied probability, whereas a lot (relatively) have heard the good effects of A/B testing. I think A/B testing is a very good method, but you need a lot of data. I'd say, when you're just starting up, don't try A/B. You won't get the data you need. It's very easy to be seduced by statistics. It doesn't matter if the stats are wrong. EDIT: seduced, not deduced.

carbocation 16y ago
It's very easy to use the wrong statistical method or to do things with your data that you shouldn't, which causes you to misinterpret the output of statistical methods. Which is more likely: (a) that this was the 1 in 1/(1-0.998) times that the correct method incorrectly rejected the null; (b) that there is actually a cryptic difference between A and B in the AB test; or (c) that the wrong method was used, or the right method was misused, or something along those lines? I would say that those are arranged in increasing order of likelihood.
- ZeroMinx 16y ago
  Agreed. And that's why I liked this post -- it's highlighting the fact that it's easy to misread statistics.
  - carbocation 16y ago
    Agreed. Also, sorry if it came across like I was disagreeing with your post; I meant to supplement it.
- StavrosK 16y ago
  There wasn't much to go wrong... The A/B testing system renders one template in two ways depending on whether the user is in the test or control group, and the template was the same in both cases. If we had made a mistake, the results wouldn't normalise when we got more data, they would have just stayed there.
  - carbocation 16y ago
    I'm saying that the mistake was overinterpreting the statistics without having a good handle on what they actually meant (part of option 'c' above). This comes off as ad hominem, but it's the most common error mode so I'd use this as a motivation to learn more stats and understand why the test was probably not telling you what you thought it was.
StavrosK 16y ago
The worst part was that we actually did make decisions based on the confidence metric when it was as low as 95%, sometimes, and then realised that we shouldn't have (we should have waited for more data). This is what most people who talk about A/B testing don't mention, that you need more data than you think, in order to get good results.

carbocation 16y ago

It's very easy to use the wrong statistical method or to do things with your data that you shouldn't, which causes you to misinterpret the output of statistical methods. Which is more likely: (a) that this was the 1 in 1/(1-0.998) times that the correct method incorrectly rejected the null; (b) that there is actually a cryptic difference between A and B in the AB test; or (c) that the wrong method was used, or the right method was misused, or something along those lines? I would say that those are arranged in increasing order of likelihood.

ZeroMinx 16y ago
Agreed. And that's why I liked this post -- it's highlighting the fact that it's easy to misread statistics.
- carbocation 16y ago
  Agreed. Also, sorry if it came across like I was disagreeing with your post; I meant to supplement it.
StavrosK 16y ago
There wasn't much to go wrong... The A/B testing system renders one template in two ways depending on whether the user is in the test or control group, and the template was the same in both cases. If we had made a mistake, the results wouldn't normalise when we got more data, they would have just stayed there.
- carbocation 16y ago
  I'm saying that the mistake was overinterpreting the statistics without having a good handle on what they actually meant (part of option 'c' above). This comes off as ad hominem, but it's the most common error mode so I'd use this as a motivation to learn more stats and understand why the test was probably not telling you what you thought it was.

StavrosK 16y ago

The worst part was that we actually did make decisions based on the confidence metric when it was as low as 95%, sometimes, and then realised that we shouldn't have (we should have waited for more data). This is what most people who talk about A/B testing don't mention, that you need more data than you think, in order to get good results.