5 ms·

If you do an A/B test and get a result that leans one way but is statistically insignificant, then it seems to me you might as well just go with that answer. No

If you do an A/B test and get a result that leans one way but is statistically insignificant, then it seems to me you might as well just go with that answer. No, you don't have good reason to believe that it's better than the alternative, but you do have good reason to believe that the choice is harmless.

btilly 14y ago
My planned second article will be entirely on that theme. :-)
prosa 14y ago
> you do have good reason to believe that the choice is harmless. The issue you will run into here is that 95% confidence means that you will only have a false positive 5% of the time. It does not mean a neutral finding is 95% likely to be neutral. The lever that controls that is statistical power, which is oft-ignored in conversations about A/B testing. Most statisticians use 80% power, which means a full 20% of neutral findings were false negatives.
- gburt 14y ago
  Which is still better than 50%. After a split test, your point estimate, whether significant or not, is still your best guess of the effect.
- mistercow 14y ago
  That is very true. However, you do have much better reason to believe that the option is neutral than you have to believe that the option is beneficial. In an example like the one given in the article, you also likely have enough statistical power to be reasonably confident that the option is close to neutral, so if you're making a negative decision, it's probably not strongly negative.
  - prosa 14y ago
    Right. And you can also ratchet up the statistical power you want, at the cost of increased sample size requirements.
darkxanthos 14y ago
This doesn't seem correct to me. The null hypothesis is that both treatments are the same. If you can't find evidence to reject the null hypothesis why would you change it? From experience I've learned this puts you back in the gut-based guessing game those employing these methods are trying to escape.

btilly 14y ago

My planned second article will be entirely on that theme. :-)

prosa 14y ago

> you do have good reason to believe that the choice is harmless. The issue you will run into here is that 95% confidence means that you will only have a false positive 5% of the time. It does not mean a neutral finding is 95% likely to be neutral. The lever that controls that is statistical power, which is oft-ignored in conversations about A/B testing. Most statisticians use 80% power, which means a full 20% of neutral findings were false negatives.

gburt 14y ago
Which is still better than 50%. After a split test, your point estimate, whether significant or not, is still your best guess of the effect.
mistercow 14y ago
That is very true. However, you do have much better reason to believe that the option is neutral than you have to believe that the option is beneficial. In an example like the one given in the article, you also likely have enough statistical power to be reasonably confident that the option is close to neutral, so if you're making a negative decision, it's probably not strongly negative.
- prosa 14y ago
  Right. And you can also ratchet up the statistical power you want, at the cost of increased sample size requirements.

darkxanthos 14y ago

This doesn't seem correct to me. The null hypothesis is that both treatments are the same. If you can't find evidence to reject the null hypothesis why would you change it? From experience I've learned this puts you back in the gut-based guessing game those employing these methods are trying to escape.