3 ms·

What's the minimum effect size you consider worthwhile?

• patio11 17y ago
  Do you mean "How positive does it have to be for you to not revert it when you end the test?" I'd collect 2% improvements all day long if they looked statistically significant. That being said, my mental goal is to run one test a week, and to collect 5% improvements on half of my tests. The miracle of compound interest means that, if I sustained that for a year, I'd increase sales by 250%. You can see my stats and tell that I've never had a year over year improvement that good, but now since the cost of A/B testing just got muuuuuuuch cheaper for me, I'm optimistic.
  • trapper 17y ago
    Yes. Effect size is the difference in means in proportion to the pooled standard deviation to the two treatments, and is a great measure of how large the difference is. All groups are different given large enough n. Whether that difference means anything is dependent on the effect size. When the effect size is large you know that whatever effect you are seeing is much less likely to be chance. It's also less susceptible to large differences in variation, which the tests you are using to test significance use as an assumption (variation is equal).