4 ms·

Can split testing be trusted? No, not if you misunderstand the most basic concepts. Non-statically significant results are, surprise surprise, not statically si

by almost 16y ago

Can split testing be trusted? No, not if you misunderstand the most basic concepts. Non-statically significant results are, surprise surprise, not statically significant.

hartror 16y ago
This is exactly the problem with people implementing split testing and they're hurting their business. They hear about it, run off and implement the a/b versions of their pages and add some tracking. Proceeding then to draw incorrect conclusions and make bad decisions based on them. They don't go read up on it enough to understand the statistics behind it. I used my stats book from Uni (I knew it had a better use than a laptop stand, a java book now has that honour) but there are plenty of tutorials out there. For example: http://visualwebsiteoptimizer.com/split-testing-blog/what-you-really-need-to-know-about-mathematics-of-ab-split-testing/ http://visualwebsiteoptimizer.com/split-testing-blog/what-yo... It really is a toss up between statistics and medicine for which is the most misunderstood of the sciences.
- ramit 16y ago
  Unfortunately, nobody wants to learn about statistics -- they just want it to work so they can make more money. I just wrote a post about behavioral change and understanding your users here: http://www.iwillteachyoutoberich.com/blog/why-personal-finance-experts-continue-writing-worthless-advice/ http://www.iwillteachyoutoberich.com/blog/why-personal-finan... That's why sites like Visual Website Optimizer are getting increasingly good. They use plain language to explain when to keep running the tests. Again: It's unrealistic to expect lay users to learn statistics. They don't want to become statistical experts, they usually just want to increase sales. This is where software can help.
  - hartror 16y ago
    I think even laymen should learn the why of the statistics, a few paragraphs about statistical significance would have helped the article's author I should think. As an aside, wow another one of my favourite bloggers responding to a comment of mine on HN this week!
paraschopra 16y ago
Even statistics can lie. Having statistically significant results does not guarantee you that the variation will work better than the control. What it actually guarantees is that it is HIGHLY likely that variation will work better than the control. So, it ultimately boils down to what are the stakes involved in an A/B test.
- almost 16y ago
  Not just highly likely, likely to specified probability. Typically 95% or 99%. If you want to be more certain (and have the quantity of data) you can look for 99.9% or greater. I'm not sure how you could see that as statistics "lying" :) EDIT: I just noticed your username and I realise that you probably already understand this :p I still think it's a bit of an odd way of putting it though.

by almost 16y ago

Can split testing be trusted? No, not if you misunderstand the most basic concepts. Non-statically significant results are, surprise surprise, not statically significant.

hartror 16y ago

This is exactly the problem with people implementing split testing and they're hurting their business. They hear about it, run off and implement the a/b versions of their pages and add some tracking. Proceeding then to draw incorrect conclusions and make bad decisions based on them. They don't go read up on it enough to understand the statistics behind it. I used my stats book from Uni (I knew it had a better use than a laptop stand, a java book now has that honour) but there are plenty of tutorials out there. For example: http://visualwebsiteoptimizer.com/split-testing-blog/what-you-really-need-to-know-about-mathematics-of-ab-split-testing/ http://visualwebsiteoptimizer.com/split-testing-blog/what-yo... It really is a toss up between statistics and medicine for which is the most misunderstood of the sciences.

ramit 16y ago
Unfortunately, nobody wants to learn about statistics -- they just want it to work so they can make more money. I just wrote a post about behavioral change and understanding your users here: http://www.iwillteachyoutoberich.com/blog/why-personal-finance-experts-continue-writing-worthless-advice/ http://www.iwillteachyoutoberich.com/blog/why-personal-finan... That's why sites like Visual Website Optimizer are getting increasingly good. They use plain language to explain when to keep running the tests. Again: It's unrealistic to expect lay users to learn statistics. They don't want to become statistical experts, they usually just want to increase sales. This is where software can help.
- hartror 16y ago
  I think even laymen should learn the why of the statistics, a few paragraphs about statistical significance would have helped the article's author I should think. As an aside, wow another one of my favourite bloggers responding to a comment of mine on HN this week!

paraschopra 16y ago

Even statistics can lie. Having statistically significant results does not guarantee you that the variation will work better than the control. What it actually guarantees is that it is HIGHLY likely that variation will work better than the control. So, it ultimately boils down to what are the stakes involved in an A/B test.

almost 16y ago
Not just highly likely, likely to specified probability. Typically 95% or 99%. If you want to be more certain (and have the quantity of data) you can look for 99.9% or greater. I'm not sure how you could see that as statistics "lying" :) EDIT: I just noticed your username and I realise that you probably already understand this :p I still think it's a bit of an odd way of putting it though.