3 ms·
Almost didn't click on this because the title seems (and is) clickbait-y, but it was actually a very useful read for me. As a founder, I'm constantly hearing a
by gnicholas 11y ago
Almost didn't click on this because the title seems (and is) clickbait-y, but it was actually a very useful read for me.
As a founder, I'm constantly hearing about A/B testing and how great these tools are. I'm not enough of a statistician to know whether everything in this article is true/valid (and would welcome a rebuttal), but the part about regression to the mean really hits home. Encouraging users to cut off testing too early means that you make them feel good ("Look, we had this huge difference!"), when in reality the difference is smaller/negligible.
I'll still do some A/B testing, but given our engineering/time constraints—and my inability to accurately vet the claims/conclusions of the testing software—I won't spend too much time on this.
- fharper1961 11y agoDeciding not to AB test because of this article would be a huge mistake. Learning from mistakes made by others, and avoiding them is what I would suggest as the take away. I work for a successful ($50M+ revenue) bootstrapped startup. And one of the reasons for the success is that AB testing became part of the company's culture, as soon as there was enough data coming in for the tests to become useful. AB testing is so important that we have built our own in-house framework that automatically gives results for our company specific KPIs.