3 ms·
It takes a lot of data to get good A/B test results and you can't run all these tests at the same time and in parallel. A lot of newer startups simply don't hav
by jonathanjaeger 13y ago
It takes a lot of data to get good A/B test results and you can't run all these tests at the same time and in parallel. A lot of newer startups simply don't have the traffic or data available to optimize like crazy. I think the key is just to focus on your core metrics, like engagement for a consumer product or whatever your metric is for a B2B company (contracts, revenue, etc.).
- usujason 13y agoI've worked with many startups to help them build a culture of data-driven decision making and I'll say from my experience that you don't need massive data to run highly valuable A/B tests. Also, with proper segmentation techniques, you can run many tests in parallel.
- jonathanjaeger 13y agoFair enough. My experience is more on A/B testing high trafficked paid lead gen pages, and I understand you can still make conclusions with less data (doesn't need to be massive). But I do agree with the author of the original post that it can be easy to worry about all these different A/B tests that aren't that illuminating in the grand scheme of things when you haven't even got a solid product.
- usujason 13y agoYep, we agree there, the advice of the author is spot on. I think too many startups, and large companies for that matter, think optimization is going to fix all the problems that are inherent in their product/business by just blindly slaping optimization vendor A's tags on the site and money just falls from the sky. Just doesn't happen like that.