4 ms·
One underused but valuable use of A/B testing is to safely roll out new features--even if you aren't really interested in optimization. I've seen many cases wh
by cle 11y ago
One underused but valuable use of A/B testing is to safely roll out new features--even if you aren't really interested in optimization. I've seen many cases where A/B testing has caught bugs that would have cost the company a significant amount of money and customer trust.
The overall utility of A/B tests is a complex topic and largely depends on the skill of the experimenter. Well-designed experiments are extraordinarily useful. A/B tests are one tool for product development, but they can't replace it.
- jacalata 11y agoI wouldn't call that A/B testing. It looks pretty similar under the hood but you'd normally call that a phased or gradual rollout and have different goals and metrics for it.
- ampersandy 11y agoAs you said, what cle described is a gradual rollout. You can do both at the same time though -- simply start your test on X% of the population and then roll it out to more people as you run your test. This is also a great way to run long-term holdouts.