10 ms·
But is that really true? Is there some reason why A/B testing necessarily means "testing small, incremental changes"? It seems to me that once you've got your
by michael_dorfman 16y ago
But is that really true? Is there some reason why A/B testing necessarily means "testing small, incremental changes"? It seems to me that once you've got your buggy, you may want to test what kind of things should power it-- horses, rabbits, internal combustion engines, windmills, etc.,-- and see what performs best.
- jacquesm 16y agoComing up with large changes to test is a lot more based on inspiration and gaining new knowledge than the kind of stuff that I see people testing in AB tests in general. Sure you could test large changes too (and I have) but AB testing works best when comparing apples with better apples. The biggest problem in testing diverse possible solutions to a problem is that you probably will have to restate the metrics by which you evaluate your total offering (the 'fitness'). You could compare cars and horse drawn buggies on a hundred fronts or more and what is 'best' depends more on circumstance than on the defining elements of both solutions, they are too different to be compared meaningfully except for their general utility. In that case the market seems to be best at deciding what is better but how would you as a buggy manufacturer test your newly minted car against your buggies without an infrastructure to support your car? It's not always that easy.