3 ms·
It seems that the author isn't disaffected with A/B testing, but rather with improper implementations thereof. A/B testing is intended to be scientific, so it
by redthrowaway 15y ago
It seems that the author isn't disaffected with A/B testing, but rather with improper implementations thereof. A/B testing is intended to be scientific, so it must therefore adhere to rigorous scientific principles of validity. Sure, there are those who implement it incorrectly, but a properly implemented A/B test should always have a (large) control group, against whom you can measure the purported improvements. Such complaints as A/B testing inherently fostering incrementalism that leads away from proper design can be countered by implementing a "proper" control at each stage. Similarly, his complaint of A/B testing being a binary proposition ignores the 0/1/many axiom of good engineering. An A/B test that forces the user to choose between two unattractive options is a stupid test. An A/B test that shows many options to be unattractive is a good test with a dissatisfying result.
Rather than question the underlying validity of the methodology, the author would be better served by questioning the validity of its implementation in the instances he's encountered.