4 ms·
Traditional A/B testing has very poor ergonomics. Experimenters are usually put in awkward conflict-of-interest situations that create multiple strong incentiv
by cle 8y ago
Traditional A/B testing has very poor ergonomics. Experimenters are usually put in awkward conflict-of-interest situations that create multiple strong incentives not to perform rigorous, disciplined, valid experiments.
Null hypothesis significance testing is fundamentally misaligned with business needs and is not a good tool for businesses. This is true in many fields of science as well, but at least they have some mechanisms that try to ensure that experiments are unbiased. Businesses often don't have the same internal and external incentives that lead to those mechanisms, and so NHST is abused even more.
- havkom 8y agoWell, I have had well run experiments which showed that the “currently internally hyped” way of doing things is completely inferior to the “old boring” and inexpensive way. The project leader was extremely hyped about doing the experiment to prove the superiority of the new hyped way. When he got the results though, it was clear that this was not to be talked about and this result would not be presented to his superiors.
- tzahola 8y ago“We’re a data-driven organization! [as long as the data fits our agenda]” It’s one of my favorite methodologies, next to “agile waterfall” and “holocracy with managers, middle-managers and minibosses”.