There's that, and also all of the common pitfalls highlighted in this famous paper: https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.00201

There's that, and also all of the common pitfalls highlighted in this famous paper: https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo... I do believe "doing A/B testing" is probably better than "not doing A/B testing", more often than not, but I think non-statisticians are usually way too comfortable with their knowledge (or lack thereof). And I have very little faith in the vast majority of A/B experiments run by people who don't know much about stats.