What the software industry calls an "A/B test" is what scientists call a "randomized controlled trial",* and it's generally considered to be the best type of ex

What the software industry calls an "A/B test" is what scientists call a "randomized controlled trial",* and it's generally considered to be the best type of experiment you can do. The fact that people routinely screw up their experiments in the ways you list and others isn't an indictment of the A/B test methodology. * Well actually the term gets used to mean a lot of things but insofar as you can give it a meaningful definition, it means randomized controlled trial.