The idea is that these kinds of sources of error will be noise, since they should be random with respect to experimental assignment in any kind of A/B test. In

The idea is that these kinds of sources of error will be noise, since they should be random with respect to experimental assignment in any kind of A/B test. In a small sample you could imagine chance imbalances, but given that I'm experimentally assigning someone to see either a thumbnail of John Krasinski or Steve Carell, it's unclear why the "I misclicked" scenario is going to have a higher rate in one rather than the other. And experiments at large companies like Netflix are almost infinitely powered. I would think a bigger threat to this approach would be being stuck in a local maxima.