I completely agree. We went through all of this stuff as our A/B testing was evolving at Thumbtack. They seem so simple, but the deeper you get you realize th

I completely agree. We went through all of this stuff as our A/B testing was evolving at Thumbtack. They seem so simple, but the deeper you get you realize they are only simple if you don't "cheat", which everybody does. And you'll only realize you are cheating in the first place if you have somebody who is maniacal about testing. If you create a culture where positive A/B tests are lauded (which is good!), then you create a lot of people who want A/B tests to finish in positive ways. For those people, it doesn't really matter if their A/B test actually improves things, only if it looks like it does. This isn't nefarious, this is just human nature, but it creates a lot of creativity and energy at finding ways to making winning A/B tests. We'd have people run a test, where their new variation got off to a bad start, then say "oh, it was a bug", then they'd fix some irrelevant thing and start over just to reset the counters. I was guilty of it sometimes. You get excited about your tests and want them to win. That is why it is critical either to have gatekeepers like an analytics team to keep you honest or have a really specific protocol on how your company runs tests and only consider results of tests that followed the protocol.