I agree. I once worked for a company that wrote a blog post on reinventing and upgrading A/B tests, but the ugly truth was that the company couldn’t run an A/B

I agree. I once worked for a company that wrote a blog post on reinventing and upgrading A/B tests, but the ugly truth was that the company couldn’t run an A/B test to save its life. Every A/B test was a disaster, there were fragmented different frameworks for A/B tests in different parts of the product, with inconsistent and unreliable data, and the core clickstream ingestion system that was the foundation for any possible way of testing would crash and go down for hours every few weeks, and product management would just silently ignore any effects of missing data or correlation between failures in different product features. Even though we had a team of statistical researchers, they were treated as if legit stats 101 concerns were just ivory tower academic hair splitting and often were silently omitted from being in A/B test design, recap or decision meetings, which were instead run by product managers with no stats training. I remember interviewing a candidate once who said he was excited about the role because of that A/B testing blog post, and I just thought - geez what a completely soulless bait and switch ploy.