This is our goal with E2E tests and monitoring — write tests that fail if and only if customers are experiencing issues — and I don’t know how to achieve that l

This is our goal with E2E tests and monitoring — write tests that fail if and only if customers are experiencing issues — and I don’t know how to achieve that level of I-can-sleep-well-at-night assurance of quality without it. We run our E2E tests continuously against prod from a different cloud region. As you wrote, the challenge is that there are literally 100 things that could cause a test to fail (some of which are outside your control), and as your team scales you’ll have to get smart about how to efficiently dispatch people to fix problems.