3 ms·

So, I'm in the privileged position of having an extremely large cluster of machines dedicated to running my team's tests. Even with that, e2e tests do take long

So, I'm in the privileged position of having an extremely large cluster of machines dedicated to running my team's tests. Even with that, e2e tests do take longer to run than unit tests. All our product's thousands of presubmit tests (millions of test cases plus random testing) might take fifteen to twenty minutes to run. The post-submits take even longer. On the other hand, fixing a bug that makes it to production and is not caught by tests takes longer still, and additionally hurts user trust. On balance, I can't recommend forgoing thorough end to end testing just to decrease test run time. Spend more money on resources to run your tests if you must, but don't skimp on end to end testing. There is no substitute.

Sodman 6y ago
I agree with you overall, in an ideal world we all have near unlimited compute power and even these "longer" e2e tests run in a few minutes, not hours. Unfortunately it's also not terribly uncommon to see these e2e suites take many hours, or sometimes even days to run. I think the happy medium might be using a dependency-aware testing pipeline that only runs tests for areas affected by the current commit's changes. I've seen Bazel used in this kind of setup with some success, for example. Ideally no bugs make it to production, but that's almost never the reality. Obviously it'll depend on your software's use-case and user demographics, but you could erode user-trust just as bad by having a trivial user-facing bug sit visible on your app/site for a few days or weeks because your CI/CD pipeline takes so long to run that it prevents you from pushing out timely releases.
- asdfasgasdgasdg 6y ago
  Yeah, another aspect here is optimizing tests properly. Of course, this also takes engineering time, but if you get your performance culture in the right place, hopefully test run costs can be commensurate to the amount of value you're delivering. That will allow you to never have a situation where your tests take inordinate amounts of time because you cannot afford resources to run them. Everything is tradeoffs, of course. You have to decide what's right for your business. But I think it's rare that there is too much or too thorough of testing. I'm guessing most orgs err in the other direction.

Sodman 6y ago

I agree with you overall, in an ideal world we all have near unlimited compute power and even these "longer" e2e tests run in a few minutes, not hours. Unfortunately it's also not terribly uncommon to see these e2e suites take many hours, or sometimes even days to run. I think the happy medium might be using a dependency-aware testing pipeline that only runs tests for areas affected by the current commit's changes. I've seen Bazel used in this kind of setup with some success, for example. Ideally no bugs make it to production, but that's almost never the reality. Obviously it'll depend on your software's use-case and user demographics, but you could erode user-trust just as bad by having a trivial user-facing bug sit visible on your app/site for a few days or weeks because your CI/CD pipeline takes so long to run that it prevents you from pushing out timely releases.

asdfasgasdgasdg 6y ago
Yeah, another aspect here is optimizing tests properly. Of course, this also takes engineering time, but if you get your performance culture in the right place, hopefully test run costs can be commensurate to the amount of value you're delivering. That will allow you to never have a situation where your tests take inordinate amounts of time because you cannot afford resources to run them. Everything is tradeoffs, of course. You have to decide what's right for your business. But I think it's rare that there is too much or too thorough of testing. I'm guessing most orgs err in the other direction.