5 ms·

FWIW I've been in both situations. One company had sketchy E2E coverage that resulted in a modicum of production bugs. I moved to a competitor of roughly same s

FWIW I've been in both situations. One company had sketchy E2E coverage that resulted in a modicum of production bugs. I moved to a competitor of roughly same size had a huge E2E suite and AFAICT results in roughly the same modicum of production bugs. But feature development at the latter moves much more slowly because of all the wait queues, flaky tests, timeouts, test maintenance overhead, etc. IMO these seem to be more of a CYA thing for managers. When a bug does get to production, you need to have something to point to. (And in my previous company of course they're scrambling now to make a big E2E testing platform). But I'm not convinced they're actually worth the effort. Edit: actually maybe I think they're worth the effort, if for no other reason than when bugs do go to production, executives tend to start micromanaging if you don't have something to point at. That can be worse than dealing with flaky tests. But what I'm not convinced of is whether they actually reduce the number of bugs that go to production.

lowbloodsugar 5y ago
You have been in two of three situations. 1. No e2e tests. 2. Heavy, flaky, slow e2e tests. 3. e2e as a driver of first class system integration Compare with 1. no unit tests 2. tons of unit tests, that regularly fail, nobody cares and check in more bad tests, "unit" tests that thread sleep and take minutes 3. CI/CD with 0 tolerance for failures or >1s tests The difference between "We have tons of tests" and "We drive development with tests" is night and day. Sure, if you slap on e2e with the mandate "They must exist", then you're going to have a shitty experience, e2e or unit.
- catern 5y ago
  >3. e2e as a driver of first class system integration Could you say more about this approach? That's how I've tried to approach end-to-end tests, but I haven't found much written about this which is specifically about e2e tests.
  - lowbloodsugar 5y ago
    I haven't written a book but just take TDD philosophy but apply it to e2e. For example, if you have an e2e test that often fails because something times out, or you've had to set a high timeout, then dig down why and fix it. Turns out you have a JVM that often has a 2s GC but no alarms, and yes, it impacts customers too. Fix that problem, don't get rid of the test (or all the tests). You got a slow third party thing, put an abstraction in front of it so that username: MyE2EUser's traffic goes to a shim. Then either negotiate with third party, or make interaction with third party system asychronous. Or does resubmitting the page order the product twice?! lol. I wish those were the old days. All that being said, I haven't done this for a couple of years now, and I have the joys of vastly simpler systems at vastly huger scale, so I may be seeing the past through rose tinted glasses.
spc476 5y ago
If you have an E2E testing platform, and a bug is found in production, can you reproduce the issue with the E2E testing platform? How easy is it?

lowbloodsugar 5y ago

You have been in two of three situations. 1. No e2e tests. 2. Heavy, flaky, slow e2e tests. 3. e2e as a driver of first class system integration Compare with 1. no unit tests 2. tons of unit tests, that regularly fail, nobody cares and check in more bad tests, "unit" tests that thread sleep and take minutes 3. CI/CD with 0 tolerance for failures or >1s tests The difference between "We have tons of tests" and "We drive development with tests" is night and day. Sure, if you slap on e2e with the mandate "They must exist", then you're going to have a shitty experience, e2e or unit.

catern 5y ago
>3. e2e as a driver of first class system integration Could you say more about this approach? That's how I've tried to approach end-to-end tests, but I haven't found much written about this which is specifically about e2e tests.
- lowbloodsugar 5y ago
  I haven't written a book but just take TDD philosophy but apply it to e2e. For example, if you have an e2e test that often fails because something times out, or you've had to set a high timeout, then dig down why and fix it. Turns out you have a JVM that often has a 2s GC but no alarms, and yes, it impacts customers too. Fix that problem, don't get rid of the test (or all the tests). You got a slow third party thing, put an abstraction in front of it so that username: MyE2EUser's traffic goes to a shim. Then either negotiate with third party, or make interaction with third party system asychronous. Or does resubmitting the page order the product twice?! lol. I wish those were the old days. All that being said, I haven't done this for a couple of years now, and I have the joys of vastly simpler systems at vastly huger scale, so I may be seeing the past through rose tinted glasses.

spc476 5y ago

If you have an E2E testing platform, and a bug is found in production, can you reproduce the issue with the E2E testing platform? How easy is it?