3 ms·

In my book, E2E tests should be on a couple of basic, mission critical things and integration tests should pick up the rest. It's far, far better to have 10 E2E

In my book, E2E tests should be on a couple of basic, mission critical things and integration tests should pick up the rest. It's far, far better to have 10 E2E tests and 1000 integration tests than 0 E2E tests and 1500 integration tests because it picks up failures in your infrastructure or weird stuff like middleware that are probably system wide(ish).

arp242 5y ago
Pretty much this. I found that having loads of E2E tests often doesn't add all that much; usually they're all doing the exact same thing test after test after test, and since these parts tend to be fairly isolated there isn't all that much that can go wrong in just that specific test. Either it works for everything, or it fails for everything. The way I've always viewed E2E tests is as "testing everything at the top layers" such as middleware and whatnot, which you can usually do with just a few (or sometimes even one) test. Other less high-level integration tests can test all the rest, and they tend to run much faster as they avoid a lot of overhead, and are a lot easier to write and reason about, especially if tests fail. I once rewrote a E2E test suite to use integration tests which gave a massive speed-up, and because the tests were a lot easier work with people actually started writing them. I added a few E2E tests (IIRC logging in, viewing the dashboard, logging out) and that was enough really.
- 3pt14159 5y ago
  Yes exactly. This mirrors my experience as well. Of course it depends on the individual setup, but making testing easier and faster has huge benefits, but a tiny amount of E2E coverage goes a long way.
tetha 5y ago
I feel like this is also where dogfooding - or drinking your own champagne - comes in, if possible. We can use our software internally and sure, there are hardware costs and manpower overheads to run an additional instance of our software, but those aren't too high. Hardware necessary to run E2E tests of all the systems at proper scale including maintenance manpower probably eclipses those efforts. And then you'd have to add dev-hours on top of the E2E costs to build and maintain mountains of E2E tests. And this has exposed really nasty bugs in common paths already, just by employees using the system.
- arp242 5y ago
  This is absolutely a good attitude, but it only works for a fairly limited class of software. I worked for things like realtor agent software, child care agency software, etc. and you can't really "dogfood" these sort of systems unless you want to become a landlord or start a childcare service.

arp242 5y ago

Pretty much this. I found that having loads of E2E tests often doesn't add all that much; usually they're all doing the exact same thing test after test after test, and since these parts tend to be fairly isolated there isn't all that much that can go wrong in just that specific test. Either it works for everything, or it fails for everything. The way I've always viewed E2E tests is as "testing everything at the top layers" such as middleware and whatnot, which you can usually do with just a few (or sometimes even one) test. Other less high-level integration tests can test all the rest, and they tend to run much faster as they avoid a lot of overhead, and are a lot easier to write and reason about, especially if tests fail. I once rewrote a E2E test suite to use integration tests which gave a massive speed-up, and because the tests were a lot easier work with people actually started writing them. I added a few E2E tests (IIRC logging in, viewing the dashboard, logging out) and that was enough really.

3pt14159 5y ago
Yes exactly. This mirrors my experience as well. Of course it depends on the individual setup, but making testing easier and faster has huge benefits, but a tiny amount of E2E coverage goes a long way.

tetha 5y ago

I feel like this is also where dogfooding - or drinking your own champagne - comes in, if possible. We can use our software internally and sure, there are hardware costs and manpower overheads to run an additional instance of our software, but those aren't too high. Hardware necessary to run E2E tests of all the systems at proper scale including maintenance manpower probably eclipses those efforts. And then you'd have to add dev-hours on top of the E2E costs to build and maintain mountains of E2E tests. And this has exposed really nasty bugs in common paths already, just by employees using the system.

arp242 5y ago
This is absolutely a good attitude, but it only works for a fairly limited class of software. I worked for things like realtor agent software, child care agency software, etc. and you can't really "dogfood" these sort of systems unless you want to become a landlord or start a childcare service.