3 ms·

> you can't say end-to-end tests are not worth it You can, actually. But here's the thing: I've never seen an honest debate on E2E within an org. When your ma

> you can't say end-to-end tests are not worth it You can, actually. But here's the thing: I've never seen an honest debate on E2E within an org. When your manager comes to you and says your team is going to start doing E2E, ask him/her if they are prepared for their schedule to slip by 30% or more. They will either slither back into their office, or (most likely) they will insist that developers write E2E in addition to their current workload of writing unit tests, writing the actual code, and all of the other overhead (pull requests, approvals, JIRA ticket maintenance, interviews, etc.) Developers are expected to pay the costs of E2E with no impact to the business. What managers do not understand and has been my experience for many years now is that E2E is at least 30% of the cost of development. And that's probably low. I recall certain features where E2E took probably 200% or more time to get working. Because, unlike most unit tests, writing E2E tests is nontrivial. You may have to invent entirely new techniques and apparatus just for a single test. If the costs of writing and maintaining E2E tests outweigh the benefits, then obviously it's not worth it. Not every bug is critical. In fact, go back 15 years and no one had any tests whatsoever. The world didn't end.

deleted 5y ago
[deleted]
3pt14159 5y ago
In my book, E2E tests should be on a couple of basic, mission critical things and integration tests should pick up the rest. It's far, far better to have 10 E2E tests and 1000 integration tests than 0 E2E tests and 1500 integration tests because it picks up failures in your infrastructure or weird stuff like middleware that are probably system wide(ish).
- arp242 5y ago
  Pretty much this. I found that having loads of E2E tests often doesn't add all that much; usually they're all doing the exact same thing test after test after test, and since these parts tend to be fairly isolated there isn't all that much that can go wrong in just that specific test. Either it works for everything, or it fails for everything. The way I've always viewed E2E tests is as "testing everything at the top layers" such as middleware and whatnot, which you can usually do with just a few (or sometimes even one) test. Other less high-level integration tests can test all the rest, and they tend to run much faster as they avoid a lot of overhead, and are a lot easier to write and reason about, especially if tests fail. I once rewrote a E2E test suite to use integration tests which gave a massive speed-up, and because the tests were a lot easier work with people actually started writing them. I added a few E2E tests (IIRC logging in, viewing the dashboard, logging out) and that was enough really.
  - 3pt14159 5y ago
    Yes exactly. This mirrors my experience as well. Of course it depends on the individual setup, but making testing easier and faster has huge benefits, but a tiny amount of E2E coverage goes a long way.
- tetha 5y ago
  I feel like this is also where dogfooding - or drinking your own champagne - comes in, if possible. We can use our software internally and sure, there are hardware costs and manpower overheads to run an additional instance of our software, but those aren't too high. Hardware necessary to run E2E tests of all the systems at proper scale including maintenance manpower probably eclipses those efforts. And then you'd have to add dev-hours on top of the E2E costs to build and maintain mountains of E2E tests. And this has exposed really nasty bugs in common paths already, just by employees using the system.
  - arp242 5y ago
    This is absolutely a good attitude, but it only works for a fairly limited class of software. I worked for things like realtor agent software, child care agency software, etc. and you can't really "dogfood" these sort of systems unless you want to become a landlord or start a childcare service.
brabel 5y ago
> In fact, go back 15 years and no one had any tests whatsoever. The world didn't end. You really need to stop repeating this, it's absurd and there was a great post here the other day explaining how in most companies, they had a large QA team that would need to approve any code, it's just that developers were not expected to write the tests themselves. > If the costs of writing and maintaining E2E tests outweigh the benefits, then obviously it's not worth it. Obviously, but the question is, what's the alternative? In the blog post, the alternative was to have contract-based acceptance tests... but that may not always be appropriate for every business. We have a huge E2E test suite where I work and I was one of the biggest contributors to creating it... as everywhere else, it's heavy, slow and hard to maintain, but replacing what we have with contract testing wouold be unfeasible because we're not a micro-service architecture, we are one big application as we're a product company.... I would love to find a better way of testing our product, but contract-based testing is definitely not the answer for us.
codefreakxff 5y ago
“In fact, go back 15 years and no one had any tests whatsoever. The world didn't end.” That’s just silly. Of course there were tests 15 years ago. Unit testing has been around since the 1950s Very little would cause the world to end. But lives have been lost and billions of dollars along the way wasted due to improper testing
- Too 5y ago
  Not in the same way as a modern CI/CD-pipeline. 15 years ago you had release-approval meetings where a dedicated QA team would present all their manually observed findings in a excel-sheet to management. E2E is a replacement for, or evolution of, the QA department, not a replacement for unit tests.

deleted 5y ago

[deleted]

3pt14159 5y ago

In my book, E2E tests should be on a couple of basic, mission critical things and integration tests should pick up the rest. It's far, far better to have 10 E2E tests and 1000 integration tests than 0 E2E tests and 1500 integration tests because it picks up failures in your infrastructure or weird stuff like middleware that are probably system wide(ish).

arp242 5y ago
Pretty much this. I found that having loads of E2E tests often doesn't add all that much; usually they're all doing the exact same thing test after test after test, and since these parts tend to be fairly isolated there isn't all that much that can go wrong in just that specific test. Either it works for everything, or it fails for everything. The way I've always viewed E2E tests is as "testing everything at the top layers" such as middleware and whatnot, which you can usually do with just a few (or sometimes even one) test. Other less high-level integration tests can test all the rest, and they tend to run much faster as they avoid a lot of overhead, and are a lot easier to write and reason about, especially if tests fail. I once rewrote a E2E test suite to use integration tests which gave a massive speed-up, and because the tests were a lot easier work with people actually started writing them. I added a few E2E tests (IIRC logging in, viewing the dashboard, logging out) and that was enough really.
- 3pt14159 5y ago
  Yes exactly. This mirrors my experience as well. Of course it depends on the individual setup, but making testing easier and faster has huge benefits, but a tiny amount of E2E coverage goes a long way.
tetha 5y ago
I feel like this is also where dogfooding - or drinking your own champagne - comes in, if possible. We can use our software internally and sure, there are hardware costs and manpower overheads to run an additional instance of our software, but those aren't too high. Hardware necessary to run E2E tests of all the systems at proper scale including maintenance manpower probably eclipses those efforts. And then you'd have to add dev-hours on top of the E2E costs to build and maintain mountains of E2E tests. And this has exposed really nasty bugs in common paths already, just by employees using the system.
- arp242 5y ago
  This is absolutely a good attitude, but it only works for a fairly limited class of software. I worked for things like realtor agent software, child care agency software, etc. and you can't really "dogfood" these sort of systems unless you want to become a landlord or start a childcare service.

brabel 5y ago

> In fact, go back 15 years and no one had any tests whatsoever. The world didn't end. You really need to stop repeating this, it's absurd and there was a great post here the other day explaining how in most companies, they had a large QA team that would need to approve any code, it's just that developers were not expected to write the tests themselves. > If the costs of writing and maintaining E2E tests outweigh the benefits, then obviously it's not worth it. Obviously, but the question is, what's the alternative? In the blog post, the alternative was to have contract-based acceptance tests... but that may not always be appropriate for every business. We have a huge E2E test suite where I work and I was one of the biggest contributors to creating it... as everywhere else, it's heavy, slow and hard to maintain, but replacing what we have with contract testing wouold be unfeasible because we're not a micro-service architecture, we are one big application as we're a product company.... I would love to find a better way of testing our product, but contract-based testing is definitely not the answer for us.

codefreakxff 5y ago

“In fact, go back 15 years and no one had any tests whatsoever. The world didn't end.” That’s just silly. Of course there were tests 15 years ago. Unit testing has been around since the 1950s Very little would cause the world to end. But lives have been lost and billions of dollars along the way wasted due to improper testing

Too 5y ago
Not in the same way as a modern CI/CD-pipeline. 15 years ago you had release-approval meetings where a dedicated QA team would present all their manually observed findings in a excel-sheet to management. E2E is a replacement for, or evolution of, the QA department, not a replacement for unit tests.