5 ms·

This is an endless debate. Each situation requires a different test setup but ultimately you can't say end-to-end tests are not worth it. You can have perfectly

by msoad 5y ago

This is an endless debate. Each situation requires a different test setup but ultimately you can't say end-to-end tests are not worth it. You can have perfectly functioning units of software that are all perfectly unit tested but the units are not working together (insert a related meme GIF about working drawers colliding when opened). This can happen with strongest inter-unit communication protocols such as strong types and validation mechanisms. E2E tests are very hard to maintain but in many situations they are required.

commandlinefan 5y ago
> you can't say end-to-end tests are not worth it Or, from another perspective - you are doing end-to-end testing, the question is whether you're doing it before production or if your customers are doing it for you...
- coryrc 5y ago
  Tests are never 100%, customers will always find issues. They made the choice to allow slightly more bugs to production than before, not to go from never having bugs in production to having some.
  - alex-freire 5y ago
    we didn't see more bugs in production when we sunseted E2E. in fact, complementing contract tests with acceptance tests we saw less bugs in production and more productive test creation and maintanence. :-D
- rgoulter 5y ago
  > you are doing end-to-end testing, the question is whether you're doing it before production or if your customers are doing it for you. This reminds me of the point I've seen made elsewhere: E2E tests can be complemented by use of monitoring metrics, healthchecks, etc. for providing confidence that the system is working as intended (or for spotting cases where it's not working as intended).
rmbyrro 5y ago
They didn't say E2E aren't worth it (they actually said it worked for them early on) Their point is that it's not scalable as they think contract testing is
- lowbloodsugar 5y ago
  Contract testing isn't E2E testing and doesn't provide the same guarantees. So they are replacing a high cost, high value system with a lower cost, lower value system. So they are literally saying that they don't think the extra value of E2E testing is worth the extra cost. Personally, I think they are wrong, and the problems they have with their E2E tests are problems with their implementation. Fixing them would benefit the customer experience and the developer experience as well as reducing the costs of the E2E. They are absolutely going to have critical customer impact (they are fintech ffs) that their E2E tests would have caught. Of course, whether that actually tanks their business depends on other factors. So accepting more critical issues may the right thing for the success of the company. Robinhood customers were greatly pissed off by its behavior, and yet it doesn't seem to have hurt them too much. But I wouldn't fucking boast about it!
asdfasgasdgasdg 5y ago
I can hardly think of a situation where I'd want no end to end test. I think one misconception is that there has to be a single end to end test. Really what you want is a variety of end to end tests examining the functionality of different parts of the system. But the system under test is still the whole system, not the units. These partial end to end tests can still be quick to run, as long as you keep system startup time down. For example, I work on a system that builds text indexes on an underlying database management system. We take an input mutation with logical changes and then use that to determine what additional index updates are required. This all happens automatically when our users write. There are two ways to test this. The old way was that we instantiated the top level class that did the changes and manually constructed mutations that look like user mutations. Then we examined the mutations produced by our top level class. I recently converted this test to use the public write and read apis of the database to instead write data to a test instance and then check that the index contents was as expected. The public api is more stable than our private one and is resilient to internal refactorings. It's also more amenable to ad hoc queries of the type you generally do in tests. And it ends up not being much slower, since our test for various sad reasons still had to start the database engine even though it was mostly unused. All in all, I was able to make the test faster (3 minutes -> 1.5 minutes) and less brittle, while using less code and getting more coverage of what we actually care about. I think wins like this are commonly available when moving from unit to end to end testing, as long as you keep system startup time down.
- afiori 5y ago
  there was a nice talk putting this as an example where React's engineering teams designed their test to be future proof: always test the public API; your end to end tests should survive a major refactoring/rewrite
  - asdfasgasdgasdg 5y ago
    Hard to overstate the benefit of not having to rewrite your tests when you refactor. That's a concrete and difficult to ignore benefit of using higher level APIs. I find that the cost of writing a test is often not less than the cost of writing the code. If that is correct, it means that the system lifetime cost of authorship is much higher with unit tests. If all tests are unit tests the cost is almost 100% higher.
Veuxdo 5y ago
E2E tests aren't worth it if they produce false positives and don't prevent defects from reaching production. By definition. Too many devs treat automated testing as a goal in and of itself.
- alex-freire 5y ago
  well said!
- mbrodersen 5y ago
  So you are saying that X is not worth it if X is crap? Who wouldn’t agree with that? What if X was implemented well and did what it was supposed to do? We use end-to-end production and it works extremely well. I have had zero production problems for years now. All issues were found by the end-to-end tests before deploying.
  - Veuxdo 5y ago
    "So you are saying that X is not worth it if X is crap?" I am because it needs to be heard. Like I said, too many devs treat writing tests as a goal unto itself.
deckard1 5y ago
> you can't say end-to-end tests are not worth it You can, actually. But here's the thing: I've never seen an honest debate on E2E within an org. When your manager comes to you and says your team is going to start doing E2E, ask him/her if they are prepared for their schedule to slip by 30% or more. They will either slither back into their office, or (most likely) they will insist that developers write E2E in addition to their current workload of writing unit tests, writing the actual code, and all of the other overhead (pull requests, approvals, JIRA ticket maintenance, interviews, etc.) Developers are expected to pay the costs of E2E with no impact to the business. What managers do not understand and has been my experience for many years now is that E2E is at least 30% of the cost of development. And that's probably low. I recall certain features where E2E took probably 200% or more time to get working. Because, unlike most unit tests, writing E2E tests is nontrivial. You may have to invent entirely new techniques and apparatus just for a single test. If the costs of writing and maintaining E2E tests outweigh the benefits, then obviously it's not worth it. Not every bug is critical. In fact, go back 15 years and no one had any tests whatsoever. The world didn't end.
- deleted 5y ago
  [deleted]
- 3pt14159 5y ago
  In my book, E2E tests should be on a couple of basic, mission critical things and integration tests should pick up the rest. It's far, far better to have 10 E2E tests and 1000 integration tests than 0 E2E tests and 1500 integration tests because it picks up failures in your infrastructure or weird stuff like middleware that are probably system wide(ish).
  - arp242 5y ago
    Pretty much this. I found that having loads of E2E tests often doesn't add all that much; usually they're all doing the exact same thing test after test after test, and since these parts tend to be fairly isolated there isn't all that much that can go wrong in just that specific test. Either it works for everything, or it fails for everything. The way I've always viewed E2E tests is as "testing everything at the top layers" such as middleware and whatnot, which you can usually do with just a few (or sometimes even one) test. Other less high-level integration tests can test all the rest, and they tend to run much faster as they avoid a lot of overhead, and are a lot easier to write and reason about, especially if tests fail. I once rewrote a E2E test suite to use integration tests which gave a massive speed-up, and because the tests were a lot easier work with people actually started writing them. I added a few E2E tests (IIRC logging in, viewing the dashboard, logging out) and that was enough really.

by msoad 5y ago

commandlinefan 5y ago

> you can't say end-to-end tests are not worth it Or, from another perspective - you are doing end-to-end testing, the question is whether you're doing it before production or if your customers are doing it for you...

coryrc 5y ago
Tests are never 100%, customers will always find issues. They made the choice to allow slightly more bugs to production than before, not to go from never having bugs in production to having some.
- alex-freire 5y ago
  we didn't see more bugs in production when we sunseted E2E. in fact, complementing contract tests with acceptance tests we saw less bugs in production and more productive test creation and maintanence. :-D
rgoulter 5y ago
> you are doing end-to-end testing, the question is whether you're doing it before production or if your customers are doing it for you. This reminds me of the point I've seen made elsewhere: E2E tests can be complemented by use of monitoring metrics, healthchecks, etc. for providing confidence that the system is working as intended (or for spotting cases where it's not working as intended).

rmbyrro 5y ago

They didn't say E2E aren't worth it (they actually said it worked for them early on) Their point is that it's not scalable as they think contract testing is

lowbloodsugar 5y ago
Contract testing isn't E2E testing and doesn't provide the same guarantees. So they are replacing a high cost, high value system with a lower cost, lower value system. So they are literally saying that they don't think the extra value of E2E testing is worth the extra cost. Personally, I think they are wrong, and the problems they have with their E2E tests are problems with their implementation. Fixing them would benefit the customer experience and the developer experience as well as reducing the costs of the E2E. They are absolutely going to have critical customer impact (they are fintech ffs) that their E2E tests would have caught. Of course, whether that actually tanks their business depends on other factors. So accepting more critical issues may the right thing for the success of the company. Robinhood customers were greatly pissed off by its behavior, and yet it doesn't seem to have hurt them too much. But I wouldn't fucking boast about it!

asdfasgasdgasdg 5y ago

I can hardly think of a situation where I'd want no end to end test. I think one misconception is that there has to be a single end to end test. Really what you want is a variety of end to end tests examining the functionality of different parts of the system. But the system under test is still the whole system, not the units. These partial end to end tests can still be quick to run, as long as you keep system startup time down. For example, I work on a system that builds text indexes on an underlying database management system. We take an input mutation with logical changes and then use that to determine what additional index updates are required. This all happens automatically when our users write. There are two ways to test this. The old way was that we instantiated the top level class that did the changes and manually constructed mutations that look like user mutations. Then we examined the mutations produced by our top level class. I recently converted this test to use the public write and read apis of the database to instead write data to a test instance and then check that the index contents was as expected. The public api is more stable than our private one and is resilient to internal refactorings. It's also more amenable to ad hoc queries of the type you generally do in tests. And it ends up not being much slower, since our test for various sad reasons still had to start the database engine even though it was mostly unused. All in all, I was able to make the test faster (3 minutes -> 1.5 minutes) and less brittle, while using less code and getting more coverage of what we actually care about. I think wins like this are commonly available when moving from unit to end to end testing, as long as you keep system startup time down.

afiori 5y ago
there was a nice talk putting this as an example where React's engineering teams designed their test to be future proof: always test the public API; your end to end tests should survive a major refactoring/rewrite
- asdfasgasdgasdg 5y ago
  Hard to overstate the benefit of not having to rewrite your tests when you refactor. That's a concrete and difficult to ignore benefit of using higher level APIs. I find that the cost of writing a test is often not less than the cost of writing the code. If that is correct, it means that the system lifetime cost of authorship is much higher with unit tests. If all tests are unit tests the cost is almost 100% higher.

Veuxdo 5y ago

E2E tests aren't worth it if they produce false positives and don't prevent defects from reaching production. By definition. Too many devs treat automated testing as a goal in and of itself.

alex-freire 5y ago
well said!
mbrodersen 5y ago
So you are saying that X is not worth it if X is crap? Who wouldn’t agree with that? What if X was implemented well and did what it was supposed to do? We use end-to-end production and it works extremely well. I have had zero production problems for years now. All issues were found by the end-to-end tests before deploying.
- Veuxdo 5y ago
  "So you are saying that X is not worth it if X is crap?" I am because it needs to be heard. Like I said, too many devs treat writing tests as a goal unto itself.

deckard1 5y ago

> you can't say end-to-end tests are not worth it You can, actually. But here's the thing: I've never seen an honest debate on E2E within an org. When your manager comes to you and says your team is going to start doing E2E, ask him/her if they are prepared for their schedule to slip by 30% or more. They will either slither back into their office, or (most likely) they will insist that developers write E2E in addition to their current workload of writing unit tests, writing the actual code, and all of the other overhead (pull requests, approvals, JIRA ticket maintenance, interviews, etc.) Developers are expected to pay the costs of E2E with no impact to the business. What managers do not understand and has been my experience for many years now is that E2E is at least 30% of the cost of development. And that's probably low. I recall certain features where E2E took probably 200% or more time to get working. Because, unlike most unit tests, writing E2E tests is nontrivial. You may have to invent entirely new techniques and apparatus just for a single test. If the costs of writing and maintaining E2E tests outweigh the benefits, then obviously it's not worth it. Not every bug is critical. In fact, go back 15 years and no one had any tests whatsoever. The world didn't end.

deleted 5y ago
[deleted]
3pt14159 5y ago
In my book, E2E tests should be on a couple of basic, mission critical things and integration tests should pick up the rest. It's far, far better to have 10 E2E tests and 1000 integration tests than 0 E2E tests and 1500 integration tests because it picks up failures in your infrastructure or weird stuff like middleware that are probably system wide(ish).
- arp242 5y ago
  Pretty much this. I found that having loads of E2E tests often doesn't add all that much; usually they're all doing the exact same thing test after test after test, and since these parts tend to be fairly isolated there isn't all that much that can go wrong in just that specific test. Either it works for everything, or it fails for everything. The way I've always viewed E2E tests is as "testing everything at the top layers" such as middleware and whatnot, which you can usually do with just a few (or sometimes even one) test. Other less high-level integration tests can test all the rest, and they tend to run much faster as they avoid a lot of overhead, and are a lot easier to write and reason about, especially if tests fail. I once rewrote a E2E test suite to use integration tests which gave a massive speed-up, and because the tests were a lot easier work with people actually started writing them. I added a few E2E tests (IIRC logging in, viewing the dashboard, logging out) and that was enough really.