6 ms·

E2E is not a replacement for unit tests. E2E tests are more difficult to write, as they can require a lot of setup to get all the exact scenario needed for each

by ysavir 3y ago

E2E is not a replacement for unit tests. E2E tests are more difficult to write, as they can require a lot of setup to get all the exact scenario needed for each piece of functionality we need to test, they tend to be flaky, they can be very slow, etc. The purpose of unit testing is to isolate the various scenarios by making sure that each unit/component/whatever correctly responds to its various inputs. In this way, we only need to set up the relevant scenario for that unit. If we tested each of these piece of functionality in an E2E test, then each test would also need to set up the user session, settings, etc. It would be a nightmare to author and maintain, compared to the simple, straight forward unit tests. In contrast, E2E tests are good at making sure that the whole system works well together, and that each unit is getting inputs and providing outputs, and that the page doesn't break in the process. Both are needed to maintain confidence, and each solves a different problem.

cmckn 3y ago
Agreed. An e2e test suite can continue passing with significant bug rot throughout the system, as they’re often given generous timeouts, retries for flakes, etc. So once you realize something is really, truly broken; it’s going to be a nightmare to track down and properly fix if the e2e suite is your only data point.
hinkley 3y ago
I have seen projects where people push back on making major functional changes and the only rational explanation is that they don’t want to have to rewrite a bunch of brittle E2E tests. That really took the value of E2E down several notches in my estimation.
- TuringNYC 3y ago
  >> I have seen projects where people push back on making major functional changes and the only rational explanation is that they don’t want to have to rewrite a bunch of brittle E2E tests. Same for any testing. it should be part of the cost and Cost Benefit Analysis.
  - hinkley 3y ago
    There are ways to write unit tests that make it a lot cheaper to change requirements. But I can still count the number of people I've worked with who write them that way. Right now I'm still trying to convince people that fakes and mocks are not the same thing.
zer00eyz 3y ago
> they tend to be flaky If your E2E test is flaky, then how is your production environment reliable? > they can require a lot of setup Setup vs mock. > they can be very slow, etc. We have a good handle on scaling, slow is now a product of the testing time being smeared across your org like peanut butter. > The purpose of unit testing... Does E2E replace unit testing, no. Things that are already units or esoteric (business "logic", you know it's in your code base). But if your writing mocks your better off un fucking your environment that E2E testing is viable, and make it a first class citizen. Everything is a crud app of some sort: web, api, mobile, smoke signal... Dropping data in these front ends, can be validated out of the back end, and in the app itself. To put a fine point on it: Unit tests make me feel good about checking in. End to end tests make me confident to deploy!
- TuringNYC 3y ago
  >> If your E2E test is flaky, then how is your production environment reliable? IME, E2E tests are flaky despite the underlying system not being flaky because E2E has to do a series of things, often including waiting/polling, and in real life people will wait longer or compensate but in tests you want to do it all perfectly and fast.
  - colinmorelli 3y ago
    I think in this scenario, different people will have a different perspective on whether asking people to "wait longer" constitutes flakiness or not. From an end-user's perspective, there's very little difference between "the task didn't work" and "the task worked, but took longer than your patience to wait." Both appear to be broken.
    - ysavir 3y ago
      The end user isn't exposed to it. The flakiness comes from needing an automated browser interaction acting against a page with changing input. If your page is static, sure, the E2E test is probably straightforward and quick. But the moment any dynamic functionality is introduced to the page, the test's interactions need to account for the page's current context (or wait for an intended context). And that has to be coordinated amongst the actual test code, the testing browser, the driver, etc. If anyone one of them hiccups, you may end with a false negative, and those happen more often than we'd like.
corethree 3y ago
No but if choosing one or the other E2E is the far better choice. Ideally you have both.
- Scubabear68 3y ago
  This is my experience as well. Unit test suites tend to be huge, which costs effort to maintain, and they generally tell you little about the actual quality of the software. A good end to end test suite will give you a much truer indication of the state of your systems. Of course I don’t say do zero unit tests, they have value, but I think too many orgs are too zealous about unit tests and too soft on higher level testing.

by ysavir 3y ago

cmckn 3y ago

Agreed. An e2e test suite can continue passing with significant bug rot throughout the system, as they’re often given generous timeouts, retries for flakes, etc. So once you realize something is really, truly broken; it’s going to be a nightmare to track down and properly fix if the e2e suite is your only data point.

hinkley 3y ago

I have seen projects where people push back on making major functional changes and the only rational explanation is that they don’t want to have to rewrite a bunch of brittle E2E tests. That really took the value of E2E down several notches in my estimation.

TuringNYC 3y ago
>> I have seen projects where people push back on making major functional changes and the only rational explanation is that they don’t want to have to rewrite a bunch of brittle E2E tests. Same for any testing. it should be part of the cost and Cost Benefit Analysis.
- hinkley 3y ago
  There are ways to write unit tests that make it a lot cheaper to change requirements. But I can still count the number of people I've worked with who write them that way. Right now I'm still trying to convince people that fakes and mocks are not the same thing.

zer00eyz 3y ago

> they tend to be flaky If your E2E test is flaky, then how is your production environment reliable? > they can require a lot of setup Setup vs mock. > they can be very slow, etc. We have a good handle on scaling, slow is now a product of the testing time being smeared across your org like peanut butter. > The purpose of unit testing... Does E2E replace unit testing, no. Things that are already units or esoteric (business "logic", you know it's in your code base). But if your writing mocks your better off un fucking your environment that E2E testing is viable, and make it a first class citizen. Everything is a crud app of some sort: web, api, mobile, smoke signal... Dropping data in these front ends, can be validated out of the back end, and in the app itself. To put a fine point on it: Unit tests make me feel good about checking in. End to end tests make me confident to deploy!

TuringNYC 3y ago
>> If your E2E test is flaky, then how is your production environment reliable? IME, E2E tests are flaky despite the underlying system not being flaky because E2E has to do a series of things, often including waiting/polling, and in real life people will wait longer or compensate but in tests you want to do it all perfectly and fast.
- colinmorelli 3y ago
  I think in this scenario, different people will have a different perspective on whether asking people to "wait longer" constitutes flakiness or not. From an end-user's perspective, there's very little difference between "the task didn't work" and "the task worked, but took longer than your patience to wait." Both appear to be broken.
  - ysavir 3y ago
    The end user isn't exposed to it. The flakiness comes from needing an automated browser interaction acting against a page with changing input. If your page is static, sure, the E2E test is probably straightforward and quick. But the moment any dynamic functionality is introduced to the page, the test's interactions need to account for the page's current context (or wait for an intended context). And that has to be coordinated amongst the actual test code, the testing browser, the driver, etc. If anyone one of them hiccups, you may end with a false negative, and those happen more often than we'd like.

corethree 3y ago

No but if choosing one or the other E2E is the far better choice. Ideally you have both.

Scubabear68 3y ago
This is my experience as well. Unit test suites tend to be huge, which costs effort to maintain, and they generally tell you little about the actual quality of the software. A good end to end test suite will give you a much truer indication of the state of your systems. Of course I don’t say do zero unit tests, they have value, but I think too many orgs are too zealous about unit tests and too soft on higher level testing.