5 ms·

End-to-end tests verify high-level expectations based on the specification of the system. These high-level expectations generally stay stable over time (at leas

End-to-end tests verify high-level expectations based on the specification of the system. These high-level expectations generally stay stable over time (at least compared to the implementation details verified by lower-level tests), and therefore end-to-end tests should have the best maintainability score.

wpietri 2y ago
The expectations can be pretty stable, but because they cover so much of the system, they tend to be more fragile. End to end tests are also often flakier because they're dealing with the system at levels meant for human interaction, like by driving actual browsers. Because they encompass so much, they're also the slowest. When you have a problem with an end-to-end test, they can take way more time to debug because of that. So I'd agree with them; E2E tests are the hardest to maintain.
bastawhiz 2y ago
> end-to-end tests should have the best maintainability score. End to end tests encompass far more total details than implementation or unit tests. If you're testing a website, moving a button breaks a test. Making the button have a confirmation breaks the test. The database being slower breaks the tests. The number of items in a paginated list changing breaks the tests. You're testing not just the behavior and output of interfaces, you're testing how they're composed. The marketing team putting a banner in the wrong place breaks the tests. The product team putting a new user tour popover on the wrong button breaks the tests. The support team enabling the knowledge base integration in your on-page support widget breaks the tests. Moreover, the cost of fixing the tests is also often higher, because end to end tests are necessarily slower and more complicated. Tests often become flaky because of a larger number of dependencies on external systems. It's often not clear why a test is failing, because the test can't possibly explain why its assertion is no longer true ("The button isn't there" vs "The button now has a slightly different label").
- nemetroid 2y ago
  Not all systems are web sites. Those issues are not inherent to the concept of end-to-end testing.
  - bastawhiz 2y ago
    It still illustrates how E2E testing encompasses far more details than other forms of testing, which necessarily makes them more brittle
    - nemetroid 2y ago
      Sure, they often are less reliable, which is the R of the smurf and not the M.
      - je42 2y ago
        E2E are less maintainable because by definition. They are the type of tests that has the most dependencies on sub-systems and other systems - this means if a test fails you ll need to work more to figure out what went wrong, than in an integration test, which depends on less sub-systems.

wpietri 2y ago

The expectations can be pretty stable, but because they cover so much of the system, they tend to be more fragile. End to end tests are also often flakier because they're dealing with the system at levels meant for human interaction, like by driving actual browsers. Because they encompass so much, they're also the slowest. When you have a problem with an end-to-end test, they can take way more time to debug because of that. So I'd agree with them; E2E tests are the hardest to maintain.

bastawhiz 2y ago

> end-to-end tests should have the best maintainability score. End to end tests encompass far more total details than implementation or unit tests. If you're testing a website, moving a button breaks a test. Making the button have a confirmation breaks the test. The database being slower breaks the tests. The number of items in a paginated list changing breaks the tests. You're testing not just the behavior and output of interfaces, you're testing how they're composed. The marketing team putting a banner in the wrong place breaks the tests. The product team putting a new user tour popover on the wrong button breaks the tests. The support team enabling the knowledge base integration in your on-page support widget breaks the tests. Moreover, the cost of fixing the tests is also often higher, because end to end tests are necessarily slower and more complicated. Tests often become flaky because of a larger number of dependencies on external systems. It's often not clear why a test is failing, because the test can't possibly explain why its assertion is no longer true ("The button isn't there" vs "The button now has a slightly different label").

nemetroid 2y ago
Not all systems are web sites. Those issues are not inherent to the concept of end-to-end testing.
- bastawhiz 2y ago
  It still illustrates how E2E testing encompasses far more details than other forms of testing, which necessarily makes them more brittle
  - nemetroid 2y ago
    Sure, they often are less reliable, which is the R of the smurf and not the M.
    - je42 2y ago
      E2E are less maintainable because by definition. They are the type of tests that has the most dependencies on sub-systems and other systems - this means if a test fails you ll need to work more to figure out what went wrong, than in an integration test, which depends on less sub-systems.