5 ms·

End to end tests are less precise to triage when they break, and they break more often than unit tests. As the author said, they show all bugs, so they break a

by r0s 6y ago

End to end tests are less precise to triage when they break, and they break more often than unit tests. As the author said, they show all bugs, so they break a lot. When the test is broken, it's not providing coverage until it's fixed. Which is why I advocate for less assertive testing in that case: https://assertless.org/ https://assertless.org/ The priority of fixing end to end tests becomes critical, and so test maintenance is more important. With a large test suite of functional tests, that effort is not sustainable in my experience. I've been there. An image comparison suite of tests has the same issue, except it may defer all assertions, which is good, but it offloads a huge amount of analysis to the engineer consuming all the output. You really do need a human to check those tiny style changes it finds, a human with extensive knowledge of the product and it's ever-changing visual quirks. I also maintain a large visual testing suite like that. Cypress is not different from Selenium in that aspect. (There's not much real difference at all that I can see.) I think end to end tests have more of a value than the traditional testing pyramid would indicate, but there's pitfalls and benefits to each type. A suite of concise unit tests will find obvious bugs with precise granularity incredibly quick on the code they cover, which can never be complete coverage, but it doesn't need to be. This can be used for things like commit hooks that end to end tests just can't.

marcosdumay 6y ago
Hum... Depending on exactly what you test, it's not only the coverage that increases with a larger scope, but the tests also break less. If you engineers things for testing¹, e2e tests need much less maintenance than unit tests. Yes, their failure is also more severe, but overall, they are a large gain on that area. But then, you may discover that your current project just can not be adapted to enable e2e tests, like some projects can not be adapted to enable unity or integration tests. The single size preaching is completely flawed. 1 - Not on the ways the unity testing fans uses that phrase obviously, but if you keep your (human and machine) interfaces stable and generic enough to survive some software evolution.
username90 6y ago
> and they break more often than unit tests. That isn't my experience, unit tests breaks all the time and has to be rewritten from simple refactoring while larger tests pass as long as you keep behavior the same. This means that the bigger tests are way better for refactoring and safely making changes to your code. If most of your tests are unit tests and you don't have bigger tests covering the same things then refactoring safely is hell.
- jschwartzi 6y ago
  If your unit tests are breaking all the time, try testing less at the unit level and more at the integration level. There are a lot of dogmatists out there who preach that unit tests should have "X % coverage" no matter what, but that's a ridiculous claim. Different projects will require different mixes of testing strategy, and part of the engineering process is arriving at your verification and validation methods, codifying them, ensuring that they provide valid output, and making sure you're using your time efficiently. If most of your time is spent fixing tests, re-evaluate whether those tests are worth doing that way. Your time might be better spent testing at a higher level or even manually.
  - username90 6y ago
    Right, I never write traditional unit tests, I just test the API boundaries as you said. Sometimes it makes sense to draw a new API boundary in the code for some complex logic that is unlikely to change, but most of my tests looks pretty close to others integration tests.
mannykannot 6y ago
Unit tests don't just fail to find bugs because they don't give complete coverage (all testing has that problem, and the author is mistaken to say that end-to-end testing shows all bugs), they fail to find bugs resulting from faulty assumptions about how the units will interact. Boeing's faulty Starliner test demonstrated how that goes [1]. The true value in unit testing is in revealing the bugs it will find early in the process. One benefit of this is there will likely be fewer to be found in the end-to-end testing (or on deployment.) There is no value in debating which is better. The important issue is how to allocate your testing budget effectively, because there is never even close to enough time to exhaustively test. [1] https://www.engadget.com/2020-02-29-boeing-starliner-failed-first-flight-report.html https://www.engadget.com/2020-02-29-boeing-starliner-failed-...
- ozim 6y ago
  Pyramid approach, inverted pyramid, crab shaped does not make sense because it tries to be catch all guideline. Same with any 'maturity models' for software projects. People are forgetting there is something like 'risk based' approach to testing. This is the only way to allocate budget effectively, where stuff that can drop half of your database should be tested well and stuff that will make your UI look funny probably less.
  - Jtsummers 6y ago
    I hate to say this, but I do think CMMI's model is a good idea at its core. The model itself is actually pretty solid. That is, everything it describes is something a good organization should do, and the requirement to document the processes + train people is a sound one. But the appraisals and how most businesses treat it are, well, moronic. The appraisals are easily gamed so that if you do everything on paper you're pretty much going to get your target level (3 is good enough for 99% of contracts if matters at all). And businesses try to treat the model as a process, when it's not. The former means that the appraisal results are useless. You cannot judge anyone by being level 3 or 5. That often just means they played the game better, not that they're actually better. The latter, though, is the worst part. CMMI is broken into a number of process areas (I'm happy to report I no longer recall the number of areas). It describes them in psuedo-process or procedural terms. That is, you could take what they describe and make a process, but it would be a very linear and simple process. What's intended is for you to say how your process maps to their model. But organizations don't do that, they do moronic things instead like scrap the above and write a whole new document that's 40 pages of nothing but plagiarizing the CMMI book saying how they do things, but it's almost entirely a lie because no one (except the sacrificial team sent to be appraised) actually does it that way.
inglor 6y ago
I work in Testim.io and I want to say it's much easier to triage and fix e2e tests than unit tests in the CI when using such a platform. Most people just don't but lots of large companies like Microsoft and Salesforce do
robertlagrant 6y ago
> There's not much real difference at all that I can see Not with a tickbox list of features, except that Cypress is insanely nice to use, whereas Selenium is fiddly and unpleasant.
bluGill 6y ago
It is easy to triage end to end tests: the ONE change you made must have broke something. You run into trouble when the test suite takes long enough to run that you forget about ONE change and make a lot of them before testing. If that is the case you are right, the more the test covers the harder it is to triage.

by r0s 6y ago

marcosdumay 6y ago

Hum... Depending on exactly what you test, it's not only the coverage that increases with a larger scope, but the tests also break less. If you engineers things for testing¹, e2e tests need much less maintenance than unit tests. Yes, their failure is also more severe, but overall, they are a large gain on that area. But then, you may discover that your current project just can not be adapted to enable e2e tests, like some projects can not be adapted to enable unity or integration tests. The single size preaching is completely flawed. 1 - Not on the ways the unity testing fans uses that phrase obviously, but if you keep your (human and machine) interfaces stable and generic enough to survive some software evolution.

username90 6y ago

> and they break more often than unit tests. That isn't my experience, unit tests breaks all the time and has to be rewritten from simple refactoring while larger tests pass as long as you keep behavior the same. This means that the bigger tests are way better for refactoring and safely making changes to your code. If most of your tests are unit tests and you don't have bigger tests covering the same things then refactoring safely is hell.

jschwartzi 6y ago
If your unit tests are breaking all the time, try testing less at the unit level and more at the integration level. There are a lot of dogmatists out there who preach that unit tests should have "X % coverage" no matter what, but that's a ridiculous claim. Different projects will require different mixes of testing strategy, and part of the engineering process is arriving at your verification and validation methods, codifying them, ensuring that they provide valid output, and making sure you're using your time efficiently. If most of your time is spent fixing tests, re-evaluate whether those tests are worth doing that way. Your time might be better spent testing at a higher level or even manually.
- username90 6y ago
  Right, I never write traditional unit tests, I just test the API boundaries as you said. Sometimes it makes sense to draw a new API boundary in the code for some complex logic that is unlikely to change, but most of my tests looks pretty close to others integration tests.

mannykannot 6y ago

Unit tests don't just fail to find bugs because they don't give complete coverage (all testing has that problem, and the author is mistaken to say that end-to-end testing shows all bugs), they fail to find bugs resulting from faulty assumptions about how the units will interact. Boeing's faulty Starliner test demonstrated how that goes [1]. The true value in unit testing is in revealing the bugs it will find early in the process. One benefit of this is there will likely be fewer to be found in the end-to-end testing (or on deployment.) There is no value in debating which is better. The important issue is how to allocate your testing budget effectively, because there is never even close to enough time to exhaustively test. [1] https://www.engadget.com/2020-02-29-boeing-starliner-failed-first-flight-report.html https://www.engadget.com/2020-02-29-boeing-starliner-failed-...

ozim 6y ago
Pyramid approach, inverted pyramid, crab shaped does not make sense because it tries to be catch all guideline. Same with any 'maturity models' for software projects. People are forgetting there is something like 'risk based' approach to testing. This is the only way to allocate budget effectively, where stuff that can drop half of your database should be tested well and stuff that will make your UI look funny probably less.
- Jtsummers 6y ago
  I hate to say this, but I do think CMMI's model is a good idea at its core. The model itself is actually pretty solid. That is, everything it describes is something a good organization should do, and the requirement to document the processes + train people is a sound one. But the appraisals and how most businesses treat it are, well, moronic. The appraisals are easily gamed so that if you do everything on paper you're pretty much going to get your target level (3 is good enough for 99% of contracts if matters at all). And businesses try to treat the model as a process, when it's not. The former means that the appraisal results are useless. You cannot judge anyone by being level 3 or 5. That often just means they played the game better, not that they're actually better. The latter, though, is the worst part. CMMI is broken into a number of process areas (I'm happy to report I no longer recall the number of areas). It describes them in psuedo-process or procedural terms. That is, you could take what they describe and make a process, but it would be a very linear and simple process. What's intended is for you to say how your process maps to their model. But organizations don't do that, they do moronic things instead like scrap the above and write a whole new document that's 40 pages of nothing but plagiarizing the CMMI book saying how they do things, but it's almost entirely a lie because no one (except the sacrificial team sent to be appraised) actually does it that way.

inglor 6y ago

I work in Testim.io and I want to say it's much easier to triage and fix e2e tests than unit tests in the CI when using such a platform. Most people just don't but lots of large companies like Microsoft and Salesforce do

robertlagrant 6y ago

> There's not much real difference at all that I can see Not with a tickbox list of features, except that Cypress is insanely nice to use, whereas Selenium is fiddly and unpleasant.

bluGill 6y ago

It is easy to triage end to end tests: the ONE change you made must have broke something. You run into trouble when the test suite takes long enough to run that you forget about ONE change and make a lot of them before testing. If that is the case you are right, the more the test covers the harder it is to triage.