4 ms·

> Would also emphasize that while unit testing is often useful for the individual developer working on a piece of code, but as far as ROI, end-to-end integratio

> Would also emphasize that while unit testing is often useful for the individual developer working on a piece of code, but as far as ROI, end-to-end integration testing has the most bang for the buck. If you have the full endpoint tested, for instance, from end-to-end, including database writes, your confidence level goes up by an order of magnitude when you have to modify that functionality. If you have to choose between investing in extensive unit testing and extensive end-to-end API or contract tests, always choose the latter. It's not so clear-cut, unfortunately. Extensive end-to-end tests can be very hard to setup, can be flakey, can take forever to run (especially if you do it on every change), etc. I agree you should have some layer of automated end-to-end testing (and not enough people do), but in the end, I think you have to work towards making the system as compositional as possible, so you can also test things extensively in isolation - the end-to-end tests then serve to make certain that the individual pieces fit together, but don't hit all the edge cases.

Aqueous 3y ago
I can think of few better investments than to have a reliable suite of end-to-end tests running as part of your merge pipeline even if they're difficult to set up. Sleep quality improves once you have this in place. Your tests don't have to be brittle or flaky. If your tests are brittle it is definitely a good investment to fix whatever's causing the brittleness rather than accept it as a fact of life. Having the tests run against an isolated data and infrastructure environment without additional noise from shared activity is a good first step.
- Tainnor 3y ago
  Distributed systems automatically bring challenges that makes it very hard to create reliable test suites. You can reduce the flakey-ness, but I don't think you can ever completely eliminate it. Beyond that, you haven't addressed the fact that a comprehensive end-to-end test suite in a complex system is really, really slow.
  - Aqueous 3y ago
    Just to clarify, I'm talking about integration testing the service itself - posting a payload, saving to the database, producing messages to a mock queue, etc. Test the entire service in isolation from other services and validate that it is behaving correctly. Not end to end across services. You should be mocking out all service dependencies and testing against the contracts for those systems. Our API tests run flawlessly every time because they write against an isolated database with well-defined endpoint and messaging contracts. They also execute all remote operations against a mock API that conforms to those contracts. This is perfectly achievable.
    - Tainnor 3y ago
      I agree that this is achievable but it seems to contradict your original assertion. Mocking out other services doesn't give you the same assurances as end to end testing the whole stack. To me it's a classic tradeoff: the more you integrate in your tests, the more meaningful they are - but also the harder to write and maintain.

Aqueous 3y ago

I can think of few better investments than to have a reliable suite of end-to-end tests running as part of your merge pipeline even if they're difficult to set up. Sleep quality improves once you have this in place. Your tests don't have to be brittle or flaky. If your tests are brittle it is definitely a good investment to fix whatever's causing the brittleness rather than accept it as a fact of life. Having the tests run against an isolated data and infrastructure environment without additional noise from shared activity is a good first step.

Tainnor 3y ago
Distributed systems automatically bring challenges that makes it very hard to create reliable test suites. You can reduce the flakey-ness, but I don't think you can ever completely eliminate it. Beyond that, you haven't addressed the fact that a comprehensive end-to-end test suite in a complex system is really, really slow.
- Aqueous 3y ago
  Just to clarify, I'm talking about integration testing the service itself - posting a payload, saving to the database, producing messages to a mock queue, etc. Test the entire service in isolation from other services and validate that it is behaving correctly. Not end to end across services. You should be mocking out all service dependencies and testing against the contracts for those systems. Our API tests run flawlessly every time because they write against an isolated database with well-defined endpoint and messaging contracts. They also execute all remote operations against a mock API that conforms to those contracts. This is perfectly achievable.
  - Tainnor 3y ago
    I agree that this is achievable but it seems to contradict your original assertion. Mocking out other services doesn't give you the same assurances as end to end testing the whole stack. To me it's a classic tradeoff: the more you integrate in your tests, the more meaningful they are - but also the harder to write and maintain.