4 ms·

In my experience, if you try to do without unit tests, your integration tests will grow to fill in the gap. This will slow you down in a two-fold way: 1. As yo

In my experience, if you try to do without unit tests, your integration tests will grow to fill in the gap. This will slow you down in a two-fold way: 1. As your integration tests will both more numerous and complex to cover corner case scenarios, the time it takes to execute the suite will grow too. And since the integration test is more opaque too, this price has to be paid by both you and all the developers that touch something that might have used your module. 2. The time at which you receive the feedback of the test is already too late. You are no longer doing experimental changes, you are trying to push this unit of work into the code base and be done with it. Either you will be tempted to gloss over any divergence you find (quality price) or you will have to execute some relevant subset of your integration testsuite during your coding/OODA loop (execution speed price).

crdoconnor 11y ago
I haven't found this to be a problem at all. Integration tests follow a power law. If you segregate your tests into "smoke" and "non-smoke", run the former before merging code and the latter continuously and randomly and in parallel, you'll get notification of 95% of regressions within 5 minutes and the other 5% within a few hours. Choosing unit tests because of speed is nearly always a premature optimization. IME everywhere that I worked with that relied exclusively on unit tests for automated regression testing didn't trust them enough to do releases without some manual user testing, so the project fell back into waterfall mode. Integration tests tested a lot more realistically and gave more freedom to refactor technical debt (since changing the code didn't require changing the test as often).
- Bahamut 11y ago
  The main benefit of unit tests is the speed one can run them at, and the feedback. Unit tests make it vastly faster to figure out the culprit in a broken sector of code almost all the time, since it narrows down what code you have to check for any logic errors. It also so happens that unit tests tend to encourage better writing of code into reuseable units. Unit tests cannot catch everything though, and for stuff like how multiple units are wired together, that's where integration tests come into play. A good codebase has both types - even combined they won't catch everything, but they should capture almost all business logic issues.
  - crdoconnor 11y ago
    I've found that where code involves complex logic, unit tests often are more effective for the reasons you gave. However, a lot (perhaps most) code is more about integrating disparate components (browsers, lots of different kinds of external APIs, databases, times, distributed task processors, etc.) rather than lots of complex logic and for that kind of code, unit tests are substantially less effective and more horrible to use whereas integration tests really shine. I think ideally, you should keep the two kinds of code strongly segregated, with clean APIs between them, so you can easily use the appropriate kind of testing for either one. I don't agree that a good code base necessarily has both types. A web app with relatively shallow logic and lots of integrations only really benefits from having integration tests. A library to convert human readable strings into dates likewise gains nothing from having integration tests. Similarly, a messy spaghetti codebase with a disgusting mix of both types of code can only really benefit from integration testing until it's substantially refactored and the two types of code are segregated.
    - dennisgorelik 11y ago
      > you should keep the two kinds of code strongly segregated, with clean APIs between them Why _strongly_ segregated? Nothing stops you to write unit tests for the methods that should be covered by unit tests and integration tests for the functionality that needs to be covered by integration tests. How would code segregation help?
      - crdoconnor 11y ago
        If the code is not well segregated you won't be able to write a clean unit test that tests the complex logic without also mocking a bunch of stuff. That test will also often break as you refactor the various connecting APIs it surrounds. Tests that break in the presence of changed API contracts (as opposed to bugs) are vastly less valuable than tests which only go red in the presence of a bug.
- vkou 11y ago
  5% regressions caught within a few hours is a huge price to pay on larger teams - especially if there are multiple other services depending on yours.
  - crdoconnor 11y ago
    Not unless you release your code to production every 20 minutes.
  - dllthomas 11y ago
    A huge price to pay relative to what what alternative?
tkinom 11y ago
I have different point of views. For me, Integration tests are always better. I normally structure two type of integrations test: 5-10 minutes tests that run before every checking. Over night, over weekend tests that runs every night and every weekend. 5-10 minutes are enough time for 200 UX features level tests for both embedded system and Web backend/frontend integration type project, for me, With those test coverage in place. The worst time I receive the feedback from the test failures are one day or one weekend. All the changes are all within my mind that might cost the failure. BTW, ALL checkin code should be passing ALL integration tests. If they don't, the highest priority is fixed the test failures and not develop any new feature.

crdoconnor 11y ago

I haven't found this to be a problem at all. Integration tests follow a power law. If you segregate your tests into "smoke" and "non-smoke", run the former before merging code and the latter continuously and randomly and in parallel, you'll get notification of 95% of regressions within 5 minutes and the other 5% within a few hours. Choosing unit tests because of speed is nearly always a premature optimization. IME everywhere that I worked with that relied exclusively on unit tests for automated regression testing didn't trust them enough to do releases without some manual user testing, so the project fell back into waterfall mode. Integration tests tested a lot more realistically and gave more freedom to refactor technical debt (since changing the code didn't require changing the test as often).

Bahamut 11y ago
The main benefit of unit tests is the speed one can run them at, and the feedback. Unit tests make it vastly faster to figure out the culprit in a broken sector of code almost all the time, since it narrows down what code you have to check for any logic errors. It also so happens that unit tests tend to encourage better writing of code into reuseable units. Unit tests cannot catch everything though, and for stuff like how multiple units are wired together, that's where integration tests come into play. A good codebase has both types - even combined they won't catch everything, but they should capture almost all business logic issues.
- crdoconnor 11y ago
  I've found that where code involves complex logic, unit tests often are more effective for the reasons you gave. However, a lot (perhaps most) code is more about integrating disparate components (browsers, lots of different kinds of external APIs, databases, times, distributed task processors, etc.) rather than lots of complex logic and for that kind of code, unit tests are substantially less effective and more horrible to use whereas integration tests really shine. I think ideally, you should keep the two kinds of code strongly segregated, with clean APIs between them, so you can easily use the appropriate kind of testing for either one. I don't agree that a good code base necessarily has both types. A web app with relatively shallow logic and lots of integrations only really benefits from having integration tests. A library to convert human readable strings into dates likewise gains nothing from having integration tests. Similarly, a messy spaghetti codebase with a disgusting mix of both types of code can only really benefit from integration testing until it's substantially refactored and the two types of code are segregated.
  - dennisgorelik 11y ago
    > you should keep the two kinds of code strongly segregated, with clean APIs between them Why _strongly_ segregated? Nothing stops you to write unit tests for the methods that should be covered by unit tests and integration tests for the functionality that needs to be covered by integration tests. How would code segregation help?
    - crdoconnor 11y ago
      If the code is not well segregated you won't be able to write a clean unit test that tests the complex logic without also mocking a bunch of stuff. That test will also often break as you refactor the various connecting APIs it surrounds. Tests that break in the presence of changed API contracts (as opposed to bugs) are vastly less valuable than tests which only go red in the presence of a bug.
vkou 11y ago
5% regressions caught within a few hours is a huge price to pay on larger teams - especially if there are multiple other services depending on yours.
- crdoconnor 11y ago
  Not unless you release your code to production every 20 minutes.
- dllthomas 11y ago
  A huge price to pay relative to what what alternative?

tkinom 11y ago

I have different point of views. For me, Integration tests are always better. I normally structure two type of integrations test: 5-10 minutes tests that run before every checking. Over night, over weekend tests that runs every night and every weekend. 5-10 minutes are enough time for 200 UX features level tests for both embedded system and Web backend/frontend integration type project, for me, With those test coverage in place. The worst time I receive the feedback from the test failures are one day or one weekend. All the changes are all within my mind that might cost the failure. BTW, ALL checkin code should be passing ALL integration tests. If they don't, the highest priority is fixed the test failures and not develop any new feature.