4 ms·

I can't agree with this. In his example (Transactions), he assumes that there is perfect knowledge of the underlying data and perfect knowledge on the assumpti

I can't agree with this. In his example (Transactions), he assumes that there is perfect knowledge of the underlying data and perfect knowledge on the assumptions made by the teams in the corner cases of each module, which is not the case in practice. Very very often, (one of the most common source of errors in programming large systems), two parts of a program are doing the "right" thing, within their assumption, but when combined together you erroneous results. Unit tests will not help you find them, as they cover the "is this gear/part working", but not the whole machine itself, e.g. is the engine actually running well... In my experience, higher level system tests have a much higher ROI and real work value then simple unit tests. Unit tests just test for made up "hypothetical" simplistic scenarios, and usually are written by the same dev. that did the functionality, and they will not expose blind spots on the implementation, while higher level system/integration tests are more likely to test on real world scenarios.

aeorgnoieang 9y ago
I don't think you're really disagreeing with Kent. And I haven't found that integration (system) tests have a higher ROI – because they're usually much more expensive to write and maintain. And they can be more brittle than your actual code. If your code depends on an external service, should your integration test actually make a (test) request to the real live service endpoints? Maybe! But the more 'real' the test is, the less of your own test it covers relative to all of the other external dependencies. I'm not sure there's much disagreement among you and I and Kent. I can imagine always maintaining some non-zero number of integration test for suitably large code-bases. But I think I get what he's getting at in terms of – ideally – wanting to minimize those kinds of tests by transforming previously unreliable parts of the system into something we can use "as a given" (like integer arithmetic).
- Chaebixi 9y ago
  > And they can be more brittle than your actual code. The brittlest tests I've ever seen have been mock-heavy unit tests. Some of them amount to little more than file-change detectors. Those tests can fail massively even when everything still actually works. They cry wolf so often the team ends up trained to unthinkingly change them so they "go green." > But I think I get what he's getting at in terms of – ideally – wanting to minimize those kinds of tests by transforming previously unreliable parts of the system into something we can use "as a given" (like integer arithmetic). A noble goal, but practically not a lot different from saying we should aim for formally verified software. After all, it's what we have for computer arithmetic example: http://www.cl.cam.ac.uk/~jrh13/slides/jnao-02jun06/slides.pdf http://www.cl.cam.ac.uk/~jrh13/slides/jnao-02jun06/slides.pd... (2006): > Intel uses formal verification quite extensively, e.g. ... verification of...floating-point unit[s]...
  - Blazespinnaker 9y ago
    My experience with unit tests is that they force a developer to think through their logic. So the tests themselves,once passing don’t help that much (though a little), but the act of writing them did have high ROI. The coverage metrics also help management understand what is baked.
  - bonesss 9y ago
    > The brittlest tests I've ever seen have been mock-heavy unit tests Not to knock mocking in general, but the need to replace multiple things at once to test one thing is just a big design smell, and I agree: their usefulness tends to be very limited in terms of logic exercised. Call me a fuddy-dud, but I'm generally working with dependancy injection when doing OOP so stubbing out app layers for test contexts feels kinda natural, and I prefer to make my test mocks the old fashioned way: manually using inheritance. It keeps the pain high if I'm doing something wrong, keeps the tests focused, and means I'm hammering the objects design from the POV of an inheritor/reuser from the get-go. I've noticed that the additions necessary to make a manual mock tend to be a good predictor of future changes required in the code-base, while framework based mocks tend to spend a lot of time re-declaring things we already know.
- crdoconnor 9y ago
  >But the more 'real' the test is, the less of your own test it covers relative to all of the other external dependencies. A strangely underappreciated facet of being a more 'real test' is that you catch more 'real' bugs. >should your integration test actually make a (test) request to the real live service endpoints? Ideally you should be able to run your integration test in a mode that mocks it and a mode that calls the real live service so that you can see problems in your code and (and avoid brittleness) and problems with the service, and distinguish them.
- Chaebixi 9y ago
  > But the more 'real' the test is, the less of your own test it covers relative to all of the other external dependencies. But the more 'real' the test is, the more of your own assumptions get about how your dependencies work get tested. Assumptions are where most bugs are born and hide. PS: Sorry for the multiple replies. HN is being glitchy and my previous comment kept getting truncated.
  - mcphage 9y ago
    > But the more 'real' the test is, the more of your own assumptions get about how your dependencies work get tested. Assumptions are where most bugs are born and hide. Very well put—I like this.
- WalterSear 9y ago
  > If your code depends on an external service, should your integration test actually make a (test) request to the real live service endpoints? That's a terrible idea. You've now coupled your tests to someone else's code. Test the endpoints separately. Wrap the endpoints to allow you convenient dependency injection. Test the wrapper. This wrapper is ideally a very thin layer of boilerplate, that you won't ever touch very much, and shouldn't take much time to write or test. But this is the key, however: use the data specifying the expected outputs of your third party API as input parameters to the test of the wrapper, and expected outputs from the wrapper as parameters for your integration tests. Keep each set of data in one place, so that changing it for one test, changes it for the others. By using shared test datasets in this way, you prevent modules from getting out of synch with each other, and can still replace/update parts in separation.
  - crdoconnor 9y ago
    >That's a terrible idea. You've now coupled your tests to someone else's code. It's a terrible idea not to. If your code isn't coupled to someone else's code your code isn't doing anything useful. Moreover, you should test against the code your code is coupled to as realistically as possible, which ideally means testing against the real thing.
    - WalterSear 9y ago
      >It's a terrible idea not to. If your code isn't coupled to someone else's code your code isn't doing anything useful. Paradoxically, you want your tests separate from the external inputs, because complete control over the inputs is required to systematically and programmatically hit every edge case with every pass of your test suite. You also need the modularity in your tests and code, so you can fix and replace 3rd party endpoints without being afraid of polluting your code's inputs, or having to rewrite your integration suite. This is where the data contracts come in: they make sure that each module of the system and its partners are speaking the same language, even after they have been replaced or updated to handle new 3rd party behaviours.
      - 9y ago
- maxxxxx 9y ago
  "And I haven't found that integration (system) tests have a higher ROI – because they're usually much more expensive to write and maintain." I think the worst code bases to modify are the ones with heavy use of mocking and dependency injection. You still get no feedback about the behavior when run against the real system and you have to maintain a potentially complex mock.
  - aeorgnoieang 9y ago
    > complex mock That's a smell for sure. I've got a codebase with lots of 'mocks' and a good amount of dependency injection, but I don't feel like it's hard to modify. In fact, the mocks and dependency injection make testing possible. But then all of the mocks are really dumb, e.g. implementing an interface in the 'dumbest' way possible. The most complex mock behavior I have is return a value supplied as a parameter to this object's constructor. The mock then is really just a way to explicitly encode assumptions about (possible) behavior of whatever it is being mocked or injected.
  - bonesss 9y ago
    I think the issue you're describing, a massive variance in test and system behaviour, is symptomatic of a poor delineation of the unit/integration test dichotomy. It's almost a skill within a skill, but the moment a mock starts taking on some life or complexity is a moment you should be backing out and refactoring. Complex mocks shouldn't exist. Mocks of complex interfaces should almost entire be sustainable by your domain objects if they're exposed to your testing 'datalayer'. So if they're required to test, and I've seen many codebases where that's true, then you've generally got a major design issue that is unaddressed in the main codebase. Personally I use mocks very sparingly coupled with a few strategic stubs (ie hard coded "databases" of domain objects), and almost always code them manually by hand. That way it's very clear if I'm doing something dumb, the sytems DI is leveraged, and the mock is highlighting design issues in a re-use/inheritance context.

aeorgnoieang 9y ago

I don't think you're really disagreeing with Kent. And I haven't found that integration (system) tests have a higher ROI – because they're usually much more expensive to write and maintain. And they can be more brittle than your actual code. If your code depends on an external service, should your integration test actually make a (test) request to the real live service endpoints? Maybe! But the more 'real' the test is, the less of your own test it covers relative to all of the other external dependencies. I'm not sure there's much disagreement among you and I and Kent. I can imagine always maintaining some non-zero number of integration test for suitably large code-bases. But I think I get what he's getting at in terms of – ideally – wanting to minimize those kinds of tests by transforming previously unreliable parts of the system into something we can use "as a given" (like integer arithmetic).

Chaebixi 9y ago
> And they can be more brittle than your actual code. The brittlest tests I've ever seen have been mock-heavy unit tests. Some of them amount to little more than file-change detectors. Those tests can fail massively even when everything still actually works. They cry wolf so often the team ends up trained to unthinkingly change them so they "go green." > But I think I get what he's getting at in terms of – ideally – wanting to minimize those kinds of tests by transforming previously unreliable parts of the system into something we can use "as a given" (like integer arithmetic). A noble goal, but practically not a lot different from saying we should aim for formally verified software. After all, it's what we have for computer arithmetic example: http://www.cl.cam.ac.uk/~jrh13/slides/jnao-02jun06/slides.pdf http://www.cl.cam.ac.uk/~jrh13/slides/jnao-02jun06/slides.pd... (2006): > Intel uses formal verification quite extensively, e.g. ... verification of...floating-point unit[s]...
- Blazespinnaker 9y ago
  My experience with unit tests is that they force a developer to think through their logic. So the tests themselves,once passing don’t help that much (though a little), but the act of writing them did have high ROI. The coverage metrics also help management understand what is baked.
- bonesss 9y ago
  > The brittlest tests I've ever seen have been mock-heavy unit tests Not to knock mocking in general, but the need to replace multiple things at once to test one thing is just a big design smell, and I agree: their usefulness tends to be very limited in terms of logic exercised. Call me a fuddy-dud, but I'm generally working with dependancy injection when doing OOP so stubbing out app layers for test contexts feels kinda natural, and I prefer to make my test mocks the old fashioned way: manually using inheritance. It keeps the pain high if I'm doing something wrong, keeps the tests focused, and means I'm hammering the objects design from the POV of an inheritor/reuser from the get-go. I've noticed that the additions necessary to make a manual mock tend to be a good predictor of future changes required in the code-base, while framework based mocks tend to spend a lot of time re-declaring things we already know.
crdoconnor 9y ago
>But the more 'real' the test is, the less of your own test it covers relative to all of the other external dependencies. A strangely underappreciated facet of being a more 'real test' is that you catch more 'real' bugs. >should your integration test actually make a (test) request to the real live service endpoints? Ideally you should be able to run your integration test in a mode that mocks it and a mode that calls the real live service so that you can see problems in your code and (and avoid brittleness) and problems with the service, and distinguish them.
Chaebixi 9y ago
> But the more 'real' the test is, the less of your own test it covers relative to all of the other external dependencies. But the more 'real' the test is, the more of your own assumptions get about how your dependencies work get tested. Assumptions are where most bugs are born and hide. PS: Sorry for the multiple replies. HN is being glitchy and my previous comment kept getting truncated.
- mcphage 9y ago
  > But the more 'real' the test is, the more of your own assumptions get about how your dependencies work get tested. Assumptions are where most bugs are born and hide. Very well put—I like this.
WalterSear 9y ago
> If your code depends on an external service, should your integration test actually make a (test) request to the real live service endpoints? That's a terrible idea. You've now coupled your tests to someone else's code. Test the endpoints separately. Wrap the endpoints to allow you convenient dependency injection. Test the wrapper. This wrapper is ideally a very thin layer of boilerplate, that you won't ever touch very much, and shouldn't take much time to write or test. But this is the key, however: use the data specifying the expected outputs of your third party API as input parameters to the test of the wrapper, and expected outputs from the wrapper as parameters for your integration tests. Keep each set of data in one place, so that changing it for one test, changes it for the others. By using shared test datasets in this way, you prevent modules from getting out of synch with each other, and can still replace/update parts in separation.
- crdoconnor 9y ago
  >That's a terrible idea. You've now coupled your tests to someone else's code. It's a terrible idea not to. If your code isn't coupled to someone else's code your code isn't doing anything useful. Moreover, you should test against the code your code is coupled to as realistically as possible, which ideally means testing against the real thing.
  - WalterSear 9y ago
    >It's a terrible idea not to. If your code isn't coupled to someone else's code your code isn't doing anything useful. Paradoxically, you want your tests separate from the external inputs, because complete control over the inputs is required to systematically and programmatically hit every edge case with every pass of your test suite. You also need the modularity in your tests and code, so you can fix and replace 3rd party endpoints without being afraid of polluting your code's inputs, or having to rewrite your integration suite. This is where the data contracts come in: they make sure that each module of the system and its partners are speaking the same language, even after they have been replaced or updated to handle new 3rd party behaviours.
    - 9y ago
maxxxxx 9y ago
"And I haven't found that integration (system) tests have a higher ROI – because they're usually much more expensive to write and maintain." I think the worst code bases to modify are the ones with heavy use of mocking and dependency injection. You still get no feedback about the behavior when run against the real system and you have to maintain a potentially complex mock.
- aeorgnoieang 9y ago
  > complex mock That's a smell for sure. I've got a codebase with lots of 'mocks' and a good amount of dependency injection, but I don't feel like it's hard to modify. In fact, the mocks and dependency injection make testing possible. But then all of the mocks are really dumb, e.g. implementing an interface in the 'dumbest' way possible. The most complex mock behavior I have is return a value supplied as a parameter to this object's constructor. The mock then is really just a way to explicitly encode assumptions about (possible) behavior of whatever it is being mocked or injected.
- bonesss 9y ago
  I think the issue you're describing, a massive variance in test and system behaviour, is symptomatic of a poor delineation of the unit/integration test dichotomy. It's almost a skill within a skill, but the moment a mock starts taking on some life or complexity is a moment you should be backing out and refactoring. Complex mocks shouldn't exist. Mocks of complex interfaces should almost entire be sustainable by your domain objects if they're exposed to your testing 'datalayer'. So if they're required to test, and I've seen many codebases where that's true, then you've generally got a major design issue that is unaddressed in the main codebase. Personally I use mocks very sparingly coupled with a few strategic stubs (ie hard coded "databases" of domain objects), and almost always code them manually by hand. That way it's very clear if I'm doing something dumb, the sytems DI is leveraged, and the mock is highlighting design issues in a re-use/inheritance context.