4 ms·

The point is: if your tests just assert which methods are called on dependencies with what arguments (or something close to that), they are extremely coupled an

The point is: if your tests just assert which methods are called on dependencies with what arguments (or something close to that), they are extremely coupled and brittle. Almost any change in the implementation will require changing such tests. And by their nature, they probably test some minor low-level things, that are not all that valuable to assert anyway. Mocks enable and encourage this kind of testing. Better tests would assert some kind of higher level properties that are much less likely to change. This often involves making the dependencies you inject during testing some more complete simulations of real implementations. Takes more up front effort to implement them, but can often be re-used between many tests. Tests on one hand help you modify software over time, but on the other hand increase maintenance burden. Being good at testing doesn't mean just writing lots of tests, but being good at judging the value of a test vs the cost of having it, which is very context dependent in itself.

kqr 3y ago
This was an insightful comment. I shouldn't write tests that assert a method on my mock has been called, I should design a mock whose behaviour changes in response to interaction, and then assert that the relevant behaviour has indeed changed! That change would still be the same amount of maintenance (if the underlying implementation changes, the mock will have to be updated to reflect that as well) but the test will communicate more clearly what the intention of the interaction is.
MoreQARespect 3y ago
>This often involves making the dependencies you inject during testing some more complete simulations of real implementations. Takes more up front effort to implement them, but can often be re-used between many tests. At higher levels, the up front effort is higher but most of that work that is much more reusable - reusable across implementations and different languages, even. With tools like playwright and MITM proxy, the state of the art for these highly reusable tools has moved forward considerably in the last 5 years. I can now "TDD" everything in a way that actually makes sense on some projects - even a tweak in CSS (via snapshot driven development). Meanwhile CPU power has also kept accelerating to the point that hermetic end to end tests that were unbearably slow can now be run on a laptop in seconds. Entire suites that used to take hours on CI can be trivially parallelized and run in minutes instead.
bombolo 3y ago
The tests you propose might take 10x longer to run. There is value in having quick tests as a first level of validation.
- Sankozi 3y ago
  Mocks are tracking and storing behaviour and almost always are doing much more than fakes (which globalreset proposes) which are just simple (and ideally, correct) implementations.
Veuxdo 3y ago
> The point is: if your tests just assert which methods are called on dependencies with what arguments (or something close to that), they are extremely coupled and brittle. I'll go further. These tests are worse than useless; they're harmful. - They're the very definition of testing an implementation - You will wasted a ton of time updating them when your implementation changes - They give a false sense of security/productivity
- SAI_Peregrinus 3y ago
  Sometimes they're still necessary. E.g. many devices need particular sequencing in their power-up between multiple rails. The Intel 82598 Ethernet Controller datasheet[1] page 129-132 shows this nicely. If you're writing a driver you have to test that the appropriate GPIO control calls happen in the right sequence (and with the right timings). Of course it's an implementation detail, different ethernet controllers have different power-up sequence requirements, and thus different drivers. Of course you don't want to have to use real hardware for these tests, especially because with some devices incorrect power-up sequencing can cause permanent damage. So you write mocks for the various hardware functions and test the driver with those, and use their counters & the test harness's logging to assert that the sequencing and timing is correct. [1] https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/82598-10-gbe-controller-datasheet.pdf https://www.intel.com/content/dam/www/public/us/en/documents...

kqr 3y ago

This was an insightful comment. I shouldn't write tests that assert a method on my mock has been called, I should design a mock whose behaviour changes in response to interaction, and then assert that the relevant behaviour has indeed changed! That change would still be the same amount of maintenance (if the underlying implementation changes, the mock will have to be updated to reflect that as well) but the test will communicate more clearly what the intention of the interaction is.

MoreQARespect 3y ago

>This often involves making the dependencies you inject during testing some more complete simulations of real implementations. Takes more up front effort to implement them, but can often be re-used between many tests. At higher levels, the up front effort is higher but most of that work that is much more reusable - reusable across implementations and different languages, even. With tools like playwright and MITM proxy, the state of the art for these highly reusable tools has moved forward considerably in the last 5 years. I can now "TDD" everything in a way that actually makes sense on some projects - even a tweak in CSS (via snapshot driven development). Meanwhile CPU power has also kept accelerating to the point that hermetic end to end tests that were unbearably slow can now be run on a laptop in seconds. Entire suites that used to take hours on CI can be trivially parallelized and run in minutes instead.

bombolo 3y ago

The tests you propose might take 10x longer to run. There is value in having quick tests as a first level of validation.

Sankozi 3y ago
Mocks are tracking and storing behaviour and almost always are doing much more than fakes (which globalreset proposes) which are just simple (and ideally, correct) implementations.

Veuxdo 3y ago

> The point is: if your tests just assert which methods are called on dependencies with what arguments (or something close to that), they are extremely coupled and brittle. I'll go further. These tests are worse than useless; they're harmful. - They're the very definition of testing an implementation - You will wasted a ton of time updating them when your implementation changes - They give a false sense of security/productivity

SAI_Peregrinus 3y ago
Sometimes they're still necessary. E.g. many devices need particular sequencing in their power-up between multiple rails. The Intel 82598 Ethernet Controller datasheet[1] page 129-132 shows this nicely. If you're writing a driver you have to test that the appropriate GPIO control calls happen in the right sequence (and with the right timings). Of course it's an implementation detail, different ethernet controllers have different power-up sequence requirements, and thus different drivers. Of course you don't want to have to use real hardware for these tests, especially because with some devices incorrect power-up sequencing can cause permanent damage. So you write mocks for the various hardware functions and test the driver with those, and use their counters & the test harness's logging to assert that the sequencing and timing is correct. [1] https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/82598-10-gbe-controller-datasheet.pdf https://www.intel.com/content/dam/www/public/us/en/documents...