3 ms·

Clean functions with limited dependencies which encode non-trivial domain logic are indeed well served by unit tests - even if theyre a bit imperative. But, ou

Clean functions with limited dependencies which encode non-trivial domain logic are indeed well served by unit tests - even if theyre a bit imperative. But, outside of contrived interview scenarios this type of code is actually somewhat niche and outside of that niche unit tests suck. Ive worked on tons of projects where it accounted for 1-2% of the code base. Unit tests not only werent capable of catching bugs they werent even capable of replicating 98% of them. Unit test theater. Meanwhile, the other day I did TDD with hermetic end to end tests using playwright, mitmproxy, mailpit and screenshot/snapshot testing. It felt like I was living in the future.

perrygeo 3y ago
In most software, the units themselves are not that interesting. Basic business logic and data transforms, surrounded by a larger codebase that glues all the integration points together into a working stateful system. I've met a few unit-testing extremists who insisted on pristine-mocked unit tests and only unit tests. Their tests were incomprehensible (a good test should serve as documentation and acceptance criteria - but their code was 90% of concerned with mocking service apis) and importantly, their unit tests repeatedly failed to catch serious regressions that should have been apparent to anyone who simply executed "main" without mocks. We had production outages on the regular. The funny thing is, the unit test extremists doubled down, rejecting the idea of integration or e2e testing entirely (yes, I'm serious) and claiming that prod failures indicate we need to unit testing EVERYTHING. They even went so far as to claim developers need not actually run their software, unit testing is fully sufficient. Not surprisingly, their code had perfectly constructed units but no overall design... the units composed into an unmaintainable dumpster fire as an actual system.
- RussianCow 3y ago
  I would generally argue that if your unit tests have mocks, they're not actually unit tests. What you're describing sounds like integration tests that completely missed the point (i.e. the "integration" but).
  - sanderjd 3y ago
    Even very small "units" have dependencies in the sense of other objects or functions that they use. Or at least they should, if concerns are being separated. It's good to use fakes or mocks to make those dependencies not depend on state external to the test, but not to structure tests around what methods are called on which dependencies in what order.
    - RussianCow 3y ago
      Yeah, I guess I would make the distinction between simple stubs and heavyweight mocks and fakes. If you're using a lot of the latter in your unit tests, that's a code smell to me; unit tests should ideally not be concerned with the state of the world, which is where mocks (especially fakes) tend to come in. Obviously there are no absolutes, but that's my rule of thumb and it's served me well.
      - sanderjd 3y ago
        I agree. Mocks tend to be complicated when the api of the dependencies they're mocking is complicated.
  - MoreQARespect 3y ago
    What you call them isn't really important. "Unit tests with mocks" or whatever they are still suck.
    - RussianCow 3y ago
      I think the distinction is still important because it frames how you should write the test. Unit tests basically just assert that, given some inputs, the output of the "unit" is what you would expect. Integration tests, on the other hand, test the business logic that ties different units of code together, which changes the assertions to be less about the direct output and more about the state of the world. For example, if you have a Twilio wrapper, the tests for that wrapper are unit tests because you're just testing that the wrapper makes the correct HTTP calls, etc; but the code using that wrapper should be tested via integration tests to ensure that the correct calls are made, that state transitions are handled properly, etc. This might sound like a meaningless distinction, but I've found that a large portion of bad tests that I've seen in practice were integration tests written in the style of unit tests (i.e. treating stateful code as if it were just a pure function of its inputs). Given that integration testing is inherently harder (as you've alluded to), I'd rather see more unit tests, but that requires an architecture that breaks code down into as small and focused of units as possible (e.g. a functional style), which isn't always feasible for practical reasons.
- sanderjd 3y ago
  Yes, this is an insane way to try to build software. I don't think it's very common at all, but I also don't think it is completely non existent.
makeitdouble 3y ago
Writing unit tests can help catch early bugs, but once they're done and green, I don't think anyone expects them to catch new and exotic bugs (it happens but it's so rare). Instead they stay as a live and enforced documentation of how the target function behaves.
- PartiallyTyped 3y ago
  .. and behaviour validation for any rewrites.
sanderjd 3y ago
I agree with everything else you said but I totally disagree that functions like that are "niche". It's true that many codebases are written such that they have few of them, but that's "accidental" not incidental. Almost nothing is pure math, but almost everything is data transformation that is deterministic given some set of parameters and dependency configuration. And mocking out external state to control the behavior of the thing being tested is great. But the point is not to say "and then this method of that dependency is called with this input", it's to control the behavior of the thing being tested, by controlling what its dependencies return.

perrygeo 3y ago

In most software, the units themselves are not that interesting. Basic business logic and data transforms, surrounded by a larger codebase that glues all the integration points together into a working stateful system. I've met a few unit-testing extremists who insisted on pristine-mocked unit tests and only unit tests. Their tests were incomprehensible (a good test should serve as documentation and acceptance criteria - but their code was 90% of concerned with mocking service apis) and importantly, their unit tests repeatedly failed to catch serious regressions that should have been apparent to anyone who simply executed "main" without mocks. We had production outages on the regular. The funny thing is, the unit test extremists doubled down, rejecting the idea of integration or e2e testing entirely (yes, I'm serious) and claiming that prod failures indicate we need to unit testing EVERYTHING. They even went so far as to claim developers need not actually run their software, unit testing is fully sufficient. Not surprisingly, their code had perfectly constructed units but no overall design... the units composed into an unmaintainable dumpster fire as an actual system.

RussianCow 3y ago
I would generally argue that if your unit tests have mocks, they're not actually unit tests. What you're describing sounds like integration tests that completely missed the point (i.e. the "integration" but).
- sanderjd 3y ago
  Even very small "units" have dependencies in the sense of other objects or functions that they use. Or at least they should, if concerns are being separated. It's good to use fakes or mocks to make those dependencies not depend on state external to the test, but not to structure tests around what methods are called on which dependencies in what order.
  - RussianCow 3y ago
    Yeah, I guess I would make the distinction between simple stubs and heavyweight mocks and fakes. If you're using a lot of the latter in your unit tests, that's a code smell to me; unit tests should ideally not be concerned with the state of the world, which is where mocks (especially fakes) tend to come in. Obviously there are no absolutes, but that's my rule of thumb and it's served me well.
    - sanderjd 3y ago
      I agree. Mocks tend to be complicated when the api of the dependencies they're mocking is complicated.
- MoreQARespect 3y ago
  What you call them isn't really important. "Unit tests with mocks" or whatever they are still suck.
  - RussianCow 3y ago
    I think the distinction is still important because it frames how you should write the test. Unit tests basically just assert that, given some inputs, the output of the "unit" is what you would expect. Integration tests, on the other hand, test the business logic that ties different units of code together, which changes the assertions to be less about the direct output and more about the state of the world. For example, if you have a Twilio wrapper, the tests for that wrapper are unit tests because you're just testing that the wrapper makes the correct HTTP calls, etc; but the code using that wrapper should be tested via integration tests to ensure that the correct calls are made, that state transitions are handled properly, etc. This might sound like a meaningless distinction, but I've found that a large portion of bad tests that I've seen in practice were integration tests written in the style of unit tests (i.e. treating stateful code as if it were just a pure function of its inputs). Given that integration testing is inherently harder (as you've alluded to), I'd rather see more unit tests, but that requires an architecture that breaks code down into as small and focused of units as possible (e.g. a functional style), which isn't always feasible for practical reasons.
sanderjd 3y ago
Yes, this is an insane way to try to build software. I don't think it's very common at all, but I also don't think it is completely non existent.

makeitdouble 3y ago

Writing unit tests can help catch early bugs, but once they're done and green, I don't think anyone expects them to catch new and exotic bugs (it happens but it's so rare). Instead they stay as a live and enforced documentation of how the target function behaves.

PartiallyTyped 3y ago
.. and behaviour validation for any rewrites.

sanderjd 3y ago

I agree with everything else you said but I totally disagree that functions like that are "niche". It's true that many codebases are written such that they have few of them, but that's "accidental" not incidental. Almost nothing is pure math, but almost everything is data transformation that is deterministic given some set of parameters and dependency configuration. And mocking out external state to control the behavior of the thing being tested is great. But the point is not to say "and then this method of that dependency is called with this input", it's to control the behavior of the thing being tested, by controlling what its dependencies return.