4 ms·

Thanks for saying this out loud. I’m a solo dev and in my project I’m doing exactly this: 90% black box integration tests and 10% unit tests for edge cases I ca

Thanks for saying this out loud. I’m a solo dev and in my project I’m doing exactly this: 90% black box integration tests and 10% unit tests for edge cases I cannot trigger otherwise. It buys me precious time to not adjust tests after refactoring. Yet it made me feel like a heretic: everyone knows the testing pyramid and it comes from Google so I must be very wrong.

fiznool 3y ago
You might be interested in the ‘testing trophy’ as an alternative to the traditional pyramid. https://kentcdodds.com/blog/write-tests https://kentcdodds.com/blog/write-tests
- imiric 3y ago
  This advice is so misguided that I'm concerned for our industry it's getting so much traction. > You really want to avoid testing implementation details because it doesn't give you very much confidence that your application is working and it slows you down when refactoring. You should very rarely have to change tests when you refactor code. Unit tests don't need to test implementation details. You could just as well make that mistake with integration or E2E tests. Black box testing is a good practice at all layers. What unit tests do is confirm that the smallest pieces of the system work as expected in isolation. Yes, you should also test them in combination with each other, but it serves you no good if you get a green integration test, when it's likely only testing a small fraction of the functionality of the units themselves. This whole "unit tests slow you down" mentality is incredibly toxic. You know what genuinely slows me down? A suite with hundreds of integration tests, each taking several seconds to run, and depend on external systems. But hey, testcontainers to the rescue, right? Tests shouldn't be a chore, but an integral part of software development. These days I suppose we can offload some of that work to AI, but even that should be done very carefully to ensure that the code is high quality and actually tests what we need. Test code is as important as application code. It's lazy to think otherwise.
  - ric2b 3y ago
    If by "smallest pieces of the system" you mean something like individual classes then you are definitely testing implementation details. Whenever you change a method's parameters in one of those internal classes you'll have unit tests breaking, even though you're just refactoring code. Unit testing at the smallest piece level calcifies the codebase by making refactors much more costly.
    - Aeolun 3y ago
      If I change something at the lowest level in my well abstracted system, only the unit tests for that component will fail, as the tests that ‘use’ that component mock the dependency. As long as the interface between components doesn’t change, you can refactor as much as you want.
      - simonw 3y ago
        I prefer having the freedom to change the interface between my components without then having to update large numbers of mocked tests.
        Aeolun 3y ago
        Sure, that's a tradeoff that you make. Personally I update my implementations more often than I update the interfaces, so I'm happy to take that hit when modifying the interface in trade for knowing exactly where my implementations break.
    - imiric 3y ago
      > If by "smallest pieces of the system" you mean something like individual classes then you are definitely testing implementation details. No, there's nothing definite about that. The "unit" itself is a matter of perspective. Tests should be written from the perspective of the API user in case of the smallest units like classes and some integration tests, and from the perspective of the end user in case of E2E tests. "Implementation details" refers to any functionality that's not visible to the user, which exists at all levels of testing. Not writing tests that rely on those details means that the test is less brittle, since all it cares about is the external interface. _This_ gives you the freedom to refactor how the unit itself works however you want. But, if you change the _external_ interface, then, yes, you will have to update your tests. If that involves a method signature change, then hopefully you have IDE tools to help you update all calling sites, which includes application code as well. Nowadays with AI assistants, this type of mechanical change is easy to automate. If you avoid testing classes, that means that you're choosing to ignore your API users, which very likely is yourself. That seems like a poor decision to make.

fiznool 3y ago

You might be interested in the ‘testing trophy’ as an alternative to the traditional pyramid. https://kentcdodds.com/blog/write-tests https://kentcdodds.com/blog/write-tests

imiric 3y ago
This advice is so misguided that I'm concerned for our industry it's getting so much traction. > You really want to avoid testing implementation details because it doesn't give you very much confidence that your application is working and it slows you down when refactoring. You should very rarely have to change tests when you refactor code. Unit tests don't need to test implementation details. You could just as well make that mistake with integration or E2E tests. Black box testing is a good practice at all layers. What unit tests do is confirm that the smallest pieces of the system work as expected in isolation. Yes, you should also test them in combination with each other, but it serves you no good if you get a green integration test, when it's likely only testing a small fraction of the functionality of the units themselves. This whole "unit tests slow you down" mentality is incredibly toxic. You know what genuinely slows me down? A suite with hundreds of integration tests, each taking several seconds to run, and depend on external systems. But hey, testcontainers to the rescue, right? Tests shouldn't be a chore, but an integral part of software development. These days I suppose we can offload some of that work to AI, but even that should be done very carefully to ensure that the code is high quality and actually tests what we need. Test code is as important as application code. It's lazy to think otherwise.
- ric2b 3y ago
  If by "smallest pieces of the system" you mean something like individual classes then you are definitely testing implementation details. Whenever you change a method's parameters in one of those internal classes you'll have unit tests breaking, even though you're just refactoring code. Unit testing at the smallest piece level calcifies the codebase by making refactors much more costly.
  - Aeolun 3y ago
    If I change something at the lowest level in my well abstracted system, only the unit tests for that component will fail, as the tests that ‘use’ that component mock the dependency. As long as the interface between components doesn’t change, you can refactor as much as you want.
    - simonw 3y ago
      I prefer having the freedom to change the interface between my components without then having to update large numbers of mocked tests.
      - Aeolun 3y ago
        Sure, that's a tradeoff that you make. Personally I update my implementations more often than I update the interfaces, so I'm happy to take that hit when modifying the interface in trade for knowing exactly where my implementations break.
  - imiric 3y ago
    > If by "smallest pieces of the system" you mean something like individual classes then you are definitely testing implementation details. No, there's nothing definite about that. The "unit" itself is a matter of perspective. Tests should be written from the perspective of the API user in case of the smallest units like classes and some integration tests, and from the perspective of the end user in case of E2E tests. "Implementation details" refers to any functionality that's not visible to the user, which exists at all levels of testing. Not writing tests that rely on those details means that the test is less brittle, since all it cares about is the external interface. _This_ gives you the freedom to refactor how the unit itself works however you want. But, if you change the _external_ interface, then, yes, you will have to update your tests. If that involves a method signature change, then hopefully you have IDE tools to help you update all calling sites, which includes application code as well. Nowadays with AI assistants, this type of mechanical change is easy to automate. If you avoid testing classes, that means that you're choosing to ignore your API users, which very likely is yourself. That seems like a poor decision to make.