3 ms·

Another day, another blog post by someone who misunderstands the purpose of unit tests and mocks, probably because they've never worked in a huge codebase with

Another day, another blog post by someone who misunderstands the purpose of unit tests and mocks, probably because they've never worked in a huge codebase with complex business logic and lots of precise edge cases.

xena 3y ago
What is the proper purpose of unit tests and mocks then?
- wavemode 3y ago
  Testing against real services (probably in some kind of staging environment) is mandatory, IMO, but it is only really useful for verifying the happy path, or some simple failure modes. Unit tests can verify the system's logic is correct in the face of complex edge cases that are difficult to replicate in integration or end-to-end testing. Unit tests also allow you to add new functionality to the codebase with confidence, as you can know that your modifications did not break some specific edge case or cause some bug regression. If you're in a dynamic language they also help ensure your refactoring did not introduce some type mismatch somewhere. Which is why I draw the conclusion that anyone who thinks unit testing and mocking boils down to `response = "derp"; assert response == "derp"`, must not have much experience working in such codebases.
ebiester 3y ago
So, which references would you point to? I've work in such environments. I've been bitten by mocks and I've been bitten by false positives from ugly E2E systems. I've seen flaky E2E tests linger indefinitely, so I know why trying to spin up the whole system for a small set of tests is bad. I also know that test data management and systems like VCR are very difficult to maintain. So, we mock because it's more painful to do otherwise. (That is; the test data management does not keep every edge case because it's extremely expensive to do so.) Yet, the mocking is painful too in the complex system. There's no way to tell if the mock's response has been changed from underneath. You introduce false confidence in your code. Most bugs, in my experience, are based on a misalignment of expectations between two separate parts of the system (or internal/external systems.) Some of this is based on changes, others are based on unknown edge cases. So, given that this is a hard problem, where do you point intermediate practitioners?
- wavemode 3y ago
  You state unit tests and mocks do not catch all bugs. I absolutely agree with you. But the answer is not, to not do it at all. The answer is to test in multiple ways. Unit, integration, E2E, manual - how much of each you need depends on what kind of system you're building.

xena 3y ago

What is the proper purpose of unit tests and mocks then?

wavemode 3y ago
Testing against real services (probably in some kind of staging environment) is mandatory, IMO, but it is only really useful for verifying the happy path, or some simple failure modes. Unit tests can verify the system's logic is correct in the face of complex edge cases that are difficult to replicate in integration or end-to-end testing. Unit tests also allow you to add new functionality to the codebase with confidence, as you can know that your modifications did not break some specific edge case or cause some bug regression. If you're in a dynamic language they also help ensure your refactoring did not introduce some type mismatch somewhere. Which is why I draw the conclusion that anyone who thinks unit testing and mocking boils down to `response = "derp"; assert response == "derp"`, must not have much experience working in such codebases.

ebiester 3y ago

So, which references would you point to? I've work in such environments. I've been bitten by mocks and I've been bitten by false positives from ugly E2E systems. I've seen flaky E2E tests linger indefinitely, so I know why trying to spin up the whole system for a small set of tests is bad. I also know that test data management and systems like VCR are very difficult to maintain. So, we mock because it's more painful to do otherwise. (That is; the test data management does not keep every edge case because it's extremely expensive to do so.) Yet, the mocking is painful too in the complex system. There's no way to tell if the mock's response has been changed from underneath. You introduce false confidence in your code. Most bugs, in my experience, are based on a misalignment of expectations between two separate parts of the system (or internal/external systems.) Some of this is based on changes, others are based on unknown edge cases. So, given that this is a hard problem, where do you point intermediate practitioners?

wavemode 3y ago
You state unit tests and mocks do not catch all bugs. I absolutely agree with you. But the answer is not, to not do it at all. The answer is to test in multiple ways. Unit, integration, E2E, manual - how much of each you need depends on what kind of system you're building.