3 ms·

Great article, not the first time I read this argument against the usage of mocks; I never understood though how to solve the problem of the explosion of the pa

Great article, not the first time I read this argument against the usage of mocks; I never understood though how to solve the problem of the explosion of the path that must be tested (usually exponential). As an example, consider a single API that uses ~3 services, and these 3 services have underneath from 1 to 5 other internal or external dependencies (such as time, an external API service, a DB repository, and so on). How can I test this API, the 3 services, and their underneath dependencies without exponential paths to test - I want to be able to cover all the paths of my code, and ensure that my test only tests a single thing (either the API, the service, or the dependency interface); otherwise, it is not an unit test. I always felt like that these type of tests without mocks works super-nice in nice situations without any external, or even complex but internal, dependency; otherwise, it becomes very very hard to test ONLY what I want, and not all the dependencies underneath. Mocks allow me to stub the behaviour of a service/dependency that I can test in a separate fashion, covering all the paths, and ensuring that each unit test covers a single unit of my code, and not the integration of all my components.

asdfasgasdgasdg 5y ago
Your dependencies do need to work reliably if you're not going to use mocks. But if they don't, then mocks might be necessary to ensure test reliability. That said if the interaction between your system and its deps is sufficiently unreliable that you need mocks for test reliability, how are you going to have any confidence in the system's production behavior? Like, if I test one of the workflows of my service, it might involve executing queries and transactions on an underlying database. But these operations have essentially 100% reliability, so the fact that these operations are being "tested" at the same time as my service do not impact test reliability. I gain nothing by mocking them out.
- lobstrosity420 5y ago
  > I gain nothing by mocking [database queries and transactions] them out. What about the fact that you will need to spin up, migrate and possibly seed a database in your CI pipeline? What about the toll this will take on the execution speed of your test suite? Additionally, consider that you also need to test the behavior of your system when the query fails, and using a mock implementation that always throws an exception is a trivial and reliable way of achieving this. > That said if the interaction between your system and it's deps is sufficiently unreliable that you need mocks for test reliability, how are you going to have any confidence in the system's production behavior? Sometimes your codebase depends on external services which are flaky for reasons beyond your control, just the way it be sometimes. Mocks are useful to ensure the system behaves a certain way when everything goes right as well as when everything goes wrong. Ultimately the article raises many good points about avoiding mocks if it can be helped, but don't forget a test that only tests the happy path of your system is not very useful. Mock an error in that dependency you expect to always work and understand what would happen, make the necessary provisions.
  - asdfasgasdgasdg 5y ago
    > What about the fact that you will need to spin up, migrate and possibly seed a database in your CI pipeline? The database system I use can be configured to start up reasonably quickly, and can be configured to operate on memfiles to reduce io pressure on the CI system. In fact, testing against the full scale local database is the only supported methodology for this particular rdbms. > Sometimes your codebase depends on external services which are flaky for reasons beyond your control, just the way it be sometimes. No doubt no doubt. As I mentioned, mocks or fakes might be necessary in a condition like this. > Ultimately the article raises many good points about avoiding mocks if it can be helped, but don't forget a test that only tests the happy path of your system is not very useful. My team uses interception and error injection for this case. We still have the real backend, but requests can be forced to fail either before or after executing on the backend.
    - lobstrosity420 5y ago
      > My team uses interception and error injection for this case. We still have the real backend, but requests can be forced to fail either before or after executing on the backend. Really cool. What technologies are you using to achieve this? We've also had to tackle stuff like this before, but I'm not sure of the optimal way of doing it.
      - asdfasgasdgasdg 5y ago
        To be honest, this part is handled manually via a test value injector. There's a global map of names to "adjusters", which are either values or callbacks that are allowed to manipulate a value. If there's a particular request we want to fail after making, we do something like this: // System under test code: auto status = DoRequest(); TestManipulate("after_request", status); if (IsError(status)) { // Handle errors. } // Test code SetTestValue("after_request", MyErrorStatus()); RunSystemUnderTest(); The module that handles test value manipulation is typically disabled by a global variable, and lives in the cold section under FDO and opt compilation, so it costs essentially nothing. It cannot be enabled in release builds due to the code that enables it being compiled out. We find in many cases this isn't really necessary, as the error handling code in most cases just bubbles up the error, and so it is not particularly interesting to test. There are a few cases where we do this where the error handling is more complex than "return same error to user" or "retry after delay." This type of thing can't always solve the problem, but it's often good enough to get things done.
- hackandtrip 5y ago
  Well, I strongly believe in testing all the queries/transactions (for example the repository pattern is super helpful to separate concerns and allow you to test only what concern your queries and the DB); but those are tested separately from the workflow of your service, if we are talking about unit tests. Why? Because otherwise either you write a gaziliion of tests, or they are unreliable. As an example, if you have in the service you wanna test various code paths, and your queries have paths of their own (what happens if you don't retrieve any object? if some property is null? and so on), I just think it gets messy quickly. My solution is to test things separately, at least in unit tests. Obviously, this has pitfalls (it's easy and fun to write green tests, so not always tests respect the interface of the components and they don't get red even if something is wrong); but that's where integration testing come into play. Have a few code paths where you touch multiple external services, and you want to test that everything actually works? Create integration tests that use either a fake or the actual service in a `staging` environment, take 10x time to run, but test the path that is most important for your logic. Obviously, if you many unit tests, you will have way less integration tests, but they serve different scopes, and one cannot substitute the other!

asdfasgasdgasdg 5y ago

Your dependencies do need to work reliably if you're not going to use mocks. But if they don't, then mocks might be necessary to ensure test reliability. That said if the interaction between your system and its deps is sufficiently unreliable that you need mocks for test reliability, how are you going to have any confidence in the system's production behavior? Like, if I test one of the workflows of my service, it might involve executing queries and transactions on an underlying database. But these operations have essentially 100% reliability, so the fact that these operations are being "tested" at the same time as my service do not impact test reliability. I gain nothing by mocking them out.

lobstrosity420 5y ago
> I gain nothing by mocking [database queries and transactions] them out. What about the fact that you will need to spin up, migrate and possibly seed a database in your CI pipeline? What about the toll this will take on the execution speed of your test suite? Additionally, consider that you also need to test the behavior of your system when the query fails, and using a mock implementation that always throws an exception is a trivial and reliable way of achieving this. > That said if the interaction between your system and it's deps is sufficiently unreliable that you need mocks for test reliability, how are you going to have any confidence in the system's production behavior? Sometimes your codebase depends on external services which are flaky for reasons beyond your control, just the way it be sometimes. Mocks are useful to ensure the system behaves a certain way when everything goes right as well as when everything goes wrong. Ultimately the article raises many good points about avoiding mocks if it can be helped, but don't forget a test that only tests the happy path of your system is not very useful. Mock an error in that dependency you expect to always work and understand what would happen, make the necessary provisions.
- asdfasgasdgasdg 5y ago
  > What about the fact that you will need to spin up, migrate and possibly seed a database in your CI pipeline? The database system I use can be configured to start up reasonably quickly, and can be configured to operate on memfiles to reduce io pressure on the CI system. In fact, testing against the full scale local database is the only supported methodology for this particular rdbms. > Sometimes your codebase depends on external services which are flaky for reasons beyond your control, just the way it be sometimes. No doubt no doubt. As I mentioned, mocks or fakes might be necessary in a condition like this. > Ultimately the article raises many good points about avoiding mocks if it can be helped, but don't forget a test that only tests the happy path of your system is not very useful. My team uses interception and error injection for this case. We still have the real backend, but requests can be forced to fail either before or after executing on the backend.
  - lobstrosity420 5y ago
    > My team uses interception and error injection for this case. We still have the real backend, but requests can be forced to fail either before or after executing on the backend. Really cool. What technologies are you using to achieve this? We've also had to tackle stuff like this before, but I'm not sure of the optimal way of doing it.
    - asdfasgasdgasdg 5y ago
      To be honest, this part is handled manually via a test value injector. There's a global map of names to "adjusters", which are either values or callbacks that are allowed to manipulate a value. If there's a particular request we want to fail after making, we do something like this: // System under test code: auto status = DoRequest(); TestManipulate("after_request", status); if (IsError(status)) { // Handle errors. } // Test code SetTestValue("after_request", MyErrorStatus()); RunSystemUnderTest(); The module that handles test value manipulation is typically disabled by a global variable, and lives in the cold section under FDO and opt compilation, so it costs essentially nothing. It cannot be enabled in release builds due to the code that enables it being compiled out. We find in many cases this isn't really necessary, as the error handling code in most cases just bubbles up the error, and so it is not particularly interesting to test. There are a few cases where we do this where the error handling is more complex than "return same error to user" or "retry after delay." This type of thing can't always solve the problem, but it's often good enough to get things done.
hackandtrip 5y ago
Well, I strongly believe in testing all the queries/transactions (for example the repository pattern is super helpful to separate concerns and allow you to test only what concern your queries and the DB); but those are tested separately from the workflow of your service, if we are talking about unit tests. Why? Because otherwise either you write a gaziliion of tests, or they are unreliable. As an example, if you have in the service you wanna test various code paths, and your queries have paths of their own (what happens if you don't retrieve any object? if some property is null? and so on), I just think it gets messy quickly. My solution is to test things separately, at least in unit tests. Obviously, this has pitfalls (it's easy and fun to write green tests, so not always tests respect the interface of the components and they don't get red even if something is wrong); but that's where integration testing come into play. Have a few code paths where you touch multiple external services, and you want to test that everything actually works? Create integration tests that use either a fake or the actual service in a `staging` environment, take 10x time to run, but test the path that is most important for your logic. Obviously, if you many unit tests, you will have way less integration tests, but they serve different scopes, and one cannot substitute the other!