5 ms·

> The heart of the argument presented is that using mocks in unit tests is problematic (...) If you use real implementations No. As you say, the moment you hav

> The heart of the argument presented is that using mocks in unit tests is problematic (...) If you use real implementations No. As you say, the moment you have a real out-of-process dependency, you no longer have a unit test, but an "integration-with-external world" test. The heart of the argument is to not mock out internal (in-process) business logic. Instead: a) do not use mocks (e.g. leveraging Moq); use fakes (aka "simulators"), i.e. proper classes having in-memory implementations of the out-of-process dependencies and crucially b) replace only out-of-process dependencies, which usually are flaky/nondeterministic. Never replace internal business logic. If you feel you need to replace internal business logic, you are not following the functional core architectural pattern. After you refactor to functional core you will no longer need to replace any dependencies in your unit test, as there won't be any - you will just call your tested pure method and make assertions on the returned value. Special case here is testing "top-level/controller" logic. Here you need to use the in-memory fakes, but there will be only few, reused across all tests, and such test will be an "end-to-end internal-business-logic integration test", but still it will have all the properties of a unit test - it will be fast, deterministic, and you will be able to run it as part of a unit test suite out-of-the-box, with no environment setup necessary. Further reading: https://enterprisecraftsmanship.com/posts/when-to-mock/ https://enterprisecraftsmanship.com/posts/when-to-mock/ > If you want to change code you will always measure the effect it will have on the code and tests are incidental, not the primary concern Exactly. Mocks are very brittle and break all the time, causing unnecessary rework. If you instead rely on a small set of fakes of out-of-process dependencies, you will drastically reduce test suite rework and improve signal-to-noise ratio. > Using a real dependency, I would be left in the unfortunate situation of depending on knowledge of the internals of that dependency. And what happens if the real dependency changes behavior, but you forget to update your mock? You will have a test that will be running against a mock that simulates obsolete behavior, no longer present in production. In worst case scenario the test will be green/passing, while there is a bug in production. Will you remember to always comb over your entire test suite and review all mocks to faithfully simulate actual production behavior? > It may not allow me to execute specific paths via pure configuration at all. If you write fakes, you will have full control over how to configure them.

Aeolun 5y ago
> Will you remember to always comb over your entire test suite and review all mocks to faithfully simulate actual production behavior? This doesn’t make sense. If I write unit tests for my ‘FunctionExecutor’, and it has a dependency with a function ‘shouldExecute’, I don’t care how that function is implemented, only that it returns a boolean. If the implementation of my ‘shouldExecute’ function changes, there is no need to update any other test as long as it still returns a boolean. I do agree that what you call fakes are better, because you get the extra guarantee of them implementing the same interface (e.g. your compiler will scream bloody murder if you don’t update both).
- spawarotti 5y ago
  > I don’t care how that function is implemented, only that it returns a boolean Let's take your example of "shouldExecute". I assume your unit test operates on some inputs (with the values provided inside the unit test itself, naturally), and "shouldExecute" has potentially some nontrivial logic in it. Say, it reads value of some environment variable and if it is right, returns "true". Now there are two possibilities: a) your test inputs are made up. For example, you never set the environment variable value, and coerce "shouldExecute" to always return "true" anyway. The problem with such a test is that it is fiction, not a test of an actual production behavior. Sure, you will test what would happen if the logic would determine it should execute given no environment variable is set, but this will never happen in production. In production lack of environment variable would result in "shouldExecute" returning "false" and you should test for *that*. So you do care about the details of "shouldExecute", because you need to be aware that it returns "true" only if appropriate environment variable is set. And if you don't care if the "shouldExecute" returns true or false, then why do you call it in the first place? What are you even testing? I hope your "shouldExecute" doesn't have any side effects you depend on, and I do hope you do not use code coverage as a goal unto itself. Plus, such a test cannot be used as "executable documentation", because the inputs are simplified to the point of irrelevance, and cannot help in understanding actual program behavior. b) your test inputs are realistic, and reflect actual production behavior. This means you will have properly set up the environment variable value so that "shouldExecute" returns true. With that, the test is similar to actual production behavior, has good bug-catching ability, and can serve as executable specification. But here again you will have to worry about "shouldExecute" implementation. --- Let me offer you another example. Imagine we want to test: Compile(SourceCode sourceCode) { ValidateSyntax(sourceCode); /* complex logic post-processing the sourceCode here */ } I could say here "I am testing Compile, and I do not care if ValidateSyntax throws an exception; I will just coerce it to return without throwing an exception". And then I can write a test that takes as input some simple sourceCode like "blablabla" and claim I have somehow tested the "/* complex logic post-processing the sourceCode here */". But this is silly, in reality such fake input would never survive the validation, and thus testing what happens after is just waste of time. Hence I need to ensure sourceCode passes the *actual* validation, and hence I need to understand the logic inside the "ValidateSyntax" method. It gets worse. Now imagine we have thoroughly tested the complex logic while using mocked out ValidateSyntax, but now the syntax has changed and thus the validate method behaves differently. If I have a mock of the "ValidateSyntax" method I might still feel good - the test is green, the coverage is still high. Except it is all I lie. I run "Compile" on some production data and it blows up. Why? Because the test was working against a "ValidateSyntax" mock that was mocking the result of validating obsolete syntax, which, now, with the changed behavior, would actually not pass validation and blow up. Basically my test was telling my "Compile" method works IF I assume syntax is still the obsolete one, but it doesn't tell me how my code behaves with the current syntax. So every time program behavior changes, I need to go through all my mocks that were duplicating that behavior, and see if they are still faithfully reflecting it. Otherwise I risk ending up with a green test suite that tests nonexistent, impossible program executions.

Aeolun 5y ago

> Will you remember to always comb over your entire test suite and review all mocks to faithfully simulate actual production behavior? This doesn’t make sense. If I write unit tests for my ‘FunctionExecutor’, and it has a dependency with a function ‘shouldExecute’, I don’t care how that function is implemented, only that it returns a boolean. If the implementation of my ‘shouldExecute’ function changes, there is no need to update any other test as long as it still returns a boolean. I do agree that what you call fakes are better, because you get the extra guarantee of them implementing the same interface (e.g. your compiler will scream bloody murder if you don’t update both).

spawarotti 5y ago
> I don’t care how that function is implemented, only that it returns a boolean Let's take your example of "shouldExecute". I assume your unit test operates on some inputs (with the values provided inside the unit test itself, naturally), and "shouldExecute" has potentially some nontrivial logic in it. Say, it reads value of some environment variable and if it is right, returns "true". Now there are two possibilities: a) your test inputs are made up. For example, you never set the environment variable value, and coerce "shouldExecute" to always return "true" anyway. The problem with such a test is that it is fiction, not a test of an actual production behavior. Sure, you will test what would happen if the logic would determine it should execute given no environment variable is set, but this will never happen in production. In production lack of environment variable would result in "shouldExecute" returning "false" and you should test for *that*. So you do care about the details of "shouldExecute", because you need to be aware that it returns "true" only if appropriate environment variable is set. And if you don't care if the "shouldExecute" returns true or false, then why do you call it in the first place? What are you even testing? I hope your "shouldExecute" doesn't have any side effects you depend on, and I do hope you do not use code coverage as a goal unto itself. Plus, such a test cannot be used as "executable documentation", because the inputs are simplified to the point of irrelevance, and cannot help in understanding actual program behavior. b) your test inputs are realistic, and reflect actual production behavior. This means you will have properly set up the environment variable value so that "shouldExecute" returns true. With that, the test is similar to actual production behavior, has good bug-catching ability, and can serve as executable specification. But here again you will have to worry about "shouldExecute" implementation. --- Let me offer you another example. Imagine we want to test: Compile(SourceCode sourceCode) { ValidateSyntax(sourceCode); /* complex logic post-processing the sourceCode here */ } I could say here "I am testing Compile, and I do not care if ValidateSyntax throws an exception; I will just coerce it to return without throwing an exception". And then I can write a test that takes as input some simple sourceCode like "blablabla" and claim I have somehow tested the "/* complex logic post-processing the sourceCode here */". But this is silly, in reality such fake input would never survive the validation, and thus testing what happens after is just waste of time. Hence I need to ensure sourceCode passes the *actual* validation, and hence I need to understand the logic inside the "ValidateSyntax" method. It gets worse. Now imagine we have thoroughly tested the complex logic while using mocked out ValidateSyntax, but now the syntax has changed and thus the validate method behaves differently. If I have a mock of the "ValidateSyntax" method I might still feel good - the test is green, the coverage is still high. Except it is all I lie. I run "Compile" on some production data and it blows up. Why? Because the test was working against a "ValidateSyntax" mock that was mocking the result of validating obsolete syntax, which, now, with the changed behavior, would actually not pass validation and blow up. Basically my test was telling my "Compile" method works IF I assume syntax is still the obsolete one, but it doesn't tell me how my code behaves with the current syntax. So every time program behavior changes, I need to go through all my mocks that were duplicating that behavior, and see if they are still faithfully reflecting it. Otherwise I risk ending up with a green test suite that tests nonexistent, impossible program executions.