8 ms·

An overly aggressive mock can work fine, but break much later

67 points by ingve 11mo ago

gsnedders 11mo ago
Arguably this is a problem in when the patch is unapplied. Presumably in the coverage case it’s being called by a trace function, which inevitably runs during test execution — and while we want the trace function to be called during the test function, we really want it without any patches the test function is using. But this arguably requires both an ability for the trace function to opt-out of patches and for the patcher to provide a way to temporarily disable all of them.
esafak 11mo ago
It is worth pointing out that you can often use containerized services as an alternative to mocking.
Izkata 11mo ago
> In Why your mock doesn’t work I explained this rule of mocking: > Mock where the object is used, not where it’s defined. For anyone looking for generic advice, this is a quirk of python due to how imports work in that language (details in the linked post) and shouldn't be considered universal.
kaashif 11mo ago
Why even mock anything in this example? You need to read the source code to work out what to mock, reaching deep inside the code to name some method to mock. But what if you just passed in the contents of the file or something? Edit: oh wait actually this is what the very last line in the blog post says! But I think it should be emphasized more!
lelandbatey 11mo ago
I feel like the #1 reason mocks break looks nothing like this and instead looks like: you change the internal behaviors of a function/method and now the mocks interact differently with the underlying code, forcing you to change the mocks. Which highlights how awful mocking as a concept is; it is of truly limited usefulness for anything but the most brittle of tests. Don't test the wrong things; if you care about some precondition, that should be an input. If you need to measure a side effect, that should be an output. Don't tweak global state to do your testing.
- bluGill 11mo ago
  Most of the real world is about manipulating the real world. For algorithms it is fine to say depend on the pure inputs/outputs. However what we care about is that global state is manipulated correctly and so the integration tests that verify that are what are important. In most cases your algorithm shouldn't be unit tested separately since it is only used in one place and changes when the users change: there is no point in extra tests. If the algorithm is used in many places comprehensive unit tests are important, but they get in the way when the algorithm is used only once and so the tests just inhibit changes to the algorithm as requirements change (you have to change the user, the integration tests, and the unit tests that are redundant). As such I disagree. Global state is what you should be testing - but you need to be smart about it. How you setup and verify global state matters. Don't confuse global state above with global state of variables, I mean the external state of the program before and after, which means network, file, time, and other IO things.
  - refactor_master 11mo ago
    IO and global state is also just inputs that can be part of arrange-act-assert. Instead of mocking your database call to always return "foo" when the word "SELECT" is in the query, insert a real "foo" in a real test database and perform a real query. Again I've heard "but what if my database/table changes so rapidly that I need the mock so I don't need to change the query all the time", in which case you ought to take a moment to write down what you're trying to accomplish, rather than using mocks to pave over poor architectural decisions. Eventually, the query fails and the mock succeeds, because they were completely unrelated. So far I've only seen mocks fail eventually and mysteriously. With setups and DI you can treat things mostly as a black box from a testing point of view, but when mocks are involved you need surgical precision to hit the right target at the right time.
    - deleted 11mo ago
      [deleted]
    - bluGill 11mo ago
      I don't understand the "but what if my database/table changes so rapidly that I need the mock so I don't need to change the query all the time". Are they changing your database schema at runtime (not just software update time when you can do migrations)? Otherwise you should have a test helper that creates your entire database schema so that when it changes you can update the schema in one place and all tests are re-run against the new one. (you should also have some insert common test data helpers that are easy to update as needed should the new schema affect the changes) I haven't seen mocks fail mysteriously. I've seen them fail often though because requirements change and instead of updating the callers (generally a small number) you end up with 200 tests failing and give up because updating all the tests is too hard. Mocks are always about implementation details - sometimes you have no choice, but the more you can test actual behavior the better.
dherls 11mo ago
This blog post talks as if mocking the `open` function is a good thing that people should be told how to do. If you are mocking anything in the standard library your code is probably structured poorly. In the example the author walks through, a cleaner way would be to have the second function take the Options as a parameter and decouple those two functions. You can then test both in isolation.
- bluGill 11mo ago
  Details matters, but good test doubles here are important. You want to capture all calls to IO and do something different. You don't want tests to break because someone has a different filesystem, didn't set their home directory as you want it setup, or worse is trying to run two different tests at the same time and the other test is changing files the other wants. Note that I said test doubles. Mocks are a bit over specific - they are about verifying functions are called at the right time with the right arguments, but the easy ability to set return values makes it easy to abuse them for other things (this abuse is good, but it is still abuse of the intent). In this case you want a fake: a smart service that when you are in a test setups a temporary directory tree that contains all the files you need in the state that particular test needs, and destroys that when the test is done (with an optional mode to keep it - useful if a test fails to see debug). Depending on your situation you may need something for network services, time, or other such things. Note that in most cases a filesystem itself is more than fast enough to use in tests, but you need isolation from other tests. There are a number of ways to create this fake, it could override open, or it could just be a GetMyProgramDir function that you override are two that I can think of.
  - jpollock 11mo ago
    Your tests are either hermetic, or they're flaky. That means the test environment needs to be defined and versioned with the code.
  - dherls 11mo ago
    Even in the case you mention you really shouldn't be overriding these methods. Your load settings method should take the path of the settings file as an argument, and then your test can set up all the fake files you want with something like python's tempfile package
BoiledCabbage 11mo ago
The mock discussion still misses the real solution, which is to refactor the code so that you have a function that simple reads the file and returns json that is essentially a wrapper around open and doesn't need to be tested. Then have your main function take in that json as a parameter (or class wrapping that json). Then your code becomes the ideal code. Stateless and with no interaction with the outside world. Then it's trivial to test just like and other function that is simple inputs translated outputs (ie pure). Every time you see the need for a mock, you're first thought should be "how can I take the 90% or 95% of this function that is pure and pull it out, and separate the impure portion (side effects and/or stateful) that now has almost no logic or complexity left in it and push it to the boundary of my codebase?" Then the complex pure part you test the heck out of, and the stateful/side effectful impure part becomes barely a wrapper over system APIs.
- retrodaredevil 11mo ago
  I agree with you, however convincing an entire team of devs to explicitly separate the interface of impure parts of code is very difficult. If you introduce a mocking library to the test portion of the codebase, most developers will start to use it as a way to shortcut any refactoring they don't want to do. I think articles like this that try to explain how to better use mocks in tests are useful, although I wish they weren't necessary.
- majormajor 11mo ago
  > Then the complex pure part you test the heck out of, and the stateful/side effectful impure part becomes barely a wrapper over system APIs. In practice the issues I see with this are that the "side effect" part is usually either: extensive enough to still justify mocking around testing it, and also intertwined enough with your logic to be hard to remove all the "pure" logic. I rarely see 90-95% of functions being pure logic vs side effects. E.g. for the first, you could have an action that requires several sequenced side effects and then your "wrapper over APIs" still needs validation of calling the right APIs in the right oder with the right params, for various scenarios. Enter mocks or fakes. (And sometimes people will get clever and say use pubsub or events for this, but... you're usually just making the full-system-level testing there harder, as well as introducing less determinism around your consistency.) For the second, something like "do steps I and J. If the API you call in step J fails, unwind the change in I." Now you've got some logic back in there. And it's not uncommon for the branching to get more complex. Were you building everything in the system from first principles, you could try to architect something where I and J can be combined or consolidated in a way to work around this; when I and J are third party dependencies, that gets harder.
joshribakoff 11mo ago
If you’re doing TDD, you could just view this as moving the “open” call to your unit test. As others point out, that encourages pure functions that can pipe in input from other sources than just file paths.
_ZeD_ 11mo ago
Honestly I don't buy it. Worse, this is one of the reason I prefer to do "minimal integration tests" instead of unit tests. Take the example snippet of code def get_user_settings() -> str: with open(Path("~/settings.json").expanduser()) as f: return json.load(f) def add_two_settings() -> int: settings = get_user_settings() return settings["opt1"] + settings["opt2"] and the very first comment just below >>> The thing we want to avoid is opening a real file and then the article goes and goes around patching stdlib stuff etc. But instead I would suggest the real way to test it is to actually create the damn file, fill it with the "normal" (fixed) content and then run the damn test. This is because after years of battling against mocks of various sort I find that creating the "real" resource is actually less finicky than monkeypatching stuff around. Apart from that; yeah, sure the code should be refactored and the paths / resources moved out of the "pure logical" steps, but 1) this is an example and 2) this is the reality of most of the actual code, just 10x more complex and 100x more costly to refactor.
- riffraff 11mo ago
  That works fine for files, but what if the integration is with a third party service for example? You can create an actual mock networked service but it's much more work. I think this is an example explaining what seems like a good practice for using mocks in python to me, the actual code in the post is barely "supporting cast".
  - efortis 11mo ago
    If it's HTTP you can create the fixtures and serve them with a mock server. I'm a frontend dev, so backend APIs are like 3rd parties to me. I use a browser extension for scraping actual backend responses, which downloads them with a filename convention the mock server understands. I mostly use it for development, but also for setting up screenshot tests. For example, PATCH /select 'api/user(locked-out).GET.423.json' screenshot the app and pixel diff it PATCH /select 'api/user.GET.200.json' screenshot…
    - leosanchez 11mo ago
z3t4 11mo ago
If you make the function pure it will be easier to test. Pass the moving parts as function parameters, then you can pass in the mocks in the actual functions when testing. Example: f = () => a+b refactor for easier testing f = (a, b) => a+b in your test you can now mock a and b
globular-toast 11mo ago
The main reason why your mock breaks later is because you refactored the code. You did the one thing tests are supposed to help you do, and the tests broke. If code was never modified, you wouldn't need automated tests. You'd just test it manually one time and never touch it again. The whole point of tests is you probably will rewrite internals later as you add new features, improve performance or just figure out better ways to write things. Mock-heavy tests are completely pointless in this respect. You end up rewriting the code and the test every time you touch it. There are really only a few reasons to use mocks at all. Like avoiding network services, nondeterminism, or performance reasons. If you need to do a lot of mocking in your tests this is a red flag and a sign that you could write your code differently. In this case you could just make the config file location an optional argument and set up one in a temp location in the tests. No mocking required and you're testing the real API of the config file module.
dhuan_ 11mo ago
Great article. In addition, updating your mocking code can often be time-consuming. To try to make this easier, I built mock[1], which streamlines the process of setting up mock services for testing. https://dhuan.github.io/mock/latest/examples.html https://dhuan.github.io/mock/latest/examples.html
DarkNova6 11mo ago
> An overly aggressive mock can work fine, but then break much later. Why? Because you are testing against implementation, not specification. You’re welcome.

67 points by ingve 11mo ago

Arguably this is a problem in when the patch is unapplied. Presumably in the coverage case it’s being called by a trace function, which inevitably runs during test execution — and while we want the trace function to be called during the test function, we really want it without any patches the test function is using. But this arguably requires both an ability for the trace function to opt-out of patches and for the patcher to provide a way to temporarily disable all of them.

esafak 11mo ago

It is worth pointing out that you can often use containerized services as an alternative to mocking.

Izkata 11mo ago

> In Why your mock doesn’t work I explained this rule of mocking: > Mock where the object is used, not where it’s defined. For anyone looking for generic advice, this is a quirk of python due to how imports work in that language (details in the linked post) and shouldn't be considered universal.

kaashif 11mo ago

Why even mock anything in this example? You need to read the source code to work out what to mock, reaching deep inside the code to name some method to mock. But what if you just passed in the contents of the file or something? Edit: oh wait actually this is what the very last line in the blog post says! But I think it should be emphasized more!

lelandbatey 11mo ago

I feel like the #1 reason mocks break looks nothing like this and instead looks like: you change the internal behaviors of a function/method and now the mocks interact differently with the underlying code, forcing you to change the mocks. Which highlights how awful mocking as a concept is; it is of truly limited usefulness for anything but the most brittle of tests. Don't test the wrong things; if you care about some precondition, that should be an input. If you need to measure a side effect, that should be an output. Don't tweak global state to do your testing.

bluGill 11mo ago
Most of the real world is about manipulating the real world. For algorithms it is fine to say depend on the pure inputs/outputs. However what we care about is that global state is manipulated correctly and so the integration tests that verify that are what are important. In most cases your algorithm shouldn't be unit tested separately since it is only used in one place and changes when the users change: there is no point in extra tests. If the algorithm is used in many places comprehensive unit tests are important, but they get in the way when the algorithm is used only once and so the tests just inhibit changes to the algorithm as requirements change (you have to change the user, the integration tests, and the unit tests that are redundant). As such I disagree. Global state is what you should be testing - but you need to be smart about it. How you setup and verify global state matters. Don't confuse global state above with global state of variables, I mean the external state of the program before and after, which means network, file, time, and other IO things.
- refactor_master 11mo ago
  IO and global state is also just inputs that can be part of arrange-act-assert. Instead of mocking your database call to always return "foo" when the word "SELECT" is in the query, insert a real "foo" in a real test database and perform a real query. Again I've heard "but what if my database/table changes so rapidly that I need the mock so I don't need to change the query all the time", in which case you ought to take a moment to write down what you're trying to accomplish, rather than using mocks to pave over poor architectural decisions. Eventually, the query fails and the mock succeeds, because they were completely unrelated. So far I've only seen mocks fail eventually and mysteriously. With setups and DI you can treat things mostly as a black box from a testing point of view, but when mocks are involved you need surgical precision to hit the right target at the right time.
  - deleted 11mo ago
    [deleted]
  - bluGill 11mo ago
    I don't understand the "but what if my database/table changes so rapidly that I need the mock so I don't need to change the query all the time". Are they changing your database schema at runtime (not just software update time when you can do migrations)? Otherwise you should have a test helper that creates your entire database schema so that when it changes you can update the schema in one place and all tests are re-run against the new one. (you should also have some insert common test data helpers that are easy to update as needed should the new schema affect the changes) I haven't seen mocks fail mysteriously. I've seen them fail often though because requirements change and instead of updating the callers (generally a small number) you end up with 200 tests failing and give up because updating all the tests is too hard. Mocks are always about implementation details - sometimes you have no choice, but the more you can test actual behavior the better.

dherls 11mo ago

This blog post talks as if mocking the `open` function is a good thing that people should be told how to do. If you are mocking anything in the standard library your code is probably structured poorly. In the example the author walks through, a cleaner way would be to have the second function take the Options as a parameter and decouple those two functions. You can then test both in isolation.

bluGill 11mo ago
Details matters, but good test doubles here are important. You want to capture all calls to IO and do something different. You don't want tests to break because someone has a different filesystem, didn't set their home directory as you want it setup, or worse is trying to run two different tests at the same time and the other test is changing files the other wants. Note that I said test doubles. Mocks are a bit over specific - they are about verifying functions are called at the right time with the right arguments, but the easy ability to set return values makes it easy to abuse them for other things (this abuse is good, but it is still abuse of the intent). In this case you want a fake: a smart service that when you are in a test setups a temporary directory tree that contains all the files you need in the state that particular test needs, and destroys that when the test is done (with an optional mode to keep it - useful if a test fails to see debug). Depending on your situation you may need something for network services, time, or other such things. Note that in most cases a filesystem itself is more than fast enough to use in tests, but you need isolation from other tests. There are a number of ways to create this fake, it could override open, or it could just be a GetMyProgramDir function that you override are two that I can think of.
- jpollock 11mo ago
  Your tests are either hermetic, or they're flaky. That means the test environment needs to be defined and versioned with the code.
- dherls 11mo ago
  Even in the case you mention you really shouldn't be overriding these methods. Your load settings method should take the path of the settings file as an argument, and then your test can set up all the fake files you want with something like python's tempfile package

BoiledCabbage 11mo ago

The mock discussion still misses the real solution, which is to refactor the code so that you have a function that simple reads the file and returns json that is essentially a wrapper around open and doesn't need to be tested. Then have your main function take in that json as a parameter (or class wrapping that json). Then your code becomes the ideal code. Stateless and with no interaction with the outside world. Then it's trivial to test just like and other function that is simple inputs translated outputs (ie pure). Every time you see the need for a mock, you're first thought should be "how can I take the 90% or 95% of this function that is pure and pull it out, and separate the impure portion (side effects and/or stateful) that now has almost no logic or complexity left in it and push it to the boundary of my codebase?" Then the complex pure part you test the heck out of, and the stateful/side effectful impure part becomes barely a wrapper over system APIs.

retrodaredevil 11mo ago
I agree with you, however convincing an entire team of devs to explicitly separate the interface of impure parts of code is very difficult. If you introduce a mocking library to the test portion of the codebase, most developers will start to use it as a way to shortcut any refactoring they don't want to do. I think articles like this that try to explain how to better use mocks in tests are useful, although I wish they weren't necessary.
majormajor 11mo ago
> Then the complex pure part you test the heck out of, and the stateful/side effectful impure part becomes barely a wrapper over system APIs. In practice the issues I see with this are that the "side effect" part is usually either: extensive enough to still justify mocking around testing it, and also intertwined enough with your logic to be hard to remove all the "pure" logic. I rarely see 90-95% of functions being pure logic vs side effects. E.g. for the first, you could have an action that requires several sequenced side effects and then your "wrapper over APIs" still needs validation of calling the right APIs in the right oder with the right params, for various scenarios. Enter mocks or fakes. (And sometimes people will get clever and say use pubsub or events for this, but... you're usually just making the full-system-level testing there harder, as well as introducing less determinism around your consistency.) For the second, something like "do steps I and J. If the API you call in step J fails, unwind the change in I." Now you've got some logic back in there. And it's not uncommon for the branching to get more complex. Were you building everything in the system from first principles, you could try to architect something where I and J can be combined or consolidated in a way to work around this; when I and J are third party dependencies, that gets harder.

joshribakoff 11mo ago

If you’re doing TDD, you could just view this as moving the “open” call to your unit test. As others point out, that encourages pure functions that can pipe in input from other sources than just file paths.

_ZeD_ 11mo ago

Honestly I don't buy it. Worse, this is one of the reason I prefer to do "minimal integration tests" instead of unit tests. Take the example snippet of code def get_user_settings() -> str: with open(Path("~/settings.json").expanduser()) as f: return json.load(f) def add_two_settings() -> int: settings = get_user_settings() return settings["opt1"] + settings["opt2"] and the very first comment just below >>> The thing we want to avoid is opening a real file and then the article goes and goes around patching stdlib stuff etc. But instead I would suggest the real way to test it is to actually create the damn file, fill it with the "normal" (fixed) content and then run the damn test. This is because after years of battling against mocks of various sort I find that creating the "real" resource is actually less finicky than monkeypatching stuff around. Apart from that; yeah, sure the code should be refactored and the paths / resources moved out of the "pure logical" steps, but 1) this is an example and 2) this is the reality of most of the actual code, just 10x more complex and 100x more costly to refactor.

riffraff 11mo ago
That works fine for files, but what if the integration is with a third party service for example? You can create an actual mock networked service but it's much more work. I think this is an example explaining what seems like a good practice for using mocks in python to me, the actual code in the post is barely "supporting cast".
- efortis 11mo ago
  If it's HTTP you can create the fixtures and serve them with a mock server. I'm a frontend dev, so backend APIs are like 3rd parties to me. I use a browser extension for scraping actual backend responses, which downloads them with a filename convention the mock server understands. I mostly use it for development, but also for setting up screenshot tests. For example, PATCH /select 'api/user(locked-out).GET.423.json' screenshot the app and pixel diff it PATCH /select 'api/user.GET.200.json' screenshot…
  - leosanchez 11mo ago

z3t4 11mo ago

If you make the function pure it will be easier to test. Pass the moving parts as function parameters, then you can pass in the mocks in the actual functions when testing. Example: f = () => a+b refactor for easier testing f = (a, b) => a+b in your test you can now mock a and b

globular-toast 11mo ago

The main reason why your mock breaks later is because you refactored the code. You did the one thing tests are supposed to help you do, and the tests broke. If code was never modified, you wouldn't need automated tests. You'd just test it manually one time and never touch it again. The whole point of tests is you probably will rewrite internals later as you add new features, improve performance or just figure out better ways to write things. Mock-heavy tests are completely pointless in this respect. You end up rewriting the code and the test every time you touch it. There are really only a few reasons to use mocks at all. Like avoiding network services, nondeterminism, or performance reasons. If you need to do a lot of mocking in your tests this is a red flag and a sign that you could write your code differently. In this case you could just make the config file location an optional argument and set up one in a temp location in the tests. No mocking required and you're testing the real API of the config file module.

dhuan_ 11mo ago

Great article. In addition, updating your mocking code can often be time-consuming. To try to make this easier, I built mock[1], which streamlines the process of setting up mock services for testing. https://dhuan.github.io/mock/latest/examples.html https://dhuan.github.io/mock/latest/examples.html

DarkNova6 11mo ago

> An overly aggressive mock can work fine, but then break much later. Why? Because you are testing against implementation, not specification. You’re welcome.