9 ms·

Why a mock doesn’t work

173 points by UkiahSmith 7y ago

llarsson 7y ago
For a person who does not code very large programs in Python, this looks scary and like something that is hardly "only one way to do it" and that the most elegant solution gives you what you want. How I import things (or the libraries I depend on!) affects how I can write my tests? Really? My own Python scripts are typically single-screen in length. And one-off stuff, where they either work or don't, basically. Is this stuff really what developers of large-scale Python programs have to take into account? Or is this blog post misinformed, because there is an obviously better and standard way?
- zzzeek 7y ago
  no, this blog post is absolutely on point and correct. it can be extremely inconvenient and complicated to get around using mocks in many situations where you want to test things, so we want to use mocks when appropriate. Then, you definitely want to mock at the most specific level possible. while mocking in a way that is specific to how modules are imported is technically "fragile", in that it is deeply dependent on the structure code that's being tested, this is not an issue in practice, because the mocks are present in our test suites that run for every code change. If a code change moves around module assignments, our test will fail, and we know that we have to adjust the test to accommodate for the change. With appropriate continuous integration and code review practices, a broken mock-oriented test can't be inadvertently pushed into a repository. This does mean that when using mocks, you need to make sure mocks were called in the way that was expected for those cases where the code might silently move off using the mock in a way that wouldn't be detectable. The "os.listdir()" example in this blog is a pretty common case, using mocks to test code that works with filesystems, where you don't need or want to get involved with actually creating filesystems which may be a complex and expensive process, especially if the test suite runs concurrent processes. if you mock the behavior of "os.listdir" to return a series of results, after the test code has been exercised, you usually want to assert using mock.mock_calls that the test code did in fact call the functions expected, unless it's clear in some other way that the test code definitely used that information. An example of this kind of code that I just helped someone with can be seen here: https://github.com/sqlalchemy/alembic/commit/02a1bf3454acb7b02942e246c19326630a8f9175#diff-c1de9c971ea0a629f46ecaf7dccad37dR817 https://github.com/sqlalchemy/alembic/commit/02a1bf3454acb7b... The Alembic test suite has a lot of test cases that go through all the trouble to build up real directory structures to test things, but that's a lot more work than just using a mock, so I use them where I can get away with this simpler approach.
  - EdSchouten 7y ago
    > The "os.listdir()" example in this blog is a pretty common case, using mocks to test code that works with filesystems, where you don't need or want to get involved with actually creating filesystems which may be a complex and expensive process, especially if the test suite runs concurrent processes. Alternative: have an abstract base class that describes a file system API. Have one implementation that builds on top of OS primitives and have another one that is a mock (potentially auto-generated through some mocking framework). That way there is no need to monkey-patch standard library functions at runtime. I did that within a project of mine, written in Go: https://github.com/buildbarn/bb-storage/blob/master/pkg/filesystem/directory.go https://github.com/buildbarn/bb-storage/blob/master/pkg/file... https://github.com/buildbarn/bb-storage/blob/master/pkg/filesystem/local_directory.go https://github.com/buildbarn/bb-storage/blob/master/pkg/file... Pretty slick that I can now use https://github.com/golang/mock https://github.com/golang/mock to automatically generate a mock of that.
    - zzzeek 7y ago
      Sure but then I have to write my real library code using an abstraction, making my code more difficult to read and maintain; a dependency injection system is then necessary in order to have the correct concrete implementations set up at runtime. In this sense, mocks are solving the problem of having code that is full of dependency-injected AbstractFooBarFileSystemWithExtraPickles style of code, which is considered to be pretty un-Pythonic. I spent many years with Java and Spring so I can attest to both sides of this equation. Those of us using Python are using it because it is an interpreted, dynamic scripting language. If I'm coding in something more rigid like C or Go, then I'd expect to have a more complex architecture in order to achieve things that are fairly simple in a scripting language.
      - EdSchouten 7y ago
        > Sure but then I have to write my real library code using an abstraction, making my code more difficult to read and maintain; It depends, right? If you suddenly wanted to let your existing set of classes read their inputs not from disk, but from some other kind of storage (e.g., files embedded in a Zip file), you'd only need to write one extra class and you're good to go. That would be a lot harder if your code called os.* directly. > a dependency injection system is then necessary in order to have the correct concrete implementations set up at runtime. If by dependency injection system you mean invoking one extra constructor in, say, main() and pass the object along as a handle, sure.
- wdroz 7y ago
  In the alternative approaches cited by the author, the article "Itamar Turner-Trauring’s article" [0] presents an example that is more representative (IMO) on what people do for "mocking". [0] - https://pythonspeed.com/articles/verified-fakes/ https://pythonspeed.com/articles/verified-fakes/
- coldtea 7y ago
  >My own Python scripts are typically single-screen in length. And one-off stuff, where they either work or don't, basically. (...) Is this stuff really what developers of large-scale Python programs have to take into account? Not the only way to go about it, but yes, the underlying concerns are absolutely stuff that developers of large-scale Python programs have to take into account. At "single screen" length you can do anything you want. Really, anything goes -- and you could ever just stare at the code a lot to find all or almost all bugs. At 1000s of lines or more (can get into 100s or 1M lines), and especially on growing and worked on code (that gets new features, refactors, etc) you need to follow other practices to ensure it all works.
- scbrg 7y ago
  > How I import things (or the libraries I depend on!) affects how I can write my tests? Really? Well, no. Everything that needs to be understood is explained in the first section, A quick aside about assignment. Everything else is just discussion about logical consequences of this behavior. If you do an assignment (and "from foo import bar" is an assignment) you're just creating another name for a value (in this case, you're creating a name foo, in your namespace (presumably, but not necessarily a module) that points at whatever the name foo in module bar happened to point at at the moment you created it). If the original (or should I say, one of the previous) name(s) to the value is reassigned that does not affect what your name is pointing at. This is really all there is to know, and once you've grokked that, everything Python falls in place.
baxtr 7y ago
Even though it’s not related, I wonder if this does apply to UI as well. I often want to see production near prototypes
stephen 7y ago
Agreed with the articles, mocks are very tied to implementation details. I almost always prefer state-based testing: https://martinfowler.com/articles/mocksArentStubs.html https://martinfowler.com/articles/mocksArentStubs.html As "observable state goes from a to b" is much closer to a business/functional requirement that will still/always be true, regardless of refactorings. Refactoring in codebases with state-based tests is a pleasure; in codebases with mock-based tests it's tedious, constantly updating tests when no semantic behavior was supposed to change. Also, mocking via module hacks like in the article (and in the JS world) is scary; modules are basically global variables so it's a very coarse grained slice point. Dependency injection is almost always better.
- squirrelicus 7y ago
  Even better than "observable state from a to b" is "a transform of value from a to b", no inspection and whitebox needed :D
  - switchbak 7y ago
    Absolutely. The more functional we can make the code, typically the easier it is to test it based only on inputs and outputs. Much of what I do is try to come up with architectures that model the logic in this way, and are still legible to my co workers.
- hinkley 7y ago
  Dependency injection leads to elaborate, brittle fixtures that become more elaborate and arcane over time. And the sunk cost fallacy has people working for hours (and I’ve witnessed pairs spend two days, that’s over three man days!) trying to maintain them. I use mocks to arrange state A without going through all the business rules involved in creating state A. But you have to expose the states to do it, which can make for pretty descriptive code but is outside of some people’s experience and so they resist it. Earlier tests have already verified all routes to State A, the next batch of test now takes it as a given. This controls test explosion by utilizing transitivity. A->B B->C implies A->C. Tons of unit tests for A->B and B->C and then you only need a couple of functional tests (say, one negative and positive) test of A->C. You’re just checking the plumbing isn’t broken. Otherwise, you get Cartesian products. You end up with elaborate, (often custom), mocks that couple all of the tests together in hard to maintain ways. You end up with tests that accidentally test the mocks/fixtures instead of the code. Some of my current coworkers do this too. I don’t know where this pattern comes from. It’s often easier to replace their two page fixtures with two or three lines of mocks per test. It’s often almost the same amount of code, but so what, each test can be read. Each test can be run individually, and behavioral changes to the code due to urges affect mostly the tests you would expect. Those tests can be fixed, rewritten, or just removed as they disagree with the new requirements. The absolute worst is fixtures with asynchronous code. Those break constantly, and often invisibly. What’s the point of a fire alarm if the damned thing doesn’t work? It’s almost worse than nothing at all.
mpweiher 7y ago
Why I don't mock: https://blog.metaobject.com/2014/05/why-i-don-mock.html https://blog.metaobject.com/2014/05/why-i-don-mock.html
- frenchman99 7y ago
  You're talking about mocking database calls though. In my line of work (insurance brokerage), we use lots of insurance APIs and they are sometimes very slow (+20 seconds / call) or completely down at random hours. There is simply no way around mocking those API calls if you want a fast and reliable testsuite.
  - j88439h84 7y ago
    Take a look at the links at the top of the post. They discuss a great alternative to mocking slow services: fakes.
  - redis_mlc 7y ago
    > or completely down at random hours Also describes "dinosaur payment company" API sandboxes (stage). So we end up testing against the production API with a staff member's credit card - well if we want to deploy any time soon. Or I guess you could mock and cross your fingers that they haven't changed the API recently without telling you. Payment APIs are the most solid, but that's a low bar considering the state of third-party APIs in the real world. If it wasn't hard for enterprises to build and manage APIs, then Google Apigee and Mulesoft wouldn't be worth billions.
  - scarface74 7y ago
    Yes we can mock databases and queries in C# var oldMen = from p in context.Persons where p.Sex == “M” && p.Age > 65; “context” could be an Entity Framework context that would generate sql at runtime, an IMongoQueryable that would generate MongoQuery at runtime or an in memory List<Person> that would generate the equivalent foreach loop with an if condition (overly simplified) while running your tests.
michaelmcmillan 7y ago
If you find yourself fighting with mocks, ask yourself: is there a deeper design problem with my code? I often find that things I can't test easily have crappy design.
- theptip 7y ago
  Couldn't agree more. I've done a few "coding dojo" sessions where my team and I start from scratch and write a new set of mock-based tests for a piece of existing code, and when it starts to get gnarly, it's always been because of an inconsistent interface, or confusing API of the code under test. Bob Martin talks about this a lot; your UTs should be thought of as first-class clients of your objects' APIs. If something is hard to test, it's probably hard to use, or abstracted at the wrong level.
c- 7y ago
"Why a mock doesn't work" *in python because of reasons fixed the title.
askvictor 7y ago
I've always been dubious of using mock for tests which involve external API calls; at best they require you to reimplement the API according to the documentation (which there may be none, or that the API doesn't follow exactly for edge cases (i.e. the things you're meant to be testing)). At worst you're implementing a very small subset of the behaviour of the API and not testing how your code responds to the other behaviour. But I haven't come across other solutions (not that I have that a heap of experience here, just contributions to a couple of oss projects).
- alexanderdmitri 7y ago
  Mocking inputs from APIs is actually great if someone besides the original developer picks up the codebase, whether from a system or even down to isolated function stubs. This is because it gives insight into the expected inputs the original developer(s) had been expecting and can elucidate the source under test. Integration tests fill the hole you're pointing out. You can even have integration tests designed to validate the mock inputs in many cases.
raymondh 7y ago
Python core developer Lisa Roach has a nice how-to video on exactly this subject: https://www.youtube.com/watch?v=ww1UsGZV8fQ https://www.youtube.com/watch?v=ww1UsGZV8fQ
raymondh 7y ago
I think Ned strikes the right balance, showing risks associated with mock objects without condemning them outright. There is no doubt that people sometimes go overboard with mocking and there is no doubt that there are situations where it is really helpful.
jabwork 7y ago
I've wrestled with this problem several times. My conclusion was mocks are just fine, but this is a wart in that there isn't "one way to do it" For the most part I can get by with one rule: always mock the module. with mock.patch('os.listdir'): will always work, even if it doesn't accomplish what you want. with mock.patch('mymodule.os.listdir') will fail if that module does not explicitly import os and instead does something like from os import listdir (perhaps because a later dev did not realize importing os directly was actually a requirement for the test and changed the code). The rule is not perfect though. In the above case, the error will actually be ImportError: No module named os This can be fixed with e.g., assert hasattr(module, 'os'), "os module is not explicitly imported" as a preamble to your test but ... it is not perfect by any means. EDIT: formatting
- nedbat 7y ago
  I don't understand what you mean by "will always work, even if it doesn't accomplish what you want." Mocking os.listdir will be useless if your product code's imports don't match it. How is this "one rule" to use?
  - jabwork 7y ago
    The os module ships with python, so the function call mocking os.listdir will always succeed even if the code being tested does not use os.listdir By mocking mymodule.os.listdir you add a requirement that mymodule actually import the os module and take advantage of mock.patch failing loudly if it does not.
pojzon 7y ago
I really don't like that the topic title is a general statement but the article under is talking strictly in python landscape as an example and not discussing the idea itself. Mocking by itself works fine. It's a good idea that works if used correctly. Misusing it or bad usage leads to issues - duh. I'm familiar with the narrative that anything that is too hard (to get right at the first try) in our field means that it is bad - but I don't agree with it at all. We are at the point where craftsmanship should be a good metric to distinguish medicore developers from experts..
- nedbat 7y ago
  What title would you prefer? "Your Python mock might not work, but it could still be a good idea if you do it right, and here I will explain how"? :) I didn't mean to imply that mocking is bad. Is that what you took from it? Why would I explain how to get mocks to work if I thought people shouldn't use mocks?
  - pojzon 7y ago
    A better question would be: "Why would I use a generic title, if I dont agree with it in my own article?" And the answer would be: " A click-bite title"..
    - nedbat 7y ago
      I'm not sure what your objection is. People try to use mocks, and they don't work because they've mocked the wrong name. I explain this in the article. The article is about why their mock didn't work. How is this a misleading title? How is this a title I don't agree with? Are we talking about the same piece and the same title? You seem to be under the impression that I am trying to tell people not to use mocks. Have you read the piece?
      - nedbat 7y ago
        I notice that the title on HN is "Why a mock doesn't work," which could be interpreted as "Mocks don't work." My title was (and still is) "Why your mock doesn't work." I don't know if that is the source of the confusion.
jchook 7y ago
“...there are other approaches to solving the problems of isolating your product code from problematic dependencies.” What are some other approaches? I have followed the redux-saga pattern with success but how else should we accomplish the same goal?
- j88439h84 7y ago
  See the links at the top of the post. Both are excellent.
gitgud 7y ago
Mocking is hard really hard, in order to mock something you need to imitate its functionality and interface. This means the mock is inherently tightly-coupled to the implementation, which is now another dependency in your system. After trying to work with mock databases and file systems, I've personally found that there's no substitute for the real thing. There's much less maintenance and greater reliability in spinning up a test environment with the exact implementation that will be used in the production environment. There are cases where mocks are the only practical solution, (embedded systems, distributed systems) but mocking is surely the last resort...
blondin 7y ago
the python mock module is one of those modules i would like to see rewritten from scratch. you won't get it right by first principles. you always need to go to the documentation. that's a sign that something is not right imo.
franzwong 7y ago
Is this python specific? Usually I replace the whole module with mock instead of the functions of it.
pramttl 7y ago
Beautiful explanation for something that tripped me up in early days of using mock/patch. Summary: (1) Variables in Python are names that refer to values. (2) (For that reason) Mock an object where it is used, not where it is defined.
alanfranz 7y ago
The author hasn't a problem with mocks. It has got a problem with monkey patching. You can use a mock along dependency injection, and never run into the problems the author has.
wojciech_bulaty 7y ago
Ned's implicit definition of a mock is narrower than the generally accepted one. He actually described a stub created by monkey-patching. A mock allows for call verifications as well. There are 3 main categories of techniques for managing dependent components used these days: 1. In-process class/method/function mocks or stubs (http://xunitpatterns.com/Mocks,%20Fakes,%20Stubs%20and%20Dummies.html http://xunitpatterns.com/Mocks,%20Fakes,%20Stubs%20and%20Dum... and https://martinfowler.com/articles/mocksArentStubs.html https://martinfowler.com/articles/mocksArentStubs.html) 1a. By monkey patching (which is what Ned has demonstrated in his article very well) 2a. By dependency injection 2. Over-the-wire API mocks or stubs (https://en.wikipedia.org/wiki/Comparison_of_API_simulation_tools https://en.wikipedia.org/wiki/Comparison_of_API_simulation_t...) 3. Virtual services/simulators (https://en.wikipedia.org/wiki/Comparison_of_API_simulation_tools https://en.wikipedia.org/wiki/Comparison_of_API_simulation_t...) It's worth keeping in mind that all of them are part of a wider group of test doubles: https://www.infoq.com/articles/stubbing-mocking-service-virtualization-differences/ https://www.infoq.com/articles/stubbing-mocking-service-virt... Other options available for decoupling from test dependencies: 1. In-memory database https://en.wikipedia.org/wiki/In-memory_database https://en.wikipedia.org/wiki/In-memory_database 2. Test container https://www.testcontainers.org/ https://www.testcontainers.org/ 3. Legacy in a box https://www.thoughtworks.com/radar/techniques/legacy-in-a-box https://www.thoughtworks.com/radar/techniques/legacy-in-a-bo...

173 points by UkiahSmith 7y ago

For a person who does not code very large programs in Python, this looks scary and like something that is hardly "only one way to do it" and that the most elegant solution gives you what you want. How I import things (or the libraries I depend on!) affects how I can write my tests? Really? My own Python scripts are typically single-screen in length. And one-off stuff, where they either work or don't, basically. Is this stuff really what developers of large-scale Python programs have to take into account? Or is this blog post misinformed, because there is an obviously better and standard way?

zzzeek 7y ago
no, this blog post is absolutely on point and correct. it can be extremely inconvenient and complicated to get around using mocks in many situations where you want to test things, so we want to use mocks when appropriate. Then, you definitely want to mock at the most specific level possible. while mocking in a way that is specific to how modules are imported is technically "fragile", in that it is deeply dependent on the structure code that's being tested, this is not an issue in practice, because the mocks are present in our test suites that run for every code change. If a code change moves around module assignments, our test will fail, and we know that we have to adjust the test to accommodate for the change. With appropriate continuous integration and code review practices, a broken mock-oriented test can't be inadvertently pushed into a repository. This does mean that when using mocks, you need to make sure mocks were called in the way that was expected for those cases where the code might silently move off using the mock in a way that wouldn't be detectable. The "os.listdir()" example in this blog is a pretty common case, using mocks to test code that works with filesystems, where you don't need or want to get involved with actually creating filesystems which may be a complex and expensive process, especially if the test suite runs concurrent processes. if you mock the behavior of "os.listdir" to return a series of results, after the test code has been exercised, you usually want to assert using mock.mock_calls that the test code did in fact call the functions expected, unless it's clear in some other way that the test code definitely used that information. An example of this kind of code that I just helped someone with can be seen here: https://github.com/sqlalchemy/alembic/commit/02a1bf3454acb7b02942e246c19326630a8f9175#diff-c1de9c971ea0a629f46ecaf7dccad37dR817 https://github.com/sqlalchemy/alembic/commit/02a1bf3454acb7b... The Alembic test suite has a lot of test cases that go through all the trouble to build up real directory structures to test things, but that's a lot more work than just using a mock, so I use them where I can get away with this simpler approach.
- EdSchouten 7y ago
  > The "os.listdir()" example in this blog is a pretty common case, using mocks to test code that works with filesystems, where you don't need or want to get involved with actually creating filesystems which may be a complex and expensive process, especially if the test suite runs concurrent processes. Alternative: have an abstract base class that describes a file system API. Have one implementation that builds on top of OS primitives and have another one that is a mock (potentially auto-generated through some mocking framework). That way there is no need to monkey-patch standard library functions at runtime. I did that within a project of mine, written in Go: https://github.com/buildbarn/bb-storage/blob/master/pkg/filesystem/directory.go https://github.com/buildbarn/bb-storage/blob/master/pkg/file... https://github.com/buildbarn/bb-storage/blob/master/pkg/filesystem/local_directory.go https://github.com/buildbarn/bb-storage/blob/master/pkg/file... Pretty slick that I can now use https://github.com/golang/mock https://github.com/golang/mock to automatically generate a mock of that.
  - zzzeek 7y ago
    Sure but then I have to write my real library code using an abstraction, making my code more difficult to read and maintain; a dependency injection system is then necessary in order to have the correct concrete implementations set up at runtime. In this sense, mocks are solving the problem of having code that is full of dependency-injected AbstractFooBarFileSystemWithExtraPickles style of code, which is considered to be pretty un-Pythonic. I spent many years with Java and Spring so I can attest to both sides of this equation. Those of us using Python are using it because it is an interpreted, dynamic scripting language. If I'm coding in something more rigid like C or Go, then I'd expect to have a more complex architecture in order to achieve things that are fairly simple in a scripting language.
    - EdSchouten 7y ago
      > Sure but then I have to write my real library code using an abstraction, making my code more difficult to read and maintain; It depends, right? If you suddenly wanted to let your existing set of classes read their inputs not from disk, but from some other kind of storage (e.g., files embedded in a Zip file), you'd only need to write one extra class and you're good to go. That would be a lot harder if your code called os.* directly. > a dependency injection system is then necessary in order to have the correct concrete implementations set up at runtime. If by dependency injection system you mean invoking one extra constructor in, say, main() and pass the object along as a handle, sure.
wdroz 7y ago
In the alternative approaches cited by the author, the article "Itamar Turner-Trauring’s article" [0] presents an example that is more representative (IMO) on what people do for "mocking". [0] - https://pythonspeed.com/articles/verified-fakes/ https://pythonspeed.com/articles/verified-fakes/
coldtea 7y ago
>My own Python scripts are typically single-screen in length. And one-off stuff, where they either work or don't, basically. (...) Is this stuff really what developers of large-scale Python programs have to take into account? Not the only way to go about it, but yes, the underlying concerns are absolutely stuff that developers of large-scale Python programs have to take into account. At "single screen" length you can do anything you want. Really, anything goes -- and you could ever just stare at the code a lot to find all or almost all bugs. At 1000s of lines or more (can get into 100s or 1M lines), and especially on growing and worked on code (that gets new features, refactors, etc) you need to follow other practices to ensure it all works.
scbrg 7y ago
> How I import things (or the libraries I depend on!) affects how I can write my tests? Really? Well, no. Everything that needs to be understood is explained in the first section, A quick aside about assignment. Everything else is just discussion about logical consequences of this behavior. If you do an assignment (and "from foo import bar" is an assignment) you're just creating another name for a value (in this case, you're creating a name foo, in your namespace (presumably, but not necessarily a module) that points at whatever the name foo in module bar happened to point at at the moment you created it). If the original (or should I say, one of the previous) name(s) to the value is reassigned that does not affect what your name is pointing at. This is really all there is to know, and once you've grokked that, everything Python falls in place.

baxtr 7y ago

Even though it’s not related, I wonder if this does apply to UI as well. I often want to see production near prototypes

stephen 7y ago

Agreed with the articles, mocks are very tied to implementation details. I almost always prefer state-based testing: https://martinfowler.com/articles/mocksArentStubs.html https://martinfowler.com/articles/mocksArentStubs.html As "observable state goes from a to b" is much closer to a business/functional requirement that will still/always be true, regardless of refactorings. Refactoring in codebases with state-based tests is a pleasure; in codebases with mock-based tests it's tedious, constantly updating tests when no semantic behavior was supposed to change. Also, mocking via module hacks like in the article (and in the JS world) is scary; modules are basically global variables so it's a very coarse grained slice point. Dependency injection is almost always better.

squirrelicus 7y ago
Even better than "observable state from a to b" is "a transform of value from a to b", no inspection and whitebox needed :D
- switchbak 7y ago
  Absolutely. The more functional we can make the code, typically the easier it is to test it based only on inputs and outputs. Much of what I do is try to come up with architectures that model the logic in this way, and are still legible to my co workers.
hinkley 7y ago
Dependency injection leads to elaborate, brittle fixtures that become more elaborate and arcane over time. And the sunk cost fallacy has people working for hours (and I’ve witnessed pairs spend two days, that’s over three man days!) trying to maintain them. I use mocks to arrange state A without going through all the business rules involved in creating state A. But you have to expose the states to do it, which can make for pretty descriptive code but is outside of some people’s experience and so they resist it. Earlier tests have already verified all routes to State A, the next batch of test now takes it as a given. This controls test explosion by utilizing transitivity. A->B B->C implies A->C. Tons of unit tests for A->B and B->C and then you only need a couple of functional tests (say, one negative and positive) test of A->C. You’re just checking the plumbing isn’t broken. Otherwise, you get Cartesian products. You end up with elaborate, (often custom), mocks that couple all of the tests together in hard to maintain ways. You end up with tests that accidentally test the mocks/fixtures instead of the code. Some of my current coworkers do this too. I don’t know where this pattern comes from. It’s often easier to replace their two page fixtures with two or three lines of mocks per test. It’s often almost the same amount of code, but so what, each test can be read. Each test can be run individually, and behavioral changes to the code due to urges affect mostly the tests you would expect. Those tests can be fixed, rewritten, or just removed as they disagree with the new requirements. The absolute worst is fixtures with asynchronous code. Those break constantly, and often invisibly. What’s the point of a fire alarm if the damned thing doesn’t work? It’s almost worse than nothing at all.

mpweiher 7y ago

Why I don't mock: https://blog.metaobject.com/2014/05/why-i-don-mock.html https://blog.metaobject.com/2014/05/why-i-don-mock.html

frenchman99 7y ago
You're talking about mocking database calls though. In my line of work (insurance brokerage), we use lots of insurance APIs and they are sometimes very slow (+20 seconds / call) or completely down at random hours. There is simply no way around mocking those API calls if you want a fast and reliable testsuite.
- j88439h84 7y ago
  Take a look at the links at the top of the post. They discuss a great alternative to mocking slow services: fakes.
- redis_mlc 7y ago
  > or completely down at random hours Also describes "dinosaur payment company" API sandboxes (stage). So we end up testing against the production API with a staff member's credit card - well if we want to deploy any time soon. Or I guess you could mock and cross your fingers that they haven't changed the API recently without telling you. Payment APIs are the most solid, but that's a low bar considering the state of third-party APIs in the real world. If it wasn't hard for enterprises to build and manage APIs, then Google Apigee and Mulesoft wouldn't be worth billions.
- scarface74 7y ago
  Yes we can mock databases and queries in C# var oldMen = from p in context.Persons where p.Sex == “M” && p.Age > 65; “context” could be an Entity Framework context that would generate sql at runtime, an IMongoQueryable that would generate MongoQuery at runtime or an in memory List<Person> that would generate the equivalent foreach loop with an if condition (overly simplified) while running your tests.

michaelmcmillan 7y ago

If you find yourself fighting with mocks, ask yourself: is there a deeper design problem with my code? I often find that things I can't test easily have crappy design.

theptip 7y ago
Couldn't agree more. I've done a few "coding dojo" sessions where my team and I start from scratch and write a new set of mock-based tests for a piece of existing code, and when it starts to get gnarly, it's always been because of an inconsistent interface, or confusing API of the code under test. Bob Martin talks about this a lot; your UTs should be thought of as first-class clients of your objects' APIs. If something is hard to test, it's probably hard to use, or abstracted at the wrong level.

c- 7y ago

"Why a mock doesn't work" *in python because of reasons fixed the title.

askvictor 7y ago

I've always been dubious of using mock for tests which involve external API calls; at best they require you to reimplement the API according to the documentation (which there may be none, or that the API doesn't follow exactly for edge cases (i.e. the things you're meant to be testing)). At worst you're implementing a very small subset of the behaviour of the API and not testing how your code responds to the other behaviour. But I haven't come across other solutions (not that I have that a heap of experience here, just contributions to a couple of oss projects).

alexanderdmitri 7y ago
Mocking inputs from APIs is actually great if someone besides the original developer picks up the codebase, whether from a system or even down to isolated function stubs. This is because it gives insight into the expected inputs the original developer(s) had been expecting and can elucidate the source under test. Integration tests fill the hole you're pointing out. You can even have integration tests designed to validate the mock inputs in many cases.

raymondh 7y ago

Python core developer Lisa Roach has a nice how-to video on exactly this subject: https://www.youtube.com/watch?v=ww1UsGZV8fQ https://www.youtube.com/watch?v=ww1UsGZV8fQ

raymondh 7y ago

I think Ned strikes the right balance, showing risks associated with mock objects without condemning them outright. There is no doubt that people sometimes go overboard with mocking and there is no doubt that there are situations where it is really helpful.

jabwork 7y ago

I've wrestled with this problem several times. My conclusion was mocks are just fine, but this is a wart in that there isn't "one way to do it" For the most part I can get by with one rule: always mock the module. with mock.patch('os.listdir'): will always work, even if it doesn't accomplish what you want. with mock.patch('mymodule.os.listdir') will fail if that module does not explicitly import os and instead does something like from os import listdir (perhaps because a later dev did not realize importing os directly was actually a requirement for the test and changed the code). The rule is not perfect though. In the above case, the error will actually be ImportError: No module named os This can be fixed with e.g., assert hasattr(module, 'os'), "os module is not explicitly imported" as a preamble to your test but ... it is not perfect by any means. EDIT: formatting

nedbat 7y ago
I don't understand what you mean by "will always work, even if it doesn't accomplish what you want." Mocking os.listdir will be useless if your product code's imports don't match it. How is this "one rule" to use?
- jabwork 7y ago
  The os module ships with python, so the function call mocking os.listdir will always succeed even if the code being tested does not use os.listdir By mocking mymodule.os.listdir you add a requirement that mymodule actually import the os module and take advantage of mock.patch failing loudly if it does not.

pojzon 7y ago

I really don't like that the topic title is a general statement but the article under is talking strictly in python landscape as an example and not discussing the idea itself. Mocking by itself works fine. It's a good idea that works if used correctly. Misusing it or bad usage leads to issues - duh. I'm familiar with the narrative that anything that is too hard (to get right at the first try) in our field means that it is bad - but I don't agree with it at all. We are at the point where craftsmanship should be a good metric to distinguish medicore developers from experts..

nedbat 7y ago
What title would you prefer? "Your Python mock might not work, but it could still be a good idea if you do it right, and here I will explain how"? :) I didn't mean to imply that mocking is bad. Is that what you took from it? Why would I explain how to get mocks to work if I thought people shouldn't use mocks?
- pojzon 7y ago
  A better question would be: "Why would I use a generic title, if I dont agree with it in my own article?" And the answer would be: " A click-bite title"..
  - nedbat 7y ago
    I'm not sure what your objection is. People try to use mocks, and they don't work because they've mocked the wrong name. I explain this in the article. The article is about why their mock didn't work. How is this a misleading title? How is this a title I don't agree with? Are we talking about the same piece and the same title? You seem to be under the impression that I am trying to tell people not to use mocks. Have you read the piece?
    - nedbat 7y ago
      I notice that the title on HN is "Why a mock doesn't work," which could be interpreted as "Mocks don't work." My title was (and still is) "Why your mock doesn't work." I don't know if that is the source of the confusion.

jchook 7y ago

“...there are other approaches to solving the problems of isolating your product code from problematic dependencies.” What are some other approaches? I have followed the redux-saga pattern with success but how else should we accomplish the same goal?

j88439h84 7y ago
See the links at the top of the post. Both are excellent.

gitgud 7y ago

Mocking is hard really hard, in order to mock something you need to imitate its functionality and interface. This means the mock is inherently tightly-coupled to the implementation, which is now another dependency in your system. After trying to work with mock databases and file systems, I've personally found that there's no substitute for the real thing. There's much less maintenance and greater reliability in spinning up a test environment with the exact implementation that will be used in the production environment. There are cases where mocks are the only practical solution, (embedded systems, distributed systems) but mocking is surely the last resort...

blondin 7y ago

the python mock module is one of those modules i would like to see rewritten from scratch. you won't get it right by first principles. you always need to go to the documentation. that's a sign that something is not right imo.

franzwong 7y ago

Is this python specific? Usually I replace the whole module with mock instead of the functions of it.

pramttl 7y ago

Beautiful explanation for something that tripped me up in early days of using mock/patch. Summary: (1) Variables in Python are names that refer to values. (2) (For that reason) Mock an object where it is used, not where it is defined.

alanfranz 7y ago

The author hasn't a problem with mocks. It has got a problem with monkey patching. You can use a mock along dependency injection, and never run into the problems the author has.

wojciech_bulaty 7y ago

Ned's implicit definition of a mock is narrower than the generally accepted one. He actually described a stub created by monkey-patching. A mock allows for call verifications as well. There are 3 main categories of techniques for managing dependent components used these days: 1. In-process class/method/function mocks or stubs (http://xunitpatterns.com/Mocks,%20Fakes,%20Stubs%20and%20Dummies.html http://xunitpatterns.com/Mocks,%20Fakes,%20Stubs%20and%20Dum... and https://martinfowler.com/articles/mocksArentStubs.html https://martinfowler.com/articles/mocksArentStubs.html) 1a. By monkey patching (which is what Ned has demonstrated in his article very well) 2a. By dependency injection 2. Over-the-wire API mocks or stubs (https://en.wikipedia.org/wiki/Comparison_of_API_simulation_tools https://en.wikipedia.org/wiki/Comparison_of_API_simulation_t...) 3. Virtual services/simulators (https://en.wikipedia.org/wiki/Comparison_of_API_simulation_tools https://en.wikipedia.org/wiki/Comparison_of_API_simulation_t...) It's worth keeping in mind that all of them are part of a wider group of test doubles: https://www.infoq.com/articles/stubbing-mocking-service-virtualization-differences/ https://www.infoq.com/articles/stubbing-mocking-service-virt... Other options available for decoupling from test dependencies: 1. In-memory database https://en.wikipedia.org/wiki/In-memory_database https://en.wikipedia.org/wiki/In-memory_database 2. Test container https://www.testcontainers.org/ https://www.testcontainers.org/ 3. Legacy in a box https://www.thoughtworks.com/radar/techniques/legacy-in-a-box https://www.thoughtworks.com/radar/techniques/legacy-in-a-bo...