10 ms·
Don’t mock what you don’t own
- rand_r 4y agoWe started employing this pattern in our code at work recently. Using the Facade pattern to create a simpler interface around a couple of things: - Feature Flag service, wrapping LaunchDarkly - Remote Storage service, wrapping S3 The application code uses the simplified services, with little knowledge of the underlying providers (e.g. S3 or LaunchDarkly). The tests use a test implementation of the simplified services. This pattern simplifies both the application code and test code and is an all-around win. It's been a game changer personally, and made me excited about software architecture and coding.
- throw827474737 4y ago> For example if an object already does have an idiomatic API, it’s probably not worth wrapping in an identical façade, just so it belongs to you. Ooooh my gwahh... your third party library better surely has an idiomatic API - otherwise search a better one or please don't expect your average developer come up with one. I have seen this too often, countless attempts of unecessary wrappers added around fine APIs because we need to own it more, test it better or replace it in some very eventual future.. and what you get is almost always less idiomatic APIs, that transform results badly like swallowing errors, and what else, often requiring me to now learn two APIs, a badly documented but owned one (or just read the source code!!!) and the real one. Sure I hear what the author says.. but practice with the average dev and this turns out different in reality :(
- deleted 4y ago[deleted]
- oweiler 4y agoWhen testing external systems it's often better to use fakes, or service virtualization, though this is sometimes not possible.
- cemoktra 4y agoPlease mock everything. It's just annoying when your CI starts failing because a third party service is down multiple times a month
- wizofaus 4y agoAnd even if not, your CI tests take hours to run because of all the network calls. Automated tests that do real calls to 3rd party services are essential too, but they should almost never be part of your standard CI scripts that get triggered by every push. But I think the article was mostly talking about mixing network/http level mocking with business logic tests, which I'd admit isn't always ideal, but IME complex test cases often require some level of compromise.
- pierrebai 4y agoWhat the blog completely passes over is that now you'd need to write tests for the DockerRegistryClient... which would be 90% of what he complains about in the first version! I mean sure, if you break down a problem into three layers instead of one, then the top layer testing will be three times cleaner... but you'll write three times as much code and three times as many tests, one for each layer. The end-result might be better layered code, but it takes time to write, and time to test and, the not-so-little downside of "clean" code layers that is often never discussed and tossed aside, you need to read and understand three layers of code to know what is really going on instead of one. It is worth it when your layers are getting large and heavy, but for three-lines-of-code thing like this... is it really worth it?
- hynek 4y agoNo, it doesn’t pass over that. It specifically points out that it’s a simplistic example for illustration that probably wouldn’t be worth it. And yet the effects are visible and so are the problems/solutions. I mean what do you want here? 1,000 LoC in a blog post so it feels worth it?
- TillE 4y agoThe more I think about this, the more I like it. It cleanly splits the problem into two testable questions of "am I using this API correctly?" and "is my own logic correct?", without adding tedious complexity. I'm generally a fan of simple wrappers for APIs when you find yourself writing a bunch of redundant code, and testability is a nice advantage.
- jjtheblunt 4y agoI can imagine an alternative is "don't mock, period. Write functional code with tests, so the hack of mocking vanishes as a consequence of composition of functional code." but that's an earful.
- scotuswroteus 4y agoLOL, I absolutely will
- spathi_fwiffo 4y agoI don't fully I agree. This seems like it has it's place, but there is very high value in a Unit Test that can run with no dependencies outside of your compiler/interpreter. Especially in interpreted languages -- to catch whatever the linters may miss. The moment you need to access the network, you depend on everything that comes with the network (outages can happen); Need a DB -- well then you need to start it, prime it; maybe you need a container/etc. That's just a ton of overhead to validate that your steps are working, and that your code is clean. Maybe for a small project this works, if you only have one layer of testing.
- nijave 4y agoI think mocking external services sometimes makes sense. As the author mentions, maybe you're adding a regression test for a fix when dependent service was misbehaving (500 errors, really really slow responses, excessive rate limiting) and sometimes getting such a scenario setup requires changing very specific pieces
- makavelijaoko 4y agohelp
- malfist 4y agoI wholeheartedly disagree with the author here. Not mocking 3rd party dependencies is a bad practice, it's how you wind up with fragile and flaky tests that can't always be ran locally. Not mocking 3rd party is done in integration or orchestration testing. Not your unit tests.
- ollien 4y agoIt depends on your language and testing framework. In C++-land, for instance, you can't mock a third party library if they don't define virtual methods, forcing you to make a wrapper (an experience with this is what taught me this mantra)
- int_19h 4y agoIf you are mocking the library in its entirety, you just define the same classes/functions that they do - virtual or not - and then link to your implementation instead of theirs for the test build.
- ollien 4y agoHeh, that's an interesting strategy. My company primarily works with gmock, and I'm not certain you could do what you describe with gmock, but yeah that would work otherwise.
- chc 4y agoThis seems like an is-ought fallacy. What you're describing is not a "you shouldn't," it's a "you can't due to limitations in the language."
- ollien 4y agoTIL about that fallacy. That's a fair point. I guess it comes in a bit differently in C++ land, because the "you shouldn't" comes into what you decide to allow to be injectable (e.g. if your constructor only accepts the third party object type, you're not gonna be able to mock it). As such, you gotta keep the principle in mind when designing your objects. That said, you're definitely right, it smells like that fallacy
- borkwow 4y ago
- a-dub 4y ago"Only mock what you own, because mocking others is rude." i once referred to a codebase as foreign, meaning (to my knowledge) not originally developed by my team or division, when justifying not writing new unit test coverage for existing code without discussion, only to realize much later that the owners were actually abroad and may have been offended by that statement. sigh
- lioeters 4y agoHeheh, in retrospect that does sound like a plausible thing someone might say, "foreign code" meaning "that code written by foreigners". I wonder if there's been studies about cultural influences on code, like how Japanese programmers write code, compared to other countries, etc. Even within the U.S., there was the whole thing about the MIT approach (do the right thing) and the "New Jersey" style (worse is better). "What do you expect from an operating system designed and implemented in New Jersey!"
- kaidon 4y ago"We can solve any problem by introducing an extra level of indirection." https://en.wikipedia.org/wiki/Fundamental_theorem_of_software_engineering https://en.wikipedia.org/wiki/Fundamental_theorem_of_softwar...
- pydry 4y ago>We need three layers of mocks to verify that an empty repositories key leads to an empty dictionary. And if I didn’t use a lambda for the json function, it would be even four layers. Yep, but those 4 or even 5 layers ought to be mostly reusable between one codebase that mocks an HTTP call and another - if they are abstracted away this line of reasoning falls apart, coz then it is simple. I think the "what to mock?" argument suffers from an overabundance of dogma and a lack of trade-off based decision making. This article is one of many examples of this.
- hynek 4y agoHey, I’m the author and happy to engage in constructive exchange with everyone who read the article and has thoughts or questions (NB the full title is ‘“Don’t Mock What a You Don’t Own” in 5 Minutes’ the principle is super old and this article is just an explanation why it makes more sense than it sounds.)
- btown 4y agoThere's a pretty significant problem here: with this approach, you lose test coverage of the code within your client wrapper. Namely, there's code like this which is not being tested: `.json()["repositories"]` Now, one might argue, those calls are so trivial that it doesn't need coverage, right? But what if you were trying to get the results from a nested key as a list and also check for an error status etc. etc.? Or what if this is XML deserialization with certain assumptions baked into the deserialization code? You could easily add complexity that you think your tests are covering, but they aren't. I prefer instead to use https://vcrpy.readthedocs.io/en/latest/ https://vcrpy.readthedocs.io/en/latest/ which helps you to create a fixture of what you don't own, and hooks every possible requests/http/urllib system to ensure you're not leaking a hit to a production service on any but your first test run. Tests even need to worry about mocks in most cases, they just need to be wrapped in a simple annotation. In a way, it's precisely the opposite of the OP's conclusion: mock what you don't own, at the lowest possible level to ensure all the code you wrote is covered. But use the right tool for the job!
- oweiler 4y agoIf you have complex logic for extracting nested JSON, put it in a function and unit test the function.
- hynek 4y agoI’m my experience, you get most of that kind of coverage with integration test/user story tests/however-you-call-them. Solutions like vcr aren’t bullet-proof either and are a variation of in-process servers that the article mentions. To put it differently: the question is not whether vcr, but _where_ vcr.
- theptip 4y agoI was going to make the same observation, and suggest the same solution, though I’d advocate for doing both things. You definitely do need at least one test getting full coverage for the facade client simply because someone could accidentally insert white space that completely breaks the logic. Carving out a facade you own makes it really easy for all of the rest of your system to behave predictably in tests; for example what if you call an analytics API on every action? Probably easier to have UTs by default assemble your app with a MockAnalyticsClient with sensible noop mocks instead of actually wiring up the real mocks required for the library client. I’d call VCR “integration-ish” tests. I argue it’s ok to check in a blob of API response data because (ideally) your remote API doesn’t change that often. Ideally you do want to be able to run the integration tests periodically (nightly?) without the recorded response to make sure your assumptions about the remote api haven’t broken. That last bit is sometimes tricky to wire up. It’s also worth noting that VCR has some nice bells and whistles to do substitution (to elide secrets) and can also store the test data in say a storage bucket, if you don’t want to set up git-LFS or put the response data in your repo. (SOAP gets expensive as you may need to store the whole WSDL for example.) The key thing IMO is keeping the VCR out of all of your business logic tests, and confined just to integration tests targeted at the facade class. Biz logic tests should be as clean as possible and simply mocking the facade helps achieve that.
- luigi23 4y ago"Don't do X, don't break the rule, you're doing it wrong"... that's why people don't write tests (well, maybe not the main reason, but that was my case at least). Do whatever that will increase confidence in code and testability. If I want to make it easier for me to develop, I'm gonna do anything that will enable me to get shit done. Test takes 0.1 second instead of 0.001? Whatever. I can improve it if needed, when I get more knowledge or I can discard it completely. When I stopped caring about 'should I use mock, or inject closure, or interface...' I started writing more tests and producing better software. Because often the end goal is about writing clearer code even with shitty tests.
- Groxx 4y agoFull agree, with one additional detail: Tests are code too. Like the code being tested, they can be shitty at times, but they are still code and should be treated with the same level of care. Which is not to say "your tests should be as DRY as a desert". That's frequently a poor fit for test code's lifecycle. But it does mean you shouldn't do stuff that wouldn't pass code review elsewhere, just because "it's just a test". Someone later has to read, understand, and maintain that code - treat it as such.
- oweiler 4y agoI get paid for code that works, not for tests Kent Beck
- deterministic 4y ago… and if tests helps me making the code work then it is worth doing.
- arubania2 4y ago> "Don't do X, don't break the rule, you're doing it wrong"... that's why people don't write tests I think that's why people don't read blog posts like this - such wording is rude and patronizing.
- lifefeed 4y agoThe implication with these types of articles is always "any tests are better than no tests, but here's how to do better tests."
- deleted 4y ago[deleted]
- spawarotti 4y agoThis rule is also recommended by section 9.2.4. "Only mock types that you own" of 2020 book by Vladimir Khorikov: "Unit Testing Principles, Practices, and Patterns". Worth noting is that book also says in 9.2.1: "Mocks are for integration tests only". I agree. In my opinion this book is the best comprehensive take on testing I ever encountered.
- loevborg 4y agoSecond the recommendation, great book even if like me you don't care about C#
- waynesonfire 4y ago> "Mocks are for integration tests only" as opposed to what unit tests? how are you suppose to "unit test" that you properly increment a varibale before inserting into the DB without mocking the DB? I disagree, mocks are for unit tests only. mocks allow you to isolate the unit to be tested. integration tests is when you start integrating dependencies together. so maybe you have a dummy database like an in-memory db, but you're certainly not mocking it. system tests is where you would use the actual db.
- spawarotti 4y ago> how are you suppose to "unit test" that you properly increment a varibale before inserting into the DB without mocking the DB? You put the logic that increments that variable into a pure function and unit test the input/output pair. Because it is now decoupled from the database, you don't have to deal with it. > mocks allow you to isolate the unit to be tested. You achieve isolation instead by refactoring to a "pure functional core and mutable outer-shell" architecture. Above I gave one example of refactoring to functional core. In the cases where you still need to deal with external dependencies in unit tests, you use in-memory implementations (aka simulators) instead of mocks. Basically there are two subtypes of unit tests - small focused unit tests, testing input/output pairs of purely functional code, and "bigger" unit tests, that check how bigger units of internal business logic collaborate together. They use the in-memory simulators to ensure the tests maintain all the properties of a good unit test: runs fast, doesn't interfere with other tests, requires zero setup, is not flaky and makes zero assumptions about the environment. So, overall - no mocks for unit testing. > mocks are for unit tests only. I do think the opposite is true, with very few exceptions. Mocking in unit testing is an anti-pattern leading to brittle tests with negative value - the cost of maintaining them is way higher than any benefit they provide. Too many false positives, too little true positives, too much rework needed when code changed but no bug was introduced, too unreadable code. > integration tests is when you start integrating dependencies together. so maybe you have a dummy database like an in-memory db, but you're certainly not mocking it. This is the only case mocking makes sense. In integration testing you want to test integration with one external endpoint. Hence you mock or simulate all the others. In unit testing there is no need for mocking as explained above. In system (end-to-end) testing there is no need for mocking because we test how everything integrates together. Naturally, there is way more nuance to the simplified statements I made above. The book elaborates on that. There is also an article by the same author about it: https://enterprisecraftsmanship.com/posts/when-to-mock/ https://enterprisecraftsmanship.com/posts/when-to-mock/
- jacobsenscott 4y agoI prefer to mock below the level of the 3rd party client library. In rubyland we use VCR to stub out the lower level network libraries to record the requests/responses. Then during the test runs your network libraries don't make real connections to the APIs - they just play back the recorded responses. Python has ports of VCR (betamax and VCR.py). The benefits of this are numerous. You don't need to manually build and verify mocks. You don't need to build a facade for every api client (the api client is already a facade). You exercise the as much of the 3rd party library code as possible on each test - this is very helpful during library upgrades. You can verify your recordings by switching your test to "live" mode every now and then (just configure your CI env do that every day or every week).
- hynek 4y ago> You don't need to build a facade for every api client (the api client is already a facade). I feel like this is covered by > Every rule and principle can be broken once you’ve fully understood its purpose. For example if an object already does have an idiomatic API, it’s probably not worth wrapping in an identical façade, just so it belongs to you. ? I personally prefer pytest-httpserver that is a bit more low-level, but I still prefer idiomatic calls over sprinkled http requests on business code.
- jacobsenscott 4y agoYes - it is covered by that. I do build facades for apis that don't have good client libraries, but for things like s3, or stripe - with solid libraries I don't. Even when I build those facades I don't mock them out - I just rely on VCR. I've never used test server like pytest-httpserver. I don't think that would buy much vs using stubbing your low level network libraries like VCR does. The killer feature of VCR is that it records the real network calls on first usage. If pytest-httpserver can do that in a proxy mode it seems like a good way to go though.
- zeroonetwothree 4y agoTesting is not one size fits all https://biggestfish.substack.com/p/testing-is-not-one-size-fits-all https://biggestfish.substack.com/p/testing-is-not-one-size-f...
- charleslmunger 4y agoCheck out this[1] API for enforcing rules like this in your codebase. [1] https://javadoc.io/doc/org.mockito/mockito-core/latest/org/mockito/plugins/DoNotMockEnforcer.html https://javadoc.io/doc/org.mockito/mockito-core/latest/org/m...
- waynesonfire 4y agohow is writing tests scalable? how do you decide what to test? three parameters, maybe three classes per parameter, -1, 0, +1.. you now have 3^3 test cases just for one function.
- klodolph 4y agoThat's a whole discussion on its own. Tests do not have to guarantee that the behavior is correct for all possible inputs. If testing all possible inputs is easy, do it. Otherwise, you write the tests which you think are most likely to catch errors. Try to exercise different code paths to catch errors in different parts of the code. Try to use different logic in your test from the logic in the code under test.
- bern4444 4y agoTesting generally can be either a great or painful experience. The more mocking you have to do (or setup for your testing environment) the more painful it is. I try and avoid mocking, or mock as low as possible. Ideally mocking only happens at the HTTP level for network requests and APIs. Having to mock internal functions (that don't make network requests) creates a frustrating experience. I also find using typescript a great way of writing little unit tests. Making sure types line up, especially when types are highly relational, is exceptionally valuable.
- jayd16 4y agoSo the supposed lesson is that generic third party APIs are hard to mock, so you need to write helper methods everywhere so you can mock those instead?
- hynek 4y agoI know you’re being facetious but: the lesson is rather that mocking generic third-party APIs is so easy, that you can end up in mocking hell real fast. And that you should take the harder route for your future sanity.
- normie3000 4y agoOr alternatively, don’t mock what you _do_ own.
- vbezhenar 4y agoIf we're talking about mocking in general, mocking foreign services might be very useful and may be even essential for good development practices. For example imagine that you're integrating with some other system. You might receive some test credentials during development time, but they'll lost eventually or something will break. So you'll have production integration, but no way to properly develop your code. So I'm always writing some emulator for foreign services. It might be as simple as mocked responses or it might contain its own little database and some primitive logic. But in the end you don't depend on someone's test system, you can run your integration tests, you can run your staging environment and so on. That might not be necessary if you're integrating with some big guys like stripe or facebook. They probably have enough resources to maintain test environments. But for some little company which don't really care about test environments and whether it corresponds to the production environment - that kind of mocking might be necessary for sanity.
- pyrolistical 4y agoI agree, but services you consume should provide their own mocks! I've written about this on how LocalStack is compensating for the lack of proper mocks from AWS: https://blog.battlefy.com/localstack-is-great-but-should-not-exist https://blog.battlefy.com/localstack-is-great-but-should-not...