11 ms·

Favor real dependencies for unit testing

151 points by feross 5y ago

ctrlaltdylan 5y ago
So TLDR - write integration tests as much as possible and only mock external dependencies like network calls?
strulovich 5y ago
Amen to the idea. - Prefer real objects, or fakes over mocks. It will make your tests usually more robust. - Use mocks when you must: to avoid networking, or other flaky things such as storage. - Use mocks for “output only objects”, for example listeners, or when verifying the output for some logging. (But, prefer a good fake) - Use mocks when you “need to get shit done”, it’s the easiest way to add tests in an area that has almost none, and the code is not designed to be easily testable. But remember this is tech debt, and try to migrate towards real objects over time. That’s my short advice I told many times. So might as well comment with it here.
- tomcam 5y ago
  It all sounds great. I agree totally in principle! I am finding that testing my fairly small Go project (static site generator, because the world definitely needs another one) chews up massive amounts of time. So I tend to avoid the testing pass for longer than I should. Any thoughts on that issue?
  - chii 5y ago
    if you don't mind spending the effort, have a quick profile to see where the tests are taking a long time - is it initializing the real components that could be mocked? Or are the tests themselves taking a long time because of other factors, such as IO etc? For IO, may be there should be an abstraction over these IO api and you use an inmemory option instead for testing.
    - bluGill 5y ago
      Of io is the problem I general solve that by testing smaller data sets. I have not found io to the local disk is slow. Of course if it is network io that is bad, but local disks are fine for the size of data in my tests
  - vlovich123 5y ago
    Make it CI’s problem. You should only really be running tests regularly for the component you’re currently working on with CI making sure you don’t have accidental regressions.
  - edoceo 5y ago
    I try to write the test first, or at least stub in the test how I think it should work. Or at least enough notes to pick it up "tomorrow". Then I have a reference for usage i can build with that in mind
  - jerf 5y ago
    It isn't immediately obvious to me why a small Go static site generator would require "massive amounts of time" to run its tests, so it's hard to answer what you're doing wrong. Are your tests perhaps just too darned big? You don't, in general, need to render 5000 pages of something to test your template doesn't crash or something. It could also just be disk access. Consider trying an in-memory file system, or if you're on linux, look at using /dev/shm which is a RAM disk. It is also possible you've snuck in a quadratic or worse time algorithm. There's nothing fundamental to the problem of a static site generator that would require such algorithms, but speaking from experience it is an environment where it's easy to loop over the return value of one function, which itself loops over something other function (quite likely the same data), which itself loops over the same structure, and it's easy to end up with O(n^3) or O(n^4) without realizing it. It's especially easy to end up with that being a "for each page" type of loop. Static site generation should be O(n), give or take small factors (maybe O(n log n) for some things technically, but at a scale where O(n log n) is practically O(n) anyhow).
  - tomcam 5y ago
    Thanks all for super helpful feedback. This was a generous gift of your time and brainpower.
- bluGill 5y ago
  I have not found storage to be flaky and so I don't mock it. Tmpfile always gives me a unique file, and that is all the fake I need. I don't even look up the various forms of temp file to see which ones don't have a race condition as in practice they never do (if I was writing encryption or other such production code I would, but for a unit test the odds of. A race causing a failure are low enough to ignore)
  - epage 5y ago
    At minimum, you need to be able to choose where the files are stored to use temp, I find thtre is still the problem of complicated setup. Also, if you are going for particular semaitics (e.g. cross process interactions), isolating that for testing it more specifically can be helpful. For me, I look for higher level abstractions and mock when possible and don't sweat testing against temp files otherwise. I've had several TDD people jump to wanting to mock each filesystem call. One was for a cross process storage API. I was trying to get them to just have a Backend interface for reading and writing instead, with tests just using an InMemory implementation as a Double.
- Tainnor 5y ago
  I think the original ideas of mocks (if you go and read Growing Object-Oriented Software, Guided by Tests) had some merit: In that style of TDD, mocks are used to discover (hopefully somewhat stable) interfaces between components, and in theory, it fits with the idea that OOP is about "objects sending messages to each other". I can believe that it's possible to write good systems with this kind of approach. Unfortunately, in practice, mocks are rarely used like that and most "OOP" designs have horrible boundaries and are really not much about message passing anymore. That leads to brittle mocks where you constantly have to change tests when you change implementation details. I have also gravitated away from classic OOP and much more towards the "functional core, imperative shell" concept as outlined in the article (although it's difficult to keep this pattern throughout a codebase, especially if you have team members). In such a system you really rarely need mocks. Agreed that fakes, when you have them, are nicer than mocks, especially when the system to be faked has a large API (i.e. use a redis fake, instead of checking the exact commands you send to redis). However, for some outside systems, writing a fake can be a lot of effort. In such a case, I think it's totally valid to write a "gateway class" that isn't unit tested (you can cover it with integration tests instead) which exposes a nice API (e.g. "storeFile(...)") and then to use mocks of that class in other tests.
  - arcticbull 5y ago
    In test cases where you extensively involve mocks, more often than not in my experience, you end up testing that your mocks do the thing you told them to.
    - mikepurvis 5y ago
      Yeah, particularly the case with a lot of "glue" type code that's really just passing stuff back and forth and not really making any decisions with it. I've always struggled to feel that mock-based testing in this scenario was anything other than busy-work.
    - pstuart 5y ago
      Yes, but the coverage numbers look great!
- mirekrusin 5y ago
  - Prefer simulators over mocks We're running business critical system for several years now, creating simulators has been one of our "secrets" that contributed to the success. Not only our tests are running in as close to production environment as possible, we're also using them for local development where developers can spin up functioning system on own dev machine.
  - valenterry 5y ago
    I agree - but it can be hard to get good ones, e.g. if you need to simulate sql query execution.
    - bluGill 5y ago
      Sqlite is a great simulation for SQL. You need to limit your SQL to the subset that us supported by both it and your target database though, which might be a problem.
      - jon-wood 5y ago
        In a world of Docker existing there’s rarely a reason not to just use your target database - I can have a non-production grade Postgres instance up and running in less time than it took me to write this message.
    - withinboredom 5y ago
      We spin up a temporary and local MySQL instance per test run with helpers to generate necessary data on-demand. All our tests therefore use real queries on a real db. Due to speed constraints, this db is shared between all the tests you run in that session, so it’s possible to influence tests after yours runs in CI. The reality is that that is pretty easy to detect and it’s caused us to write less brittle code.
      - mirekrusin 5y ago
        We're using mssql, on bootstrap we're running all migrations, seeding database, then backing it up; each test file restores its own database from backup which is much faster than alternatives; e2e tests run against simulated environment that has single database; e2e are split into concurrent runs with distinct set of tests to speed things up.
- collyw 5y ago
  Generally the more difficult a test is to write and run, the more useful it is.
mrjin 5y ago
There are also Function Tests and Integration Tests. Seems the author took Unit Tests for all three.
- howinteresting 5y ago
  No. this is a false di/trichotomy. What matters is the scope of code a test covers. Some tests cover a greater scope, some tests cover less.
vlovich123 5y ago
This is almost right except for two points: For fakes, spin up the real thing. If you’re not able to model your database transactions deterministically, then your transactions could themselves be flawed and tests are great way to catch that. Deterministic tests are not a goal in and of themselves. Controlled non determinism is valuable. This is popularized in various frameworks under the names of property checking and fuzzing which will let you know the seed to use for the failure for example so that while the runs don’t have the exact same input/output for every invocation, you get better coverage of your test space and can revisit the problematic points at any time. If you’re doing numeric simulation, make sure you are using a PRNG that’s seedable and that you log the seed at the start if you’re using a seed (and make sure time is an input parameter). Why is this technique valuable? You transitively get increasing code coverage for free through CI/coworkers running the tests AND you have a way to investigate issues sanely.
- blt 5y ago
  "Non-deterministic tests" usually refers to a test whose output depends on the execution environment in a way that cannot be controlled. For example, a multithreaded test with a race condition, or a test that uses the real-time clock. This is fundamentally different from a test that uses random numbers. Another way to look at it: Deterministic tests can be used in `git blame`. A unit test that uses a specific PRNG algorithm and sets the seed is fine. A test with a race condition is not. Fuzzing is very useful, but it's not unit testing. If you want to run a fuzzer or some other kind of endless randomized testing in CI, it should be a separate job from the unit tests (IMO).
TravHatesMe 5y ago
In other words, shouldn't we prefer integration testing, or perhaps partial integration testing. This requires an initial effort of setting up your test framework/environment, but in my experience integration tests provide good value for the time you put into testing. Rather than a granular test on a single class, test the orchestration of many classes. You end up hitting big % of code. As always, depends on the project. If we're building a rocket ship, you need both granular testing and coarse testing.
- Tainnor 5y ago
  People keep saying this all the time, but apart from the fact that nobody can agree on what an "integration test" is (because there's almost always some part of the application flow that you're stubbing out), it just becomes immediately apparent in a code base of sufficient size that "just use integration tests for everything" is only possible if you severely under-test (which usually includes thing like not properly testing for error conditions etc.).
  - code_biologist 5y ago
    What? Nobody is advocating for integration tests to the exclusion of unit tests. but apart from the fact that nobody can agree on what an "integration test" is Not being precise doesn't invalidate a guideline. The ideas that "stubbing less is better" and "testing functionality end to end is good bang-for-buck" aren't crappy because people don't agree on the details. it just becomes immediately apparent in a code base of sufficient size that "just use integration tests for everything" is only possible if you severely under-test Your parent comment said "If we're building a rocket ship, you need both granular testing and coarse testing." Nobody is advocating for integration tests to the exclusion of unit tests. If you write a hash table or a CSV parser, yes you should unit test it. But for most application-ish functionality you should reach for integration tests first. For example, testing direct message functionality in an app, checking that after a send there's a notification email queued and the recipient inbox endpoint says there's 1 unread will get you really far in 20 lines of code. Is it exhaustive? Of course not. But the simplicity is a huge virtue. if you severely under-test (which usually includes thing like not properly testing for error conditions etc.) I advocate "default to integration testing for application functionality". Those focused on unit tests often mock exactly the things most likely to break: integration points between systems. "Unit" tests of systems are often really verbose, prescriptive about internal state, and worst of all don't catch the bits that actually break.
mbfg 5y ago
the thing i see that makes this test brittle is the use of relative dates and durations that break across day light savings time changes
jetru 5y ago
inb4 someone saying something about mocking or dependency injection for testing
- sedro 5y ago
  now you have N+2 problems?
eatonphil 5y ago
In database land you can even freely bring up a number of _proprietary_ databases without needing an account or license. I spin up MySQL, Postgres, Oracle and SQL Server databases in containers in Github Actions for tests. Unfortunately IBM DB2 seems to require a license key to spin up their container. :(
kmac_ 5y ago
I have used this approach and now I perceive mocks as a bad code smell. It's worth noting that long and complicated flows need to be wrapped in sagas (which are just imperative shells) and things can get not so easy. I would love to hear what are other clean alternatives in such cases. Still, "functional core, imperative shell" is the way to go, code really fits in the head and tests actually make sense.
Supermancho 5y ago
The heart of the argument presented is that using mocks in unit tests is problematic because if an interface is changed, that will possibly break every test that involves mocking that interface and that's friction to making changes. This is silly. If you use real implementations and change the interface, you have the exact same problem except that your configuration/setup is also likely to have to change in subtle ways. Let's just assume the absurd notion that a choice of creating tech debt or changing a common interface is a real choice. If there was a serious change that could not be accounted for with easy test changes, I'm not the only one to see the tests commented out with a "TODO: fix these". Developers tend to be pragmatic. If you want to change code you will always measure the effect it will have on the code and tests are incidental, not the primary concern. Make it work. Make it right. Make it fast. I want to be able to trigger all code paths and exceptions (make it good). Using a real dependency, I would be left in the unfortunate situation of depending on knowledge of the internals of that dependency. It may not allow me to execute specific paths via pure configuration at all. I don't think using real dependencies is a good idea at all for unit tests. Integration tests are a different and I do fear that they are being confused.
- spawarotti 5y ago
  > The heart of the argument presented is that using mocks in unit tests is problematic (...) If you use real implementations No. As you say, the moment you have a real out-of-process dependency, you no longer have a unit test, but an "integration-with-external world" test. The heart of the argument is to not mock out internal (in-process) business logic. Instead: a) do not use mocks (e.g. leveraging Moq); use fakes (aka "simulators"), i.e. proper classes having in-memory implementations of the out-of-process dependencies and crucially b) replace only out-of-process dependencies, which usually are flaky/nondeterministic. Never replace internal business logic. If you feel you need to replace internal business logic, you are not following the functional core architectural pattern. After you refactor to functional core you will no longer need to replace any dependencies in your unit test, as there won't be any - you will just call your tested pure method and make assertions on the returned value. Special case here is testing "top-level/controller" logic. Here you need to use the in-memory fakes, but there will be only few, reused across all tests, and such test will be an "end-to-end internal-business-logic integration test", but still it will have all the properties of a unit test - it will be fast, deterministic, and you will be able to run it as part of a unit test suite out-of-the-box, with no environment setup necessary. Further reading: https://enterprisecraftsmanship.com/posts/when-to-mock/ https://enterprisecraftsmanship.com/posts/when-to-mock/ > If you want to change code you will always measure the effect it will have on the code and tests are incidental, not the primary concern Exactly. Mocks are very brittle and break all the time, causing unnecessary rework. If you instead rely on a small set of fakes of out-of-process dependencies, you will drastically reduce test suite rework and improve signal-to-noise ratio. > Using a real dependency, I would be left in the unfortunate situation of depending on knowledge of the internals of that dependency. And what happens if the real dependency changes behavior, but you forget to update your mock? You will have a test that will be running against a mock that simulates obsolete behavior, no longer present in production. In worst case scenario the test will be green/passing, while there is a bug in production. Will you remember to always comb over your entire test suite and review all mocks to faithfully simulate actual production behavior? > It may not allow me to execute specific paths via pure configuration at all. If you write fakes, you will have full control over how to configure them.
hansvm 5y ago
As a potential counter-argument, the use of mocks can enable testing of functionality that the current concrete implementation doesn't exercise. It's easier than one would think to accidentally rely on implementation details rather than coding just to the interface (and optionally any documented restrictions to that interface). They explicitly call out clocks as a source of non-determinism that probably should be mocked, but I'll re-use them as an example anyway because everyone is familiar with them: it's extraordinarily useful for the tests to execute nearly immediately rather than actually waiting on a clock, and rare behavior like a clock running backward, two consecutive timings being identical within the clock's resolution, or whatever other weird artifacts that your code should handle are definitely better explicitly tested rather than not mocking the clock. Other domain-specific interfaces are often similarly able to exhibit a weird edge case that ought to be explicitly tested (rather than accidentally relying on a "nice" implementation) if you really want to unit test the callers and not integration test the coupled system.
- TheReveller 5y ago
  Also error conditions - telling a fake to give you a specific error before you run the test is no different from telling a mock to do it.
  - marcosdumay 5y ago
    It is different. You tell the mock to give you an specific error with specific information, while you tell the fake to give you the error for an specific condition. It is subtle and may not be worth the time, but again, it may be.
- smaudet 5y ago
  I think you have missed the point of fakes versus mocks (as this article puts it) based on what you said - fakes should be able to do everything mocks can, more, even. But I hear you complain "but that's so much work to maintain!!" - mocks are more work to test less, if you need a better fake often, write a library which has them. What you are referring to is known as fuzzing, not unit testing, and fuzzing has a better track record than straight up "testing" generally for finding issues. Fakes can have deterministic or indeterminate behavior, the case the OP makes is for deterministic "unit" testing with more real behaviors than not. This doesn't preclude good fuzzing... "It's easier than one would think to accidentally rely on implementation details" To be somewhat philosophical, what if that 'implementation detail' is the feature? No amount of testing will save you from incompatible feature sets, to 'enable testing of functionality that the current concrete implementation doesn't exercise' you run the very real risk of baking in incompatible sets of features into the core of your app. You shouldn't be designing your app at the unit level...
hermanradtke 5y ago
I first learned this concept from Gary Bernhardt’s video Boundaries. https://www.destroyallsoftware.com/talks/boundaries https://www.destroyallsoftware.com/talks/boundaries
cassac 5y ago
I don’t get it. When I unit test, I want to only test what I’m testing. A dependency or the result of a dependency is not what I’m testing. So I mock the result of that dependency to unlink the test from the dependency. If an interface changes I generally want to know anyway, as the test may be different or even obsolete depending on the change.
- crdrost 5y ago
  I hear you, and that's how things work at my present shop. However, let me give a strong defense of the point. It turns out that the only thing you can test is a pure function. This is just the nature of a test, we pin down the inputs to a piece of code and we see what outputs it produces. All of the mocking that you are doing is an attempt to turn an impure function into a pure function. Even more extreme setups where you connect your container to a container running postgres, is trying to turn that container into a pure function in a different way. You don't need to do any of this if the function is pure in the first place. Once we've established that “functional core” is a lazier means to the same ends the question of dependencies still comes up, and the “should I mock dependencies” question becomes “should I promote this internal function call to the main I/O section and pass in its result as an argument?”... And the answer is that that changes the language with which this outermost level is written. And probably this outermost shell should be written to sound like the business logic that you are implementing, and you should never do that. If that's the logic then it means that the sort of testing you're doing is extremely particular and fussy. Because it suggests that every test should really be constructed at a business level, it is a story about your product that you want to make sure holds fast even when the internals are changed. This works really well with domain driven design, because that says that your modules should be also business level entities, so each module comes with some tests that say, here's what this sort of person interacts with the system like. So you are always testing integration of the pure functions, and why would you not. If those pure functions do not integrate together, you want to know about it, and you do not want to know about it through a persnickety test which just fixes the inputs and outputs of something that has no business relevance, because you know what happens in those cases: the developer just rewrites the test to say the opposite of what it used to say, so that it passes now. There is no semantic check on the test output because there cannot be if it is scoped too small. I think you can quibble a lot with those details but I think that's the strongest case you can make for it?
beiller 5y ago
I prefer mocks over real services because I onboard many people to a repo. Too many services attached to unit tests turns into a fatigue until eventually no one runs the tests any more I found.
spawarotti 5y ago
This article espouses what is known as "classicist testing" school of thought and rejects "mockist/London-style testing". I wholeheartedly agree with it. I have been championing it in my team of 20+ devs since many years now, to great effect. You can read more about it in the excellent book by Vladimir Khorikov from January 2020 titled "Unit Testing Principles, Practices, and Patterns".
sour-taste 5y ago
I have two services that communicate over the network. Neither does anything useful without data from the other. So I write my tests like this article suggests and inject at the boundaries of the services, the network boundary in this case. I've built tests that now only test my idea of what the services will return. IMO these aren't particularly useful tests: what happens when the remote service starts returning unexpected data or errors due load problems? I guess my point is just this article is good advice: prefer testing your actual code and dependencies, but unit testing isn't a replacement for integration testing.
kromem 5y ago
If you are testing the externalities, they aren't unit tests, but integration tests. The better advice is: continue to isolate unit tests away from real dependencies, and ALSO have integration tests that test the way the package connects to dependencies (and the other packages in the software dependent on it).
- wtetzner 5y ago
  I would argue that if you need mocks, the code isn’t well suited to unit testing and you’re better off with integration tests. If you want to unit test it, then refactor in such a way that you no longer need mocks.
quickthrower2 5y ago
I worked somewhere with a hard rule that you are not allowed to test internal classes. So if I write my own sort algorithm I am not allowed to unit test that class unless I make it public. But if the sort algorithm is to solve a specific sub problem in a library there is no need to make it public - it may not make much sense. So I had to test it “through” its consumer class(es) that is public. For illustration sake lets say a MVC controller mocked up to the eyeballs with ORM mocks, logging mocks etc. This bugged me because having direct access to a functional core I can quickly amplify the number of test cases against it and find hidden potential bugs much quicker.
- omegalulw 5y ago
  I would say that testing only the public contract is generally very helpful. When the logic gets super complex, I would think that a good comprise is to modularize your code - and test the public API of each module. A good heuristic is to see if that modules are (or could be) helpful by themselves in the future. You then write tests for a module assuming it's dependents cover their edge cases.
  - quickthrower2 5y ago
    As I wrote it I thought the same thing! I think some of the problem was org friction of adding a new module, getting approval to do so. This was a place with about 400 devs so they could really let anyone freely add modules and waiting for architecture approval would take too long for a typical ticket.
- Supermancho 5y ago
  > So if I write my own sort algorithm I am not allowed to unit test that class unless I make it public. I think there's a slow movement away from this kind of straightjacket. The compiler sees it all anyway. Class accessors serve 2 purposes. 1. They are a social tool; and this is only if you believe that developers who work on source code, which they can read, cannot be trusted to call methods to do what they need. 2. They are a convenience for opaque modules, so that users who may not have access to the source code, can avoid using APIs that may have no effect or problematic side effects, while also decluttering the API. Python allows for methods to act as if they are declared public when using a special syntax eg _backdoor(). Go allows for any method that is in the same package, to access any other method regardless of accessors...like tests for that package. In javascript, you can use rewire.js to mock methods in imported modules (which are effectively methods in closures). These solutions are an improvement to the classic rigid class access pattern that many languages are stuck with and forces the kind of theatre you have experienced.
daxfohl 5y ago
I 100% agree, but rarely have found much to put in a functional core. Most everything these days is distributed microservices talking to each other (different argument) and having more than a couple lines in a row that don't call some other service, even in a well-factored app, is almost a cause for celebration.
kjgkjhfkjf 5y ago
I agree that fakes are very good for testing, with one caveat: the fakes must be owned by the same folks that own the real implementation. When fakes are owned by anyone else, e.g. the folks writing the system under test, there's a high risk that the semantics of the fake and the real implementation will diverge, rendering the tests that use the fakes much less useful.
exdsq 5y ago
I’ve spent the last few years working on test infrastructure for blockchain projects so this question is regularly on my mind, and I’ve found the best solution is to just include the dependencies. Compute and storage is cheap enough that you can spin up these external tools and the extra level of assurance makes me sleep easier, and while flakey tests are theoretically an issue I rarely find them (and they tend to be programmatically fixable). If a unit test requires an external dependency then just use an integration test (or check it’s covered by system tests) and leave it at that.
swiftcoder 5y ago
I see another soul has rediscovered the universal truth that unit testing is only pleasant in functional programs
register 5y ago
Finally! Somebody that has the guts to say the truth.
globular-toast 5y ago
Ian Cooper talked about this (and more) 5 years ago: https://www.youtube.com/watch?v=EZ05e7EMOLM https://www.youtube.com/watch?v=EZ05e7EMOLM Well worth listening to this talk. It completely changed the way I did testing.
mkl95 5y ago
If some method has several dependencies that are tangentially related with what I'm testing, I'm going to mock the shit out of them, and let people know we should do something about it. Testing "the real thing" sounds fun until the budget for new features skyrockets, because several man-weeks are needed to get decent coverage. Your client will hate it, and your client's clients will hate it even more.
hackandtrip 5y ago
Great article, not the first time I read this argument against the usage of mocks; I never understood though how to solve the problem of the explosion of the path that must be tested (usually exponential). As an example, consider a single API that uses ~3 services, and these 3 services have underneath from 1 to 5 other internal or external dependencies (such as time, an external API service, a DB repository, and so on). How can I test this API, the 3 services, and their underneath dependencies without exponential paths to test - I want to be able to cover all the paths of my code, and ensure that my test only tests a single thing (either the API, the service, or the dependency interface); otherwise, it is not an unit test. I always felt like that these type of tests without mocks works super-nice in nice situations without any external, or even complex but internal, dependency; otherwise, it becomes very very hard to test ONLY what I want, and not all the dependencies underneath. Mocks allow me to stub the behaviour of a service/dependency that I can test in a separate fashion, covering all the paths, and ensuring that each unit test covers a single unit of my code, and not the integration of all my components.
- asdfasgasdgasdg 5y ago
  Your dependencies do need to work reliably if you're not going to use mocks. But if they don't, then mocks might be necessary to ensure test reliability. That said if the interaction between your system and its deps is sufficiently unreliable that you need mocks for test reliability, how are you going to have any confidence in the system's production behavior? Like, if I test one of the workflows of my service, it might involve executing queries and transactions on an underlying database. But these operations have essentially 100% reliability, so the fact that these operations are being "tested" at the same time as my service do not impact test reliability. I gain nothing by mocking them out.
  - lobstrosity420 5y ago
    > I gain nothing by mocking [database queries and transactions] them out. What about the fact that you will need to spin up, migrate and possibly seed a database in your CI pipeline? What about the toll this will take on the execution speed of your test suite? Additionally, consider that you also need to test the behavior of your system when the query fails, and using a mock implementation that always throws an exception is a trivial and reliable way of achieving this. > That said if the interaction between your system and it's deps is sufficiently unreliable that you need mocks for test reliability, how are you going to have any confidence in the system's production behavior? Sometimes your codebase depends on external services which are flaky for reasons beyond your control, just the way it be sometimes. Mocks are useful to ensure the system behaves a certain way when everything goes right as well as when everything goes wrong. Ultimately the article raises many good points about avoiding mocks if it can be helped, but don't forget a test that only tests the happy path of your system is not very useful. Mock an error in that dependency you expect to always work and understand what would happen, make the necessary provisions.
Toine 5y ago
Agreed. Push all external dependencies (DBs, APIs, etc) to the edge. That's it ! Why do you need a functional core though ? You can use decoupled classes and an ioc container that does the wiring for you.
throwaway984393 5y ago
When you build and sell a network security appliance, the appliance has to work properly all the time. You have to give guarantees of how it will perform, guarantee that its functions do what you claim they do, and eliminate all possible bugs. So you have to test it in every configuration possible. So you make models and build automation test frameworks and labs full of prototype equipment so you can run 10,000 tests an hour, 24/7. You automate setting up networks and nodes and passing live traffic to test detectors and rule sets. You use custom hardware to emulate ISP-levels of real traffic. You can't really mock anything. You have to be sure each function will perform as you describe. So most of this testing is end-to-end testing. Using a mock wouldn't tell you if the millions of code paths are working correctly; they'd mostly just tell you if the syntax/arguments/etc of a call were correct. Unit tests are basically one step above evaluating that your code compiles correctly, but it's not testing the code works as expected. End-to-end tests are what you can rely on, because it's real traffic to real devices. That's gonna look different for a lot of your apps, but for web apps that means getting real familiar with synthetic tests and how to test with different browsers. For SDKs it means end-to-end tests for the whole stack of functions, which is a lot more testing than you may have expected. For APIs it means getting the consumers of your APIs involved in end-to-end testing. It also means "testing in production", in the sense of spinning up new production deployments and using those as your end-to-end testing targets (rather than having a dedicated "dev/test/cert" environment which is always deviating from production). This can take a significant effort to adapt in legacy systems, but for Greenfield IaC projects is not very hard to set up.
mpweiher 5y ago
Related: Why I don't mock (2014). https://blog.metaobject.com/2014/05/why-i-don-mock.html https://blog.metaobject.com/2014/05/why-i-don-mock.html
larsrc 5y ago
So... in order to do unit testing, you should shift your entire system to be written in functional style, and then write integration tests instead of unit tests? Because that's what you get when you use the real components.

151 points by feross 5y ago

So TLDR - write integration tests as much as possible and only mock external dependencies like network calls?

Amen to the idea. - Prefer real objects, or fakes over mocks. It will make your tests usually more robust. - Use mocks when you must: to avoid networking, or other flaky things such as storage. - Use mocks for “output only objects”, for example listeners, or when verifying the output for some logging. (But, prefer a good fake) - Use mocks when you “need to get shit done”, it’s the easiest way to add tests in an area that has almost none, and the code is not designed to be easily testable. But remember this is tech debt, and try to migrate towards real objects over time. That’s my short advice I told many times. So might as well comment with it here.

tomcam 5y ago
It all sounds great. I agree totally in principle! I am finding that testing my fairly small Go project (static site generator, because the world definitely needs another one) chews up massive amounts of time. So I tend to avoid the testing pass for longer than I should. Any thoughts on that issue?
- chii 5y ago
  if you don't mind spending the effort, have a quick profile to see where the tests are taking a long time - is it initializing the real components that could be mocked? Or are the tests themselves taking a long time because of other factors, such as IO etc? For IO, may be there should be an abstraction over these IO api and you use an inmemory option instead for testing.
  - bluGill 5y ago
    Of io is the problem I general solve that by testing smaller data sets. I have not found io to the local disk is slow. Of course if it is network io that is bad, but local disks are fine for the size of data in my tests
- vlovich123 5y ago
  Make it CI’s problem. You should only really be running tests regularly for the component you’re currently working on with CI making sure you don’t have accidental regressions.
- edoceo 5y ago
  I try to write the test first, or at least stub in the test how I think it should work. Or at least enough notes to pick it up "tomorrow". Then I have a reference for usage i can build with that in mind
- jerf 5y ago
  It isn't immediately obvious to me why a small Go static site generator would require "massive amounts of time" to run its tests, so it's hard to answer what you're doing wrong. Are your tests perhaps just too darned big? You don't, in general, need to render 5000 pages of something to test your template doesn't crash or something. It could also just be disk access. Consider trying an in-memory file system, or if you're on linux, look at using /dev/shm which is a RAM disk. It is also possible you've snuck in a quadratic or worse time algorithm. There's nothing fundamental to the problem of a static site generator that would require such algorithms, but speaking from experience it is an environment where it's easy to loop over the return value of one function, which itself loops over something other function (quite likely the same data), which itself loops over the same structure, and it's easy to end up with O(n^3) or O(n^4) without realizing it. It's especially easy to end up with that being a "for each page" type of loop. Static site generation should be O(n), give or take small factors (maybe O(n log n) for some things technically, but at a scale where O(n log n) is practically O(n) anyhow).
- tomcam 5y ago
  Thanks all for super helpful feedback. This was a generous gift of your time and brainpower.
bluGill 5y ago
I have not found storage to be flaky and so I don't mock it. Tmpfile always gives me a unique file, and that is all the fake I need. I don't even look up the various forms of temp file to see which ones don't have a race condition as in practice they never do (if I was writing encryption or other such production code I would, but for a unit test the odds of. A race causing a failure are low enough to ignore)
- epage 5y ago
  At minimum, you need to be able to choose where the files are stored to use temp, I find thtre is still the problem of complicated setup. Also, if you are going for particular semaitics (e.g. cross process interactions), isolating that for testing it more specifically can be helpful. For me, I look for higher level abstractions and mock when possible and don't sweat testing against temp files otherwise. I've had several TDD people jump to wanting to mock each filesystem call. One was for a cross process storage API. I was trying to get them to just have a Backend interface for reading and writing instead, with tests just using an InMemory implementation as a Double.
Tainnor 5y ago
I think the original ideas of mocks (if you go and read Growing Object-Oriented Software, Guided by Tests) had some merit: In that style of TDD, mocks are used to discover (hopefully somewhat stable) interfaces between components, and in theory, it fits with the idea that OOP is about "objects sending messages to each other". I can believe that it's possible to write good systems with this kind of approach. Unfortunately, in practice, mocks are rarely used like that and most "OOP" designs have horrible boundaries and are really not much about message passing anymore. That leads to brittle mocks where you constantly have to change tests when you change implementation details. I have also gravitated away from classic OOP and much more towards the "functional core, imperative shell" concept as outlined in the article (although it's difficult to keep this pattern throughout a codebase, especially if you have team members). In such a system you really rarely need mocks. Agreed that fakes, when you have them, are nicer than mocks, especially when the system to be faked has a large API (i.e. use a redis fake, instead of checking the exact commands you send to redis). However, for some outside systems, writing a fake can be a lot of effort. In such a case, I think it's totally valid to write a "gateway class" that isn't unit tested (you can cover it with integration tests instead) which exposes a nice API (e.g. "storeFile(...)") and then to use mocks of that class in other tests.
- arcticbull 5y ago
  In test cases where you extensively involve mocks, more often than not in my experience, you end up testing that your mocks do the thing you told them to.
  - mikepurvis 5y ago
    Yeah, particularly the case with a lot of "glue" type code that's really just passing stuff back and forth and not really making any decisions with it. I've always struggled to feel that mock-based testing in this scenario was anything other than busy-work.
  - pstuart 5y ago
    Yes, but the coverage numbers look great!
mirekrusin 5y ago
- Prefer simulators over mocks We're running business critical system for several years now, creating simulators has been one of our "secrets" that contributed to the success. Not only our tests are running in as close to production environment as possible, we're also using them for local development where developers can spin up functioning system on own dev machine.
- valenterry 5y ago
  I agree - but it can be hard to get good ones, e.g. if you need to simulate sql query execution.
  - bluGill 5y ago
    Sqlite is a great simulation for SQL. You need to limit your SQL to the subset that us supported by both it and your target database though, which might be a problem.
    - jon-wood 5y ago
      In a world of Docker existing there’s rarely a reason not to just use your target database - I can have a non-production grade Postgres instance up and running in less time than it took me to write this message.
  - withinboredom 5y ago
    We spin up a temporary and local MySQL instance per test run with helpers to generate necessary data on-demand. All our tests therefore use real queries on a real db. Due to speed constraints, this db is shared between all the tests you run in that session, so it’s possible to influence tests after yours runs in CI. The reality is that that is pretty easy to detect and it’s caused us to write less brittle code.
    - mirekrusin 5y ago
      We're using mssql, on bootstrap we're running all migrations, seeding database, then backing it up; each test file restores its own database from backup which is much faster than alternatives; e2e tests run against simulated environment that has single database; e2e are split into concurrent runs with distinct set of tests to speed things up.
collyw 5y ago
Generally the more difficult a test is to write and run, the more useful it is.

mrjin 5y ago

There are also Function Tests and Integration Tests. Seems the author took Unit Tests for all three.

howinteresting 5y ago
No. this is a false di/trichotomy. What matters is the scope of code a test covers. Some tests cover a greater scope, some tests cover less.

vlovich123 5y ago

This is almost right except for two points: For fakes, spin up the real thing. If you’re not able to model your database transactions deterministically, then your transactions could themselves be flawed and tests are great way to catch that. Deterministic tests are not a goal in and of themselves. Controlled non determinism is valuable. This is popularized in various frameworks under the names of property checking and fuzzing which will let you know the seed to use for the failure for example so that while the runs don’t have the exact same input/output for every invocation, you get better coverage of your test space and can revisit the problematic points at any time. If you’re doing numeric simulation, make sure you are using a PRNG that’s seedable and that you log the seed at the start if you’re using a seed (and make sure time is an input parameter). Why is this technique valuable? You transitively get increasing code coverage for free through CI/coworkers running the tests AND you have a way to investigate issues sanely.

blt 5y ago
"Non-deterministic tests" usually refers to a test whose output depends on the execution environment in a way that cannot be controlled. For example, a multithreaded test with a race condition, or a test that uses the real-time clock. This is fundamentally different from a test that uses random numbers. Another way to look at it: Deterministic tests can be used in `git blame`. A unit test that uses a specific PRNG algorithm and sets the seed is fine. A test with a race condition is not. Fuzzing is very useful, but it's not unit testing. If you want to run a fuzzer or some other kind of endless randomized testing in CI, it should be a separate job from the unit tests (IMO).

TravHatesMe 5y ago

In other words, shouldn't we prefer integration testing, or perhaps partial integration testing. This requires an initial effort of setting up your test framework/environment, but in my experience integration tests provide good value for the time you put into testing. Rather than a granular test on a single class, test the orchestration of many classes. You end up hitting big % of code. As always, depends on the project. If we're building a rocket ship, you need both granular testing and coarse testing.

Tainnor 5y ago
People keep saying this all the time, but apart from the fact that nobody can agree on what an "integration test" is (because there's almost always some part of the application flow that you're stubbing out), it just becomes immediately apparent in a code base of sufficient size that "just use integration tests for everything" is only possible if you severely under-test (which usually includes thing like not properly testing for error conditions etc.).
- code_biologist 5y ago
  What? Nobody is advocating for integration tests to the exclusion of unit tests. but apart from the fact that nobody can agree on what an "integration test" is Not being precise doesn't invalidate a guideline. The ideas that "stubbing less is better" and "testing functionality end to end is good bang-for-buck" aren't crappy because people don't agree on the details. it just becomes immediately apparent in a code base of sufficient size that "just use integration tests for everything" is only possible if you severely under-test Your parent comment said "If we're building a rocket ship, you need both granular testing and coarse testing." Nobody is advocating for integration tests to the exclusion of unit tests. If you write a hash table or a CSV parser, yes you should unit test it. But for most application-ish functionality you should reach for integration tests first. For example, testing direct message functionality in an app, checking that after a send there's a notification email queued and the recipient inbox endpoint says there's 1 unread will get you really far in 20 lines of code. Is it exhaustive? Of course not. But the simplicity is a huge virtue. if you severely under-test (which usually includes thing like not properly testing for error conditions etc.) I advocate "default to integration testing for application functionality". Those focused on unit tests often mock exactly the things most likely to break: integration points between systems. "Unit" tests of systems are often really verbose, prescriptive about internal state, and worst of all don't catch the bits that actually break.

mbfg 5y ago

the thing i see that makes this test brittle is the use of relative dates and durations that break across day light savings time changes

jetru 5y ago

inb4 someone saying something about mocking or dependency injection for testing

sedro 5y ago
now you have N+2 problems?

eatonphil 5y ago

In database land you can even freely bring up a number of _proprietary_ databases without needing an account or license. I spin up MySQL, Postgres, Oracle and SQL Server databases in containers in Github Actions for tests. Unfortunately IBM DB2 seems to require a license key to spin up their container. :(

kmac_ 5y ago

I have used this approach and now I perceive mocks as a bad code smell. It's worth noting that long and complicated flows need to be wrapped in sagas (which are just imperative shells) and things can get not so easy. I would love to hear what are other clean alternatives in such cases. Still, "functional core, imperative shell" is the way to go, code really fits in the head and tests actually make sense.

Supermancho 5y ago

The heart of the argument presented is that using mocks in unit tests is problematic because if an interface is changed, that will possibly break every test that involves mocking that interface and that's friction to making changes. This is silly. If you use real implementations and change the interface, you have the exact same problem except that your configuration/setup is also likely to have to change in subtle ways. Let's just assume the absurd notion that a choice of creating tech debt or changing a common interface is a real choice. If there was a serious change that could not be accounted for with easy test changes, I'm not the only one to see the tests commented out with a "TODO: fix these". Developers tend to be pragmatic. If you want to change code you will always measure the effect it will have on the code and tests are incidental, not the primary concern. Make it work. Make it right. Make it fast. I want to be able to trigger all code paths and exceptions (make it good). Using a real dependency, I would be left in the unfortunate situation of depending on knowledge of the internals of that dependency. It may not allow me to execute specific paths via pure configuration at all. I don't think using real dependencies is a good idea at all for unit tests. Integration tests are a different and I do fear that they are being confused.

spawarotti 5y ago
> The heart of the argument presented is that using mocks in unit tests is problematic (...) If you use real implementations No. As you say, the moment you have a real out-of-process dependency, you no longer have a unit test, but an "integration-with-external world" test. The heart of the argument is to not mock out internal (in-process) business logic. Instead: a) do not use mocks (e.g. leveraging Moq); use fakes (aka "simulators"), i.e. proper classes having in-memory implementations of the out-of-process dependencies and crucially b) replace only out-of-process dependencies, which usually are flaky/nondeterministic. Never replace internal business logic. If you feel you need to replace internal business logic, you are not following the functional core architectural pattern. After you refactor to functional core you will no longer need to replace any dependencies in your unit test, as there won't be any - you will just call your tested pure method and make assertions on the returned value. Special case here is testing "top-level/controller" logic. Here you need to use the in-memory fakes, but there will be only few, reused across all tests, and such test will be an "end-to-end internal-business-logic integration test", but still it will have all the properties of a unit test - it will be fast, deterministic, and you will be able to run it as part of a unit test suite out-of-the-box, with no environment setup necessary. Further reading: https://enterprisecraftsmanship.com/posts/when-to-mock/ https://enterprisecraftsmanship.com/posts/when-to-mock/ > If you want to change code you will always measure the effect it will have on the code and tests are incidental, not the primary concern Exactly. Mocks are very brittle and break all the time, causing unnecessary rework. If you instead rely on a small set of fakes of out-of-process dependencies, you will drastically reduce test suite rework and improve signal-to-noise ratio. > Using a real dependency, I would be left in the unfortunate situation of depending on knowledge of the internals of that dependency. And what happens if the real dependency changes behavior, but you forget to update your mock? You will have a test that will be running against a mock that simulates obsolete behavior, no longer present in production. In worst case scenario the test will be green/passing, while there is a bug in production. Will you remember to always comb over your entire test suite and review all mocks to faithfully simulate actual production behavior? > It may not allow me to execute specific paths via pure configuration at all. If you write fakes, you will have full control over how to configure them.

hansvm 5y ago

As a potential counter-argument, the use of mocks can enable testing of functionality that the current concrete implementation doesn't exercise. It's easier than one would think to accidentally rely on implementation details rather than coding just to the interface (and optionally any documented restrictions to that interface). They explicitly call out clocks as a source of non-determinism that probably should be mocked, but I'll re-use them as an example anyway because everyone is familiar with them: it's extraordinarily useful for the tests to execute nearly immediately rather than actually waiting on a clock, and rare behavior like a clock running backward, two consecutive timings being identical within the clock's resolution, or whatever other weird artifacts that your code should handle are definitely better explicitly tested rather than not mocking the clock. Other domain-specific interfaces are often similarly able to exhibit a weird edge case that ought to be explicitly tested (rather than accidentally relying on a "nice" implementation) if you really want to unit test the callers and not integration test the coupled system.

TheReveller 5y ago
Also error conditions - telling a fake to give you a specific error before you run the test is no different from telling a mock to do it.
- marcosdumay 5y ago
  It is different. You tell the mock to give you an specific error with specific information, while you tell the fake to give you the error for an specific condition. It is subtle and may not be worth the time, but again, it may be.
smaudet 5y ago
I think you have missed the point of fakes versus mocks (as this article puts it) based on what you said - fakes should be able to do everything mocks can, more, even. But I hear you complain "but that's so much work to maintain!!" - mocks are more work to test less, if you need a better fake often, write a library which has them. What you are referring to is known as fuzzing, not unit testing, and fuzzing has a better track record than straight up "testing" generally for finding issues. Fakes can have deterministic or indeterminate behavior, the case the OP makes is for deterministic "unit" testing with more real behaviors than not. This doesn't preclude good fuzzing... "It's easier than one would think to accidentally rely on implementation details" To be somewhat philosophical, what if that 'implementation detail' is the feature? No amount of testing will save you from incompatible feature sets, to 'enable testing of functionality that the current concrete implementation doesn't exercise' you run the very real risk of baking in incompatible sets of features into the core of your app. You shouldn't be designing your app at the unit level...

hermanradtke 5y ago

I first learned this concept from Gary Bernhardt’s video Boundaries. https://www.destroyallsoftware.com/talks/boundaries https://www.destroyallsoftware.com/talks/boundaries

cassac 5y ago

I don’t get it. When I unit test, I want to only test what I’m testing. A dependency or the result of a dependency is not what I’m testing. So I mock the result of that dependency to unlink the test from the dependency. If an interface changes I generally want to know anyway, as the test may be different or even obsolete depending on the change.

crdrost 5y ago
I hear you, and that's how things work at my present shop. However, let me give a strong defense of the point. It turns out that the only thing you can test is a pure function. This is just the nature of a test, we pin down the inputs to a piece of code and we see what outputs it produces. All of the mocking that you are doing is an attempt to turn an impure function into a pure function. Even more extreme setups where you connect your container to a container running postgres, is trying to turn that container into a pure function in a different way. You don't need to do any of this if the function is pure in the first place. Once we've established that “functional core” is a lazier means to the same ends the question of dependencies still comes up, and the “should I mock dependencies” question becomes “should I promote this internal function call to the main I/O section and pass in its result as an argument?”... And the answer is that that changes the language with which this outermost level is written. And probably this outermost shell should be written to sound like the business logic that you are implementing, and you should never do that. If that's the logic then it means that the sort of testing you're doing is extremely particular and fussy. Because it suggests that every test should really be constructed at a business level, it is a story about your product that you want to make sure holds fast even when the internals are changed. This works really well with domain driven design, because that says that your modules should be also business level entities, so each module comes with some tests that say, here's what this sort of person interacts with the system like. So you are always testing integration of the pure functions, and why would you not. If those pure functions do not integrate together, you want to know about it, and you do not want to know about it through a persnickety test which just fixes the inputs and outputs of something that has no business relevance, because you know what happens in those cases: the developer just rewrites the test to say the opposite of what it used to say, so that it passes now. There is no semantic check on the test output because there cannot be if it is scoped too small. I think you can quibble a lot with those details but I think that's the strongest case you can make for it?

beiller 5y ago

I prefer mocks over real services because I onboard many people to a repo. Too many services attached to unit tests turns into a fatigue until eventually no one runs the tests any more I found.

spawarotti 5y ago

This article espouses what is known as "classicist testing" school of thought and rejects "mockist/London-style testing". I wholeheartedly agree with it. I have been championing it in my team of 20+ devs since many years now, to great effect. You can read more about it in the excellent book by Vladimir Khorikov from January 2020 titled "Unit Testing Principles, Practices, and Patterns".

sour-taste 5y ago

I have two services that communicate over the network. Neither does anything useful without data from the other. So I write my tests like this article suggests and inject at the boundaries of the services, the network boundary in this case. I've built tests that now only test my idea of what the services will return. IMO these aren't particularly useful tests: what happens when the remote service starts returning unexpected data or errors due load problems? I guess my point is just this article is good advice: prefer testing your actual code and dependencies, but unit testing isn't a replacement for integration testing.

kromem 5y ago

If you are testing the externalities, they aren't unit tests, but integration tests. The better advice is: continue to isolate unit tests away from real dependencies, and ALSO have integration tests that test the way the package connects to dependencies (and the other packages in the software dependent on it).

wtetzner 5y ago
I would argue that if you need mocks, the code isn’t well suited to unit testing and you’re better off with integration tests. If you want to unit test it, then refactor in such a way that you no longer need mocks.

quickthrower2 5y ago

I worked somewhere with a hard rule that you are not allowed to test internal classes. So if I write my own sort algorithm I am not allowed to unit test that class unless I make it public. But if the sort algorithm is to solve a specific sub problem in a library there is no need to make it public - it may not make much sense. So I had to test it “through” its consumer class(es) that is public. For illustration sake lets say a MVC controller mocked up to the eyeballs with ORM mocks, logging mocks etc. This bugged me because having direct access to a functional core I can quickly amplify the number of test cases against it and find hidden potential bugs much quicker.

omegalulw 5y ago
I would say that testing only the public contract is generally very helpful. When the logic gets super complex, I would think that a good comprise is to modularize your code - and test the public API of each module. A good heuristic is to see if that modules are (or could be) helpful by themselves in the future. You then write tests for a module assuming it's dependents cover their edge cases.
- quickthrower2 5y ago
  As I wrote it I thought the same thing! I think some of the problem was org friction of adding a new module, getting approval to do so. This was a place with about 400 devs so they could really let anyone freely add modules and waiting for architecture approval would take too long for a typical ticket.
Supermancho 5y ago
> So if I write my own sort algorithm I am not allowed to unit test that class unless I make it public. I think there's a slow movement away from this kind of straightjacket. The compiler sees it all anyway. Class accessors serve 2 purposes. 1. They are a social tool; and this is only if you believe that developers who work on source code, which they can read, cannot be trusted to call methods to do what they need. 2. They are a convenience for opaque modules, so that users who may not have access to the source code, can avoid using APIs that may have no effect or problematic side effects, while also decluttering the API. Python allows for methods to act as if they are declared public when using a special syntax eg _backdoor(). Go allows for any method that is in the same package, to access any other method regardless of accessors...like tests for that package. In javascript, you can use rewire.js to mock methods in imported modules (which are effectively methods in closures). These solutions are an improvement to the classic rigid class access pattern that many languages are stuck with and forces the kind of theatre you have experienced.

daxfohl 5y ago

I 100% agree, but rarely have found much to put in a functional core. Most everything these days is distributed microservices talking to each other (different argument) and having more than a couple lines in a row that don't call some other service, even in a well-factored app, is almost a cause for celebration.

kjgkjhfkjf 5y ago

I agree that fakes are very good for testing, with one caveat: the fakes must be owned by the same folks that own the real implementation. When fakes are owned by anyone else, e.g. the folks writing the system under test, there's a high risk that the semantics of the fake and the real implementation will diverge, rendering the tests that use the fakes much less useful.

exdsq 5y ago

I’ve spent the last few years working on test infrastructure for blockchain projects so this question is regularly on my mind, and I’ve found the best solution is to just include the dependencies. Compute and storage is cheap enough that you can spin up these external tools and the extra level of assurance makes me sleep easier, and while flakey tests are theoretically an issue I rarely find them (and they tend to be programmatically fixable). If a unit test requires an external dependency then just use an integration test (or check it’s covered by system tests) and leave it at that.

swiftcoder 5y ago

I see another soul has rediscovered the universal truth that unit testing is only pleasant in functional programs

Finally! Somebody that has the guts to say the truth.

globular-toast 5y ago

Ian Cooper talked about this (and more) 5 years ago: https://www.youtube.com/watch?v=EZ05e7EMOLM https://www.youtube.com/watch?v=EZ05e7EMOLM Well worth listening to this talk. It completely changed the way I did testing.

mkl95 5y ago

If some method has several dependencies that are tangentially related with what I'm testing, I'm going to mock the shit out of them, and let people know we should do something about it. Testing "the real thing" sounds fun until the budget for new features skyrockets, because several man-weeks are needed to get decent coverage. Your client will hate it, and your client's clients will hate it even more.

hackandtrip 5y ago

Great article, not the first time I read this argument against the usage of mocks; I never understood though how to solve the problem of the explosion of the path that must be tested (usually exponential). As an example, consider a single API that uses ~3 services, and these 3 services have underneath from 1 to 5 other internal or external dependencies (such as time, an external API service, a DB repository, and so on). How can I test this API, the 3 services, and their underneath dependencies without exponential paths to test - I want to be able to cover all the paths of my code, and ensure that my test only tests a single thing (either the API, the service, or the dependency interface); otherwise, it is not an unit test. I always felt like that these type of tests without mocks works super-nice in nice situations without any external, or even complex but internal, dependency; otherwise, it becomes very very hard to test ONLY what I want, and not all the dependencies underneath. Mocks allow me to stub the behaviour of a service/dependency that I can test in a separate fashion, covering all the paths, and ensuring that each unit test covers a single unit of my code, and not the integration of all my components.

asdfasgasdgasdg 5y ago
Your dependencies do need to work reliably if you're not going to use mocks. But if they don't, then mocks might be necessary to ensure test reliability. That said if the interaction between your system and its deps is sufficiently unreliable that you need mocks for test reliability, how are you going to have any confidence in the system's production behavior? Like, if I test one of the workflows of my service, it might involve executing queries and transactions on an underlying database. But these operations have essentially 100% reliability, so the fact that these operations are being "tested" at the same time as my service do not impact test reliability. I gain nothing by mocking them out.
- lobstrosity420 5y ago
  > I gain nothing by mocking [database queries and transactions] them out. What about the fact that you will need to spin up, migrate and possibly seed a database in your CI pipeline? What about the toll this will take on the execution speed of your test suite? Additionally, consider that you also need to test the behavior of your system when the query fails, and using a mock implementation that always throws an exception is a trivial and reliable way of achieving this. > That said if the interaction between your system and it's deps is sufficiently unreliable that you need mocks for test reliability, how are you going to have any confidence in the system's production behavior? Sometimes your codebase depends on external services which are flaky for reasons beyond your control, just the way it be sometimes. Mocks are useful to ensure the system behaves a certain way when everything goes right as well as when everything goes wrong. Ultimately the article raises many good points about avoiding mocks if it can be helped, but don't forget a test that only tests the happy path of your system is not very useful. Mock an error in that dependency you expect to always work and understand what would happen, make the necessary provisions.

Toine 5y ago

Agreed. Push all external dependencies (DBs, APIs, etc) to the edge. That's it ! Why do you need a functional core though ? You can use decoupled classes and an ioc container that does the wiring for you.

throwaway984393 5y ago

When you build and sell a network security appliance, the appliance has to work properly all the time. You have to give guarantees of how it will perform, guarantee that its functions do what you claim they do, and eliminate all possible bugs. So you have to test it in every configuration possible. So you make models and build automation test frameworks and labs full of prototype equipment so you can run 10,000 tests an hour, 24/7. You automate setting up networks and nodes and passing live traffic to test detectors and rule sets. You use custom hardware to emulate ISP-levels of real traffic. You can't really mock anything. You have to be sure each function will perform as you describe. So most of this testing is end-to-end testing. Using a mock wouldn't tell you if the millions of code paths are working correctly; they'd mostly just tell you if the syntax/arguments/etc of a call were correct. Unit tests are basically one step above evaluating that your code compiles correctly, but it's not testing the code works as expected. End-to-end tests are what you can rely on, because it's real traffic to real devices. That's gonna look different for a lot of your apps, but for web apps that means getting real familiar with synthetic tests and how to test with different browsers. For SDKs it means end-to-end tests for the whole stack of functions, which is a lot more testing than you may have expected. For APIs it means getting the consumers of your APIs involved in end-to-end testing. It also means "testing in production", in the sense of spinning up new production deployments and using those as your end-to-end testing targets (rather than having a dedicated "dev/test/cert" environment which is always deviating from production). This can take a significant effort to adapt in legacy systems, but for Greenfield IaC projects is not very hard to set up.

mpweiher 5y ago

Related: Why I don't mock (2014). https://blog.metaobject.com/2014/05/why-i-don-mock.html https://blog.metaobject.com/2014/05/why-i-don-mock.html

larsrc 5y ago

So... in order to do unit testing, you should shift your entire system to be written in functional style, and then write integration tests instead of unit tests? Because that's what you get when you use the real components.