15 ms·

How to test without mocking

181 points by alexzeitler 2y ago

tail_exchange 2y ago
The problem is when IOService has edge cases. When building the mock, does it address the edge cases? When you want to find bugs by testing, the tests need to test the real world. So to work, the mock for IOService needs to model the edge cases of IOService. Does it? Does the developer know they need to model the edge cases, or the mock is not helping with finding bugs? Do you even know the edge cases of IOService? When IOService is a database service, does your mock work for records that do not exist? Or return more than one record? It depends. Mocks are used to remove variables from the experiment you are running (the tests) and see if it behaves under very specific conditions. If you want to test how the code behaves when a specific row is returned by the database, but instead the dependency returns something else, then you are not testing that use case anymore. Reproducibility also has its values. But yes, you can definitely make your mocks return errors and fail in a myriad of ways. Not to say you should mock everything. Of course having proper integration tests is also important, but articles like these will rarely tell you to have a good balance between them, and will instead tell you that something is correct and something else is wrong. You should do what makes sense for that specific case and exercise your abilities to make the right choice, and not blindly follow instructions you read in a blog post.
- sisve 2y ago
  100% agree. Maybe you should be the amazing cto. Context always matters.
- watermelon0 2y ago
  I totally agree, there is a balance between what makes sense to mock, and what needs proper integration tests. Additionally, just using integration tests does not guarantee that edge cases are covered, and you can just as easily write integration tests for happy path, without thinking about the rest.
dboreham 2y ago
I've been doing this stuff (software) for a very long time and if it hadn't been invented by others, I'd never have thought of Mocking. It's that stupid of an idea. When I first came across it used in anger in a large project it took me a while to get my head around what was going on. When the penny dropped I remember a feeling of doom, like I had realized I was in The Matrix. Don't work there, and don't work with mock-people any more.
- ams92 2y ago
  With containerization it’s very quick to spin up test dependencies as well as part of your CICD. Why mock calls to a datastore when it’s super easy to spin up an ephemeral postgresql instance to test on?
  - randomdata 2y ago
    > Why mock calls to a datastore when it’s super easy to spin up an ephemeral postgresql instance to test on? It's actually super hard to get Postgres to fail, which is what you will be most interested in testing. Granted, you would probably use stubbing for that instead.
  - UK-AL 2y ago
    Because you have 40000 tests, and an in memory object means they can run in seconds. And the real thing runs in minutes.
  - specialist 2y ago
    Yup. Section "Easier to Test IO" in TFA.
- sampullman 2y ago
  I don't like mocking either, but there are periodically situations where I've found it useful. Sometimes there is a complex system (whether of your own design or not) that isn't amenable to integration/e2e testing, and the interesting parts can't be easily unit tested due to external or tightly coupled dependencies. Of course you can always pick it apart and refactor so it can be unit tested, but sometimes the effort required makes mocking look pretty appealing.
bitcoin_anon 2y ago
Your mock object is a joke; that object is mocking you. For needing it. ~ Rich Hickey
- alexfromapex 2y ago
  Don’t really care about advice from the guy who invented an incredibly non pragmatic programming language. I also honestly had to look up who he was. So his sage advice hasn’t brought him much fame.
KaiserPro 2y ago
The problem is that mocks are normally used to avoid testing services/outside code. For example making a wrapper around a message system if you don't mock, it tests both your code and the message system. However the the overhead of keeping the mocking system up to date is a pain in the balls.
- devit 2y ago
  Testing both your code and the message system is exactly what you want, since if the message system is broken in a way that upstream didn't catch, you want to learn about it during testing and not production, if possible.
  - jon_richards 2y ago
    I’m still mad about the time I was told to mock a payment gateway in tests even though they had a testing environment and then got a slew of bug reports from people whose company names had punctuation (and thus failed the name validation the payment gateway was secretly running).
    - UK-Al05 2y ago
      Could you just run some behaviour tests against the gateway? You can get the best of both.
    - mehagar 2y ago
      That reminds me that Stripe actually maintains (or used to) their own mock for their Ruby package. This puts the burden on maintaining the mock on the library owner, where it is more likely that they would implement the mock correctly, edge cases and all.
    - mvdtnz 2y ago
      You should be unit testing components that interact with the payment gateway. This could involve dozens of even hundreds of tests, where the gateway should be mocked. These tests should be fast and reliable. In addition, you should have a small suite of integration/E2E tests against a real test instance of the gateway. These tests may be slow and unreliable (because of the real network involved) but catch those hairy issues you failed to find in unit tests. Also, when your integration suite (or customer reports) discovers that the payment gateway fails on punctuation, another unit test should be added with a mock that responds the same way, and an E2E test added with punctuation in the input data and a failure expectation. What makes you so certain you would have included punctuation in the input data if the test had not mocked the gateway?
christkv 2y ago
The golden rule was to only mock your own code. Make a facade around the framework class using an interface and mock that if needed to decouple your tests. Then write integration tests against your implementation of the interface. The moment you mock other people’s code you have brittle tests.
graypegg 2y ago
Tests are a tool for you, the developer. They have good effects for other people, but developers are the people that directly interact with them. When something fails, it's a developer that has to figure out what change they wrote introduced a regression. They're just tools, not some magic incantation that protects you from bugs. I think the author might be conflating good tests with good enough tests. If IOService is handled by a different team, I expect them to assure IOService behaves how it should, probably using tests. The reason we're mocking IOService is because it's a variable that I can remove, that makes the errors I get from a test run MUCH easier to read. We're just looking at the logic in one module/class/method/function. It's less conceptually good to mock things in tests, since I'm not testing the entire app that we actually ship, but full no-mocks E2E tests are harder to write and interpret when something goes wrong. I think that makes them a less useful tool. The thing I do agree on, is assuming your mocks should only model the happy path. I'd say if something can throw an exception, you should at least include that in a mock. (as a stubbed method that always throws) but making the burden of reimplementing your dependancies mandatory, or relying on them in tests is going to mean you write less tests, and get worse failure messages. Like everything, it depends eh?
- specialist 2y ago
  With I/O in general, I've observed that socket, protocol, and serialization logic are often tightly coupled. If they're decoupled, there's no need to mock protocol or serialization. There's some cliché wrt "don't call me, I'll call you" as advice how to flip the call stack. Sorry, no example handy (on mobile). But the gist is to avoid nested calls, flattening the code paths. Less like a Russian doll, more like a Lego instructions. In defense of mocks, IoC frameworks like Spring pretty much necessitate doing the wrong thing.
- deleted 2y ago
  [deleted]
- campbel 2y ago
  This 100%. I'm not sure how the author managed to create consistent failure cases using real service dependencies, but in my code I find mocks to be the easiest way to test error scenarios.
worksonmine 2y ago
If I already have passing tests for anything function A might do, I can safely assume it will behave the same when called from B, C and D.
- t43562 2y ago
  In some languages A might free a memory allocation e.g. after communicating with some server. If B also frees that memory then there is a bug. Presumably this means B's tests are wrong/incomplete. If B was mocking A to avoid the IO, you might not find out.
solatic 2y ago
If you're writing a CRUD app and mocking your database calls instead of just starting an actual Postgres instance before running the tests, you're probably using mocking wrong. If you're writing a custom frontend for GitHub using the GitHub API and don't bother writing a decent set of mocks for how you expect the GitHub API to behave, your app will quickly require either full manual QA at best or become untestable at worst. Some APIs are very stable, and testing against the API itself can hit rate limiting, bans, and other anti-abuse mechanisms that introduce all kinds of instability to your test suite. Use the right tools to solve your problems.
- Fire-Dragon-DoL 2y ago
  100% agree
- arcticbull 2y ago
  When you write tests with mocks you almost always at some point end up with tests that test your mocks lol, and tests that test that you wrote the tests you think you wrote -- not the software itself. I’ve never been thrilled by tests that rely on mocking — it usually means you need to re-express your module interface boundary. Mocks for me fall into the class of software I affectionately call “load-bearing paint.” It’s basically universally the wrong tool for any given job but that really doesn’t stop people. Putting in a data class or similar model object and a delegate is usually sufficient and a much better tool.
  - mehagar 2y ago
    I agree that if you need to write mocks, it's likely that your interfaces are poorly defined. This is one of the claimed benefits of test driven development - writing the tests first forces you to design the code in a way that cleanly separates modules so they can be tested.
    - UK-Al05 2y ago
      You have to mock/fake when modules call dependencies. Your way means you only ever have siblings. With an orchestrator pulling results out of one module and pushing it into another.
whoisthemachine 2y ago
> Modelling the happy path is great for refactoring - even a necessity, but doesn’t help with finding bugs. This is a common misconception (one that I also initially held). Unit tests aren't meant to find bugs, they're meant to protect against regressions, and in doing so, act as a documentation of how a component is supposed to behave in response to different input.
- UK-AL 2y ago
  They can do both.
- postalrat 2y ago
  I think writing tests as a form of documentation is a waste of time. If I'm using a component I don't want to read unit tests to figure out what it should do. Unit tests are most often used to cover a few more lines that need coverage. That's the value they provide.
  - Izkata 2y ago
    Documentation for other developers of that code, not users of that code.
    - deleted 2y ago
      [deleted]
    - curun1r 2y ago
      Also worth noting that "other developers" includes the original developer when they return to the code several months later.
  - randomdata 2y ago
    A well designed API will generally allow users to understand usage without any additional documentation, sure. However, those who modify the API in the future will want to know every last detail that you knew when you were writing it originally. That must be documented to ensure that they don't get something wrong and break things – and for their general sanity. That is, unless you hate future developers for some reason. You could do it in Word instead, I suppose, but if you write it in code then a computer can validate that the documentation you wrote is true. That brings tremendous value.
marvstazar 2y ago
If you inherited a project with no tests at all, mocking is a lifesaver. It allows you to only worry about specific aspects of the application so you can start writing and running tests. I agree though that if not done properly, it can be overused and can make your tests practically worthless.
john-radio 2y ago
[flagged]
mehagar 2y ago
The catch-22 with refactoring to be able to write unit tests is that refactoring introduces risk as you are changing code, and you need tests to help reduce that risk. But you can't easily write tests without refactoring. This has been a very difficult problem for the team I'm currently on. The only strategy I'm aware of is described in `Working Effectively With Legacy Code`, where you start by writing throwaway unit or E2E tests that give you "cover" for being able to refactor. These tests depend on the implementation or may use mocking just to get started. Then you refactor, and write better unit tests. Then get rid of the throwaway tests.
- t43562 2y ago
  Why get rid of working e2e tests? IMO they are more useful than unit tests at finding the kinds of problems that stop a release/deployment. You can attack from both directions: e2e tests make sure that certain processes work in fairly ordinary situations, then look for little things that you can unit test without huge refactoring. When you've pushed these as far as you can, section off some area and start refactoring it. Do your best to limit your refactoring to single aspects or areas so that you are never biting off more than you can chew. Don't expect everything to become wonderful in one PR. Your e2e tests will catch some errors and when you look at what those commonly are then you can see how to best improve your tests to catch them earlier and save yourself time. In python I had stupid errors often - syntax errors in try-catch blocks or other things like that. If I used a linter first then I caught many of those errors very quickly. I was working on a build system so I mocked the build - created a much simpler and shorter build - so I could catch dumb errors fast, before I ran the longer e2e test on the full build. IMO you need to progress to your vision but trying to reach it in one step is very dangerous. Make life better piece by piece. You can even do PRs where you only add comments to the existing files and classes (not too much detail but answering questions like "why" is this file/class here). This helps to make sure you really understand the current system is doing before you change it. I once added type hints everywhere to a legacy python program - it wasn't as helpful as I'd hoped but it did prevent some issues while I was refactoring.
UK-AL 2y ago
In general I have fake IO object and a real IO object. Then run the same bunch of tests against them to make sure behaviour matches. You have verified your mock has the same behaviour as the real thing. I then run unit tests against the fake io object. I don't mock internals, only boundaries. If for whatever reason i want to test it against the real db i can simply swap out the fake for the real object.
languagehacker 2y ago
I don't think mocking is an anti-pattern. Using only unit tests and then mocking everything probably is. Mocks have a perfectly viable place in testing. They help establish boundaries and avoid side effects that are not pertinent to the logic being tested. I would reference the testing pyramid when thinking about where to be spending time in unit tests vs. integration tests vs. end to end tests. What introduces risk is if we're mocking behaviors that aren't being tested further up the pyramid.
- mehagar 2y ago
  I like the testing pyramid specifically because it captures the tradeoffs between the different kinds of tests. Mocks can come in handy, but like anything else can be abused. We need a "Mock this, not that" kind of guide.
- hot_gril 2y ago
  I think the better advice that goes with the spirit of the article is, prioritize integration testing over unit testing if you're constrained on time.
jasonlotito 2y ago
Go ahead. Don't mock that external service that you rely on for an API. Now you need to have multiple keys, one for each developer, or share keys separate from various environments? Does it not offer dev/test/staging/prod keys? Well, now you need to share those keys. Does it only offer Prod keys? Now you are stuck sharing that. API request limits? Now you are eating through that just to run tests. And let's not forget that testing things locally means you are mocking the network, or lack-thereof. "Mocking is an anti-pattern" is a great sentiment if you ignore costs or restrictions in the real world.
- bottlepalm 2y ago
  Also the intermittent failures of your tests relying on unstable dependencies.
  - t43562 2y ago
    If your dependencies are unstable then that is very important to know! If it means you have to add forms of resilience then that's good for your code perhaps?
    - bottlepalm 2y ago
      The point is it makes you tests nondeterministic which is a huge problem with large fluid teams and thousands of tests. They need to be deterministic.
- t43562 2y ago
  That is a fairly good reason for trying to use external systems/tools that make testing easy/cheap to do. So a good approach would be to have tests where you can run with the mock and then run the same tests with the real system. Anything you catch with the mock saves you from using the costly system but you still get real testing.
time0ut 2y ago
Maybe I am missing something, but how else would I test various exception handling paths? There is a whole world of errors that can occur during IO. What happens if I get a 500 from that web service call? How does my code handle a timeout? What if the file isn't found? It is often only possible to simulate these scenarios using a mock or similar. These are also code paths you really want to understand.
- lonelyasacloud 2y ago
  I don't think there's a disagreement; the author states "Whenever I look at mocks, they mostly have the same problem as all unit tests that I see, they only model the happy path". So by corollary their opinion of the correct usage of mocking would also include modelling brokenness.
- dgb23 2y ago
  Put a small data interface around your IO, have it return DATA | NOT_FOUND etc. Then your tests don't need behavioral mocks or DI, they just need the different shapes of data and you test your own code instead of whatever your IO dependency is or some simulation thereof.
  - myfavoritetings 2y ago
    isn't that just mocking with extra steps?
    - dgb23 2y ago
      You could call the data you generate for the tests "mocks". But they really aren't "mocks" in the sense of behavioral mocks via IoC/DI and you don't need to manipulate them via some kind of interface in order to put them into the right state for your particular tests. There are some extra steps, but you get extremely simple and reliable tests in return. In many(!) cases you already have a data interface, especially with HTTP/REST APIs. All you need to do is simply not bury the IO call down the stack and maybe describe the failure conditions as plain data in your signature and voila. (This is not a replacement for higher order testing like, manual, E2E or integration tests. But it certainly beats unit testing with mocks IMO.)
cryptonector 2y ago
Mocking is useful for testing small parts of your programs/libraries. For full-scale testing you really need to not emulate because any emulation will be woefully incomplete, so you're going to have to spin up a virtual network with all the services you need including DNS.
skybrian 2y ago
Mocks can be useful when there is a standard protocol and you want to document and test that your code follows the protocol exactly, doing the same steps, independently of whether some other component also follows the protocol. It tests something different from whether or not two components work together after you change both of them. It takes time to come up with good protocols that will remain stable and it might not be worth the effort to test it when the protocol design is new and still in flux, and you don’t have alternative implementations anyway. This is often the case for two internal modules in the same system. If you ever want to change the interface, you can change both of them, so an integration test will be a better way to ensure that functionality survives protocol changes. Database access tends to be a bad thing to mock because the interface is very wide: “you can run any SQL transaction here.” You don’t want to make changing the SQL harder to do. Any equivalent SQL transaction should be allowed if it reads or writes the same data. Compare with testing serialization: do you want to make sure the format remains stable and you can load old saves, or do you just want a round trip test? It would be premature to test backwards compatibility when you haven’t shipped and don’t have any data you want to preserve yet.
rglover 2y ago
When I implemented the test suite for my JS framework [1], I realized that there was a ton of cruft and noise in most test set ups. The solution? Just start a mirror of the app [2] and its database(s) on different ports and run the tests against that. Do away with mocks/stubs in favor of just calling the code you're testing, intentionally using a test-only settings file (e.g., so you can use a dev account for third-party APIs). You can easily write clean up code in your test this way and be certain what you've built works. [1] https://cheatcode.co/joystick https://cheatcode.co/joystick [2] A mirror of the app/db creates a worry-free test env that can easily be reset without messing up your dev env.
jeppester 2y ago
Anti-Pattern is a word that sounds smart and educated, but is rarely used against something that does not have a legit use case. This article did not change my opinion on the subject. The word anti-pattern is confusing in itself. "Anti" is usually a prefix for something that battles or goes against the word that it prefixes. In my opinion a better word be a hostile or adverse pattern.
lowbloodsugar 2y ago
“Mocks only test the happy path.” This is a problem with the test authors, not mocks. “All the bugs are when talking to an actual database.” Databases have rules that need to be fillowed, and a lot of those can be tested very quickly with mocks. The combined system can have bugs, so don’t only use mocks. Mocks and unit tests are not a substitute for all the other tests you need to do. How this person can claim to be a CTO I have no idea.
- evnix 2y ago
  He probably meant it takes more effort to create mocks for all the negative cases. In most cases you won't have the time or the bandwidth to do this. Try mocking DB triggers, views, access rules etc in mocks and you will know why most teams don't bother mocking but use the real thing instead. And about the comment about him being a CTO. Well he is a CTO and you?
  - lowbloodsugar 2y ago
    Then he should have said that. Is not clear communication a requirement for CTO these days? Everything you are describing is about actually testing the database. A database is a complex server and things like db triggers and store procedures should be tested isolation too. And then you have integration tests too. My team just found a bug that wasn’t covered in a unit test. We found it in a long running API test. And so we added a unit test for the specific low level miss, and a quick integration test too.
bni 2y ago
Most IO nowadays in my context is to call some REST API. I prefer to use nock (https://github.com/nock/nock https://github.com/nock/nock) With that I can create an environment for my test to run in without changing anything about the implementation. The article does not seem to bring up this way to do it.
mbonnet 2y ago
I agree with this in principle, but some things can only be mocked, like AWS interactions in a test/CI environment.
exabrial 2y ago
Sigh, no no no, no, no it's not. In fact, Mocking is an essential tool for writing _unit_ tests; you know, testing exactly one thing (a 'unit') at a time. In Java for instance, a 'unit' would be a single static method, or a single class. Other languages will have different definitions of these terms, but the essential point would be "smallest reasonable grouping of code that can be executed, preferably deterministically" The problem is people conflate the various levels of integration tests. You actually should* have both: Full unit test coverage + an integration test to prove all of the pieces work together successfully. Small unit tests with mocks will point you _very quickly_ to exactly where a problem is a codebase by pointing out the effects of contract changes. Large integration tests prove your product meets requirements, and also that individual components (often written by different teams) work together. They are two different things with two different goals. * Important Caveat on the word 'should': Testing de-risks a build. However, if your business product is a risk itself (lets say you're hedging a startup on NFTs going wild), then your testing should reflect the amount of risk you're underwilling to spend money on. Unit testing in general speeds up development cycles, but takes time to develop. A good Software Engineering leader recognizes the risks in both the business side and development side and finds a balance. As a product matures, so should the thoroughness of it's testing.
brushfoot 2y ago
Bad article. Some of the advice is good, like decoupling I/O and logic where that makes sense. But the general idea of mocking being an anti-pattern is overreach. This kind of thinking is overly rigid/idealistic: > And with Postgres you can easily copy a test database with a random name from a template for each test. So there is your easy setup. > You need to test reality. Instead of mocking, invest in end-to-end (E2E) testing. "Easily" is like "just." The ease or difficulty is relative to skill, time, team size, infrastructure, and so on. As for testing reality, sure. But there's also a place for unit tests and partial integration tests. In some situations, mocking makes sense. In others, full E2E testing is better. Sometimes both might make sense in the same project. Use the right tool for the job.
- CuriouslyC 2y ago
  I've worked with a lot of pro-mocking engineers, and the time they spent on mocks easily outstripped the time a good build engineer would have spent creating a fast reusable test framework using real databases/dummy services/etc. The mocks won not because they were better or more efficient, but because of lack of deeper engineering skill and cargo culting.
  - mvdtnz 2y ago
    That engineer may have spent a couple of dozen hours on their mock. But the engineers who spent time on a test framework that uses real databases will soak up thousands of developer hours in CI time over the next decade.
    - erik_seaberg 2y ago
      Each mock needs to be maintained, sanity checking against the latest behavior of the actual dependency. And CI costs hardware, not developer time if he has anything else to work on.
    - remixff2400 2y ago
      Spinning up DB instances is a lot faster now than it used to be in the past. There are even modules for in-memory instances of certain databases. The speed of a unit test vs. that of one that uses an actual database is small enough for it to be a real consideration. That being said, of course, "it depends" on your use case. But I've found setting up this sort of test environment quite a bit easier now than writing database mocks, a lot less time-and-maintenance intensive, and relatively quick to run in any environment. (Also, in a decade, I'm pretty confident this gap will get even smaller, while the time to maintain mocks will stay constant)
DotaFan 2y ago
I've mocked a lot in my past. Last 2 years I've been using fakes explicitly, although it has an overhead, I like it as there is less maintenance and refactoring with tests.
aleksiy123 2y ago
If the normal way of testing is passing some parameters to test some code is what I'm going to call "outside-in" testing. Then mocking is "inside-out" testing. You check that your code is passing the right params/request to some dependency and reacting correctly to the output/response. Its really the same thing and you can flip between them by "inverting". Sometimes mocking just makes much more sense, and sometimes just passing paramaters to a function directly does. The end goal is the same: test some unit of codes behaviour against some specific state/situation. They have their place but like all testing should be layered with other types of test to "test in depth".
kyrofa 2y ago
Wow, some great examples in here for how to use mocks wrong. I get the impression the author has just never seen tests that use mocks properly, honestly. The various refactorings contained in here are fine, of course, but I see no reason to call the entire use of mocks an anti-pattern. They're a tool, and they need to be used properly. Let's not throw the baby out with the bath water.
imabotbeep2937 2y ago
Anyone who is overly zealous about anything is always wrong in the end. Including testing. "Why would people mock everything? Why not stand up a real test db and test on it?" Because the test zealous have explicitly declared that EACH test should be atomic. Yes you can find these people at major tech conferences. Each test should mock its own db, web service, etc. Every single time. And it should do that in no more than a few milliseconds, so that the entire project compiles in no more than 2mins, even for the largest and most complex corporate projects. And these tests should be fully end-to-end, even for complex microservices across complex networking architecture. Some of you may be rolling on the floor laughing at how naive and time-consuming such a project would be. We all agree such testing is a noble goal. But you need a team of absolute geniuses who do nothing but write "clever" code all day to get there in any sizeable project. My organization won't hire or pay those people, no matter what they say about having 100% coverage. We just do the best we can, cheat, and lower the targets as necessary.
- cwbrandsma 2y ago
  Lets not forget how long it would take to spin up an enterprise database, even in memory, there are hundreds (or thousands) of tables. Also there can be multiple databases with their own schema, and each require a fair amount of data in some of those tables just to do anything..
alganet 2y ago
Argument by exaggeration: For car crash tests, we should always use full humans. A test dummy might have a lot of sensors and be constructed to behave like a human in a crash, but you'll never get the full crash details with a doll. Notice the problem here? This argument does not consider the costs and risks associated with each approach. For testing, IO is very expensive. It leads to huge CI setups and testsuites that take multiple hours to run. There is no way around this except using some kind of test double.
deleted 2y ago
[deleted]
sethammons 2y ago
This is a solid article. So many mocks that, at the end, verify you set up your mock amd that 1=1. One paragraph I think is missing: error handling. You want units to be able to error so you can validate error handling which is _very_ hard on E2E tests. You can simulate disk full or db errors and make sure things fall back or log as expected. This can be done with fakes. Mocks are a specific type of test double that I have very little use of.
CuriouslyC 2y ago
I prefer dependency injection instead of mocking. Not only is injecting a "mock" service better than monkey patch mocks in pretty much all cases, but it's an actually useful architectural feature beyond testing.
- leni536 2y ago
  That's the only way to mock in some languages/testing frameworks. In C++ monkey patching would be quite difficult, but DI is simple. googlemock works this way.
jiehong 2y ago
Many comments are about the danger of over mocking, which is right. But, I’ve also suffered the opposite: having to use a lib that assumes it only runs in production, and always initialises some context no matter what (up to assuming only a specific VM would be used, never ever elsewhere, especially not in local) In the wild, I’ve rarely (if ever) saw code that was too testable. Too complex for no reason? Yes.
pmarreck 2y ago
What if we put the people in charge of an interface, also in charge of the mock for that interface that others can then use in their tests?
nineplay 2y ago
For some reason this article gives me flashbacks to the new CTO who comes in and declares 'micro-services' or 'containers' as the perfect solution for some problem that no one has actually run into. The article's author has had their pain points, but it doesn't mean all mocking is bad everywhere in every use case. I wrote some code recently that detects cycle errors in objects with inheritance and I mocked the DB calls. - Did I test for DB failures? No, but that's not the goal of the tests. - Could I have refactored the code to not rely on DB calls? Yes, but every refactor risks the introduction of more bugs. - Could I have launched a temporary DB instance and used that instead? Yes, but there's no obvious reason that would have been easier and cleaner than mocking DB calls. In python it wasn't hard to implement. It was the first time I'd used the mock library so naturally there was learning overhead but that's unavoidable - any solution would have learning overhead.
pmarreck 2y ago
> When you add UI-driven tests (and you should have some), I disagree. If you want to send your test suite into the toilet, add a headless browser driver and nondeterministic assertions based on it. Most output that becomes UI can be tested; the rest can be checked by a quick QA.
w10-1 2y ago
How can one write an article about testing that doesn't even mention the invariants you're trying to validate (by construction or testing)? That's the minimum context for addressing any QA solution. The GoF pattern book did list patterns, but it primarily argued for a simple language about patterns: context, problem, solution, limitations. It's clear. The blog-o-sphere recipe of click-bait, straw-man, glib advice designed not to guide practice but to project authority (and promise career advancement) is the exact opposite, because it obfuscates. The point of writing is to give people tools they can apply in their proximal situation. Are you really testing if your solutions start by refactoring the code to be more testable? That's more like design if not architecture -- excellent, but well beyond scope (and clearly in the CTO's bailiwick). And as for mocks: they're typically designed to represent subsystems at integration points (not responses to functions or IO/persistence subsystems). How hard is that to say? The CTO's way is not to win the argument but to lead organizations by teaching applicable principles, providing guardrails, and motivating people to do the right thing. Sorry to be exasperated and formulaic, but I think we can do better.
jweir 2y ago
I tell my developers that each mock you use costs you $100. Maybe it is worth it, but probably not. No I don't really charge em – but it gets the idea across that mocks have costs that you don't always see up front.
- alexfromapex 2y ago
  Maybe you’re writing them incorrectly then? I’ve written several that were for core app features used in 30ish test cases on a team with 7 engineers and they’ve worked flawlessly for over two years.
  - jweir 2y ago
    No, it is that mocks can hide interface changes. So if you have a mock, then you need to test that the interface works without the mock. And if you are doing that, why not just skip the mock? foo calls x(user, date) foo mock # tests pass x changes to x(user, time) but the tests for foo do not change, tests still pass, runtime errors. If you have static/strong typing the compiler will pick this up – but for dynamic languages you have a problem.
bunderbunder 2y ago
I used to love mocks, once upon a time. Nowadays, though, I internally sigh when I see them. I've come to the opinion that test doubles of any kind should be used as a last resort. They're a very useful tool for hacking testability into legacy code that's not particularly testable. But in a newer codebase they should be treated as a code smell. Code that needs mocks to be tested tends to be code that is overly stateful (read: temporally coupled), or that doesn't obey the Law of Demeter, or that does a poor job of pushing I/O to the edge where it belongs. And those are all design elements that make code brittle in ways that mocking can't actually fix; it can only sweep it under the carpet.
- dionian 2y ago
  Seconded. I shudder when I think back to the "test driven development" days when I wrote so much throwaway test code. Later, you try to refactor the app and it's another 50% of effort to update all the tests. The solution is to avoid it in the way you described.
  - bunderbunder 2y ago
    I'm curious if you've seen Ian Cooper's excellent talk, "TDD, Where Did It All Go Wrong?" https://www.youtube.com/watch?v=EZ05e7EMOLM https://www.youtube.com/watch?v=EZ05e7EMOLM One of his key points is that throwaway test code is only a problem if you neglect to throw it away.
- throwaway894345 2y ago
  I think the answer, like most things, is "it depends". Specifically it depends on the complexity of the thing you're mocking. Mocking a database is a bad idea because there's a ton of complexity incumbent in Postgres that your mocks are masking, so a test that mocks a database isn't actually giving you much confidence that your thing works, but if your interface is a "FooStore" (even one that is backed by a database), you can probably mock that just fine so long as your concrete implementation has "unit tests" with the database in the loop. Additionally, mocking/faking is often the only way to simulate error conditions. If you are testing a client that calls to a remote service, you will have to handle I/O errors or unexpected responses, and that requires mocking or faking the remote service (or rather, the client side transport stack). But yeah, I definitely think mocks should be used judiciously, and I _really_ think monkeypatch-based mocking is a travesty (one of the best parts about testing is that it pushes you toward writing maintainable, composable code, and monkey patching removes that incentive--it's also just a lot harder to do correctly).
serverlessmom 2y ago
oooooh I love this, I was writing against mocks a few months ago, and love to see others talking about more sophisticated testing methods: https://www.signadot.com/blog/why-developers-shouldnt-write-mocks-a-guide-to-modern-testing https://www.signadot.com/blog/why-developers-shouldnt-write-...
choeger 2y ago
A radical point of view. And as such it is of course wrong ;). First of all, there are languages where dry-running your code with all parameters mocked is still a valid test run. Python, js, and Perl for instance make it very simple to have a stupid error in the routine that crashes every run. But more importantly, a unit test usually executes inside the same process as the code. That gives you tremendous introspection capabilities and control over the execution flow. Testing for a specific path or scenario is exactly what you should do there. Finally, what if not mocks, are in-memory filesystems or databases? They, too, won't show all the behaviors that the real thing will do. And so so test containers or even full dedicated environments. It's all going to be an approximation.
- projektfu 2y ago
  I can foresee the "not what I mean" answers to everything. Oh, sure it's a fake DB but that's not a mock. Oh, yeah, you need to test with something that always makes an error but that's not a mock. Eventually, what they mean is that if it sucks, it's what they're talking about, and you should never do that. If it was really useful, it's not a mock.
RonnieOwnsLexus 2y ago
i dont get it. I if am taking a dependency on database or another class and i mock it using its interface, what is the harm in it? Essentially i have tested that given my dependencies working correctly my class would also work as expected.
- rkeene2 2y ago
  Almost -- you're testing that given your mocking implementation perfectly mirrors what the dependency would do given the inputs tested with that your functions produce the correct outputs (and hopefully you also verified the side-effects). The article is stating that almost nobody goes through the trouble of implementing a mock database perfectly, they just do something like make a single call return some hard-coded data. While this works a bit, it means that if the database ever changes its interface you have to remember to notice and implement that change as well.
looperhacks 2y ago
As a Java (mostly Spring) dev, I use mocks a lot to separate different components from each other, if I only want to test one of them. If your code only contains tests that mock other things, you're missing something, as others have pointed out. But just because you have a good coverage of integration testing, doesn't mean that writing isolated unit tests are bad. I find it much easier to diagnose a problem in a tight unit test than in a integration test that covers half the project. Some criticism to the article: The "more unit testing" section reminds me of junior devs asking why they can't test private methods in Java. If I'm testing a unit, I want to test the contract it promises (in this case, a method that does some checks and then sends something). That the behavior is split between multiple methods is an implementation detail, and writing tests around that makes changes harder (now I can't refactor the methods without also having to update the tests, even if the contract doesn't change) and it doesn't even test the contract! (There's nothing that makes sure that the mail is actually sent - we could be testing methods that aren't used by anything but the test code) For the "easier to test IO" section: just don't. Your tests now depend on some in-memory implementation that will behave differently than the real thing. That's just mocking with extra steps, you still don't know whether your application will work. If you want to do io, do the real io "Separation of logic and IO": this is in general the right thing to do, but the way it's described is weird. First, it does the same as in the "more unit testing" section with the same problems. Then, the code is refactored until it's barely understandable and the article even admits it with the Greenspan quote. In the end, the production code is worse, just to ... Not test whether there's actually some code doing the IO. I actually think there are some good ideas in there: separating the logic from the IO (and treating them as separate units) is important, not just for better testability, but also for easier refactoring and (if done with care) to be easier to reason about. In the end, you will need both unit and integration tests (and if your system is large enough, e2e tests). Whether you're using mocks for your unit tests or not, doesn't make much of a difference in the grand picture. Just don't mock stuff in integration or e2e if you absolutely can't prevent it.
0xbadcafebee 2y ago
Mocks aren't an anti-pattern. Anti-patterns are a "common response to a recurring problem, usually ineffective, risking being highly counterproductive". On the contrary, mocks are a common response to a recurring problem which are often effective and have no greater risk than a great many alternative testing methodologies. They do solve problems and they are useful. But like literally anything else in the universe: it depends, and they don't solve every problem. You wanna know how to test without mocking? Use any kind of test. Seriously, just make a test. I don't care what kind of test it is, just have one. When you notice a problem your testing doesn't catch, improve your testing. Rinse, repeat. I don't care what kind of 10x rockstar uber-genius you think you are, you're going to be doing this anyway no matter what super amazing testing strategy you come up with, so just start on it now. Are there some ways of testing that are more effective than others? Yes, but it depends. If testing were simple, easy, straightforward and universal we wouldn't be debating how to do it. (about 99% of the time I'm disappointed in these clickbait blog posts upvoted on HN. they are shallow and brief (it's a blog post, not a book), yet quite often dismissive of perfectly reasonable alternatives, and in the absence of any other information, misleading. it would be better to just describe the problem and how the author solved it, and leave out the clickbaity sweeping generalizations and proclamations)
mvdtnz 2y ago
Frightening that there is someone out there calling themselves a CTO and offering CTO coaching who doesn't understand what unit testing is.
sevensor 2y ago
If you take the article's advice to move everything that's not IO into pure, testable code (which is good), what's left is code that does IO. What are you even testing when you call such a procedure? At that point, it's mostly calls into other people's code. Maybe that's a good place to draw the line on testing things?
binary132 2y ago
Never understood what was so hard about dependency inversion
kevwil 2y ago
"What to do instead of Mocking" should be more like "Enhance your use of Mocks with better unit tests and integration tests". The listed complaints sound more like problems with sloppy/lazy coding practices than actual problems with mocks.
worik 2y ago
Some testing frameworks (thinking of my experience with Flutter) insist on mocking From memory the HTTP client 404s every request in testing mode
alexfromapex 2y ago
It is so cringe to see bad advice like this being given. Yes, you can write mocks incorrectly. You should not model them after the "happy path" but you should make sure they cover the most use-cases both good and bad. I have been a senior or principal engineer on teams that did both of these approaches and the non-mocking approach is terrible because you end up with separate tests that have colliding data. It's slower using a real database back-end and becomes a mess and leads to issues where your database is heavily coupled to your test code which is the real anti-pattern. Then a year or two later when you want to change databases or database architectures you're screwed because you have to go into a bunch of tests manually and change things. The whole point of the mocks is it makes everything modular.
gchaincl 2y ago
> This often happens with an in-memory database, which is compatible to your main database you do use in production, but is faster. Not sure how this will solve edge cases problems described at the beginning of the article
nerder92 2y ago
Everytime I read about this kind of argument against mock is usually due to a misunderstanding on why unit tests with mocking exists in the first place. The underlying assumption is that tests are a quality assurance tool but I think that this is true only for E2E (possibly in production). In outside in TDD unit test are used as a design tool not a QA one, and mocking is a convenient way to quickly do that without the need of implementing the next layer, mock usually don’t replace an implemented service that does IO they implemented a noop that triggers an exception (so they your E2E won’t pass until you implement that) The problem is in the name, unit test should be called implementation spec or in-code documentation. Each layer of testing has its roles and serves a different purpose.
tpoacher 2y ago
Mocking is an indeed an anti pattern ... when dealing with tests that pretend to be unit tests but are not actually unit tests (e.g. needing to be aware if IO edge-cases, to quote the article). But tests that are not actually unit tests masquerading as unit tests and vice versa is arguably the bigger problem here. Not mocking per se.

181 points by alexzeitler 2y ago

The problem is when IOService has edge cases. When building the mock, does it address the edge cases? When you want to find bugs by testing, the tests need to test the real world. So to work, the mock for IOService needs to model the edge cases of IOService. Does it? Does the developer know they need to model the edge cases, or the mock is not helping with finding bugs? Do you even know the edge cases of IOService? When IOService is a database service, does your mock work for records that do not exist? Or return more than one record? It depends. Mocks are used to remove variables from the experiment you are running (the tests) and see if it behaves under very specific conditions. If you want to test how the code behaves when a specific row is returned by the database, but instead the dependency returns something else, then you are not testing that use case anymore. Reproducibility also has its values. But yes, you can definitely make your mocks return errors and fail in a myriad of ways. Not to say you should mock everything. Of course having proper integration tests is also important, but articles like these will rarely tell you to have a good balance between them, and will instead tell you that something is correct and something else is wrong. You should do what makes sense for that specific case and exercise your abilities to make the right choice, and not blindly follow instructions you read in a blog post.

sisve 2y ago
100% agree. Maybe you should be the amazing cto. Context always matters.
watermelon0 2y ago
I totally agree, there is a balance between what makes sense to mock, and what needs proper integration tests. Additionally, just using integration tests does not guarantee that edge cases are covered, and you can just as easily write integration tests for happy path, without thinking about the rest.

dboreham 2y ago

I've been doing this stuff (software) for a very long time and if it hadn't been invented by others, I'd never have thought of Mocking. It's that stupid of an idea. When I first came across it used in anger in a large project it took me a while to get my head around what was going on. When the penny dropped I remember a feeling of doom, like I had realized I was in The Matrix. Don't work there, and don't work with mock-people any more.

ams92 2y ago
With containerization it’s very quick to spin up test dependencies as well as part of your CICD. Why mock calls to a datastore when it’s super easy to spin up an ephemeral postgresql instance to test on?
- randomdata 2y ago
  > Why mock calls to a datastore when it’s super easy to spin up an ephemeral postgresql instance to test on? It's actually super hard to get Postgres to fail, which is what you will be most interested in testing. Granted, you would probably use stubbing for that instead.
- UK-AL 2y ago
  Because you have 40000 tests, and an in memory object means they can run in seconds. And the real thing runs in minutes.
- specialist 2y ago
  Yup. Section "Easier to Test IO" in TFA.
sampullman 2y ago
I don't like mocking either, but there are periodically situations where I've found it useful. Sometimes there is a complex system (whether of your own design or not) that isn't amenable to integration/e2e testing, and the interesting parts can't be easily unit tested due to external or tightly coupled dependencies. Of course you can always pick it apart and refactor so it can be unit tested, but sometimes the effort required makes mocking look pretty appealing.

bitcoin_anon 2y ago

Your mock object is a joke; that object is mocking you. For needing it. ~ Rich Hickey

alexfromapex 2y ago
Don’t really care about advice from the guy who invented an incredibly non pragmatic programming language. I also honestly had to look up who he was. So his sage advice hasn’t brought him much fame.

KaiserPro 2y ago

The problem is that mocks are normally used to avoid testing services/outside code. For example making a wrapper around a message system if you don't mock, it tests both your code and the message system. However the the overhead of keeping the mocking system up to date is a pain in the balls.

devit 2y ago
Testing both your code and the message system is exactly what you want, since if the message system is broken in a way that upstream didn't catch, you want to learn about it during testing and not production, if possible.
- jon_richards 2y ago
  I’m still mad about the time I was told to mock a payment gateway in tests even though they had a testing environment and then got a slew of bug reports from people whose company names had punctuation (and thus failed the name validation the payment gateway was secretly running).
  - UK-Al05 2y ago
    Could you just run some behaviour tests against the gateway? You can get the best of both.
  - mehagar 2y ago
    That reminds me that Stripe actually maintains (or used to) their own mock for their Ruby package. This puts the burden on maintaining the mock on the library owner, where it is more likely that they would implement the mock correctly, edge cases and all.
  - mvdtnz 2y ago
    You should be unit testing components that interact with the payment gateway. This could involve dozens of even hundreds of tests, where the gateway should be mocked. These tests should be fast and reliable. In addition, you should have a small suite of integration/E2E tests against a real test instance of the gateway. These tests may be slow and unreliable (because of the real network involved) but catch those hairy issues you failed to find in unit tests. Also, when your integration suite (or customer reports) discovers that the payment gateway fails on punctuation, another unit test should be added with a mock that responds the same way, and an E2E test added with punctuation in the input data and a failure expectation. What makes you so certain you would have included punctuation in the input data if the test had not mocked the gateway?

christkv 2y ago

The golden rule was to only mock your own code. Make a facade around the framework class using an interface and mock that if needed to decouple your tests. Then write integration tests against your implementation of the interface. The moment you mock other people’s code you have brittle tests.

graypegg 2y ago

Tests are a tool for you, the developer. They have good effects for other people, but developers are the people that directly interact with them. When something fails, it's a developer that has to figure out what change they wrote introduced a regression. They're just tools, not some magic incantation that protects you from bugs. I think the author might be conflating good tests with good enough tests. If IOService is handled by a different team, I expect them to assure IOService behaves how it should, probably using tests. The reason we're mocking IOService is because it's a variable that I can remove, that makes the errors I get from a test run MUCH easier to read. We're just looking at the logic in one module/class/method/function. It's less conceptually good to mock things in tests, since I'm not testing the entire app that we actually ship, but full no-mocks E2E tests are harder to write and interpret when something goes wrong. I think that makes them a less useful tool. The thing I do agree on, is assuming your mocks should only model the happy path. I'd say if something can throw an exception, you should at least include that in a mock. (as a stubbed method that always throws) but making the burden of reimplementing your dependancies mandatory, or relying on them in tests is going to mean you write less tests, and get worse failure messages. Like everything, it depends eh?

specialist 2y ago
With I/O in general, I've observed that socket, protocol, and serialization logic are often tightly coupled. If they're decoupled, there's no need to mock protocol or serialization. There's some cliché wrt "don't call me, I'll call you" as advice how to flip the call stack. Sorry, no example handy (on mobile). But the gist is to avoid nested calls, flattening the code paths. Less like a Russian doll, more like a Lego instructions. In defense of mocks, IoC frameworks like Spring pretty much necessitate doing the wrong thing.
deleted 2y ago
[deleted]
campbel 2y ago
This 100%. I'm not sure how the author managed to create consistent failure cases using real service dependencies, but in my code I find mocks to be the easiest way to test error scenarios.

worksonmine 2y ago

If I already have passing tests for anything function A might do, I can safely assume it will behave the same when called from B, C and D.

t43562 2y ago
In some languages A might free a memory allocation e.g. after communicating with some server. If B also frees that memory then there is a bug. Presumably this means B's tests are wrong/incomplete. If B was mocking A to avoid the IO, you might not find out.

solatic 2y ago

If you're writing a CRUD app and mocking your database calls instead of just starting an actual Postgres instance before running the tests, you're probably using mocking wrong. If you're writing a custom frontend for GitHub using the GitHub API and don't bother writing a decent set of mocks for how you expect the GitHub API to behave, your app will quickly require either full manual QA at best or become untestable at worst. Some APIs are very stable, and testing against the API itself can hit rate limiting, bans, and other anti-abuse mechanisms that introduce all kinds of instability to your test suite. Use the right tools to solve your problems.

Fire-Dragon-DoL 2y ago
100% agree
arcticbull 2y ago
When you write tests with mocks you almost always at some point end up with tests that test your mocks lol, and tests that test that you wrote the tests you think you wrote -- not the software itself. I’ve never been thrilled by tests that rely on mocking — it usually means you need to re-express your module interface boundary. Mocks for me fall into the class of software I affectionately call “load-bearing paint.” It’s basically universally the wrong tool for any given job but that really doesn’t stop people. Putting in a data class or similar model object and a delegate is usually sufficient and a much better tool.
- mehagar 2y ago
  I agree that if you need to write mocks, it's likely that your interfaces are poorly defined. This is one of the claimed benefits of test driven development - writing the tests first forces you to design the code in a way that cleanly separates modules so they can be tested.
  - UK-Al05 2y ago
    You have to mock/fake when modules call dependencies. Your way means you only ever have siblings. With an orchestrator pulling results out of one module and pushing it into another.

whoisthemachine 2y ago

> Modelling the happy path is great for refactoring - even a necessity, but doesn’t help with finding bugs. This is a common misconception (one that I also initially held). Unit tests aren't meant to find bugs, they're meant to protect against regressions, and in doing so, act as a documentation of how a component is supposed to behave in response to different input.

UK-AL 2y ago
They can do both.
postalrat 2y ago
I think writing tests as a form of documentation is a waste of time. If I'm using a component I don't want to read unit tests to figure out what it should do. Unit tests are most often used to cover a few more lines that need coverage. That's the value they provide.
- Izkata 2y ago
  Documentation for other developers of that code, not users of that code.
  - deleted 2y ago
    [deleted]
  - curun1r 2y ago
    Also worth noting that "other developers" includes the original developer when they return to the code several months later.
- randomdata 2y ago
  A well designed API will generally allow users to understand usage without any additional documentation, sure. However, those who modify the API in the future will want to know every last detail that you knew when you were writing it originally. That must be documented to ensure that they don't get something wrong and break things – and for their general sanity. That is, unless you hate future developers for some reason. You could do it in Word instead, I suppose, but if you write it in code then a computer can validate that the documentation you wrote is true. That brings tremendous value.

marvstazar 2y ago

If you inherited a project with no tests at all, mocking is a lifesaver. It allows you to only worry about specific aspects of the application so you can start writing and running tests. I agree though that if not done properly, it can be overused and can make your tests practically worthless.

john-radio 2y ago

[flagged]

mehagar 2y ago

The catch-22 with refactoring to be able to write unit tests is that refactoring introduces risk as you are changing code, and you need tests to help reduce that risk. But you can't easily write tests without refactoring. This has been a very difficult problem for the team I'm currently on. The only strategy I'm aware of is described in `Working Effectively With Legacy Code`, where you start by writing throwaway unit or E2E tests that give you "cover" for being able to refactor. These tests depend on the implementation or may use mocking just to get started. Then you refactor, and write better unit tests. Then get rid of the throwaway tests.

t43562 2y ago
Why get rid of working e2e tests? IMO they are more useful than unit tests at finding the kinds of problems that stop a release/deployment. You can attack from both directions: e2e tests make sure that certain processes work in fairly ordinary situations, then look for little things that you can unit test without huge refactoring. When you've pushed these as far as you can, section off some area and start refactoring it. Do your best to limit your refactoring to single aspects or areas so that you are never biting off more than you can chew. Don't expect everything to become wonderful in one PR. Your e2e tests will catch some errors and when you look at what those commonly are then you can see how to best improve your tests to catch them earlier and save yourself time. In python I had stupid errors often - syntax errors in try-catch blocks or other things like that. If I used a linter first then I caught many of those errors very quickly. I was working on a build system so I mocked the build - created a much simpler and shorter build - so I could catch dumb errors fast, before I ran the longer e2e test on the full build. IMO you need to progress to your vision but trying to reach it in one step is very dangerous. Make life better piece by piece. You can even do PRs where you only add comments to the existing files and classes (not too much detail but answering questions like "why" is this file/class here). This helps to make sure you really understand the current system is doing before you change it. I once added type hints everywhere to a legacy python program - it wasn't as helpful as I'd hoped but it did prevent some issues while I was refactoring.

UK-AL 2y ago

In general I have fake IO object and a real IO object. Then run the same bunch of tests against them to make sure behaviour matches. You have verified your mock has the same behaviour as the real thing. I then run unit tests against the fake io object. I don't mock internals, only boundaries. If for whatever reason i want to test it against the real db i can simply swap out the fake for the real object.

languagehacker 2y ago

I don't think mocking is an anti-pattern. Using only unit tests and then mocking everything probably is. Mocks have a perfectly viable place in testing. They help establish boundaries and avoid side effects that are not pertinent to the logic being tested. I would reference the testing pyramid when thinking about where to be spending time in unit tests vs. integration tests vs. end to end tests. What introduces risk is if we're mocking behaviors that aren't being tested further up the pyramid.

mehagar 2y ago
I like the testing pyramid specifically because it captures the tradeoffs between the different kinds of tests. Mocks can come in handy, but like anything else can be abused. We need a "Mock this, not that" kind of guide.
hot_gril 2y ago
I think the better advice that goes with the spirit of the article is, prioritize integration testing over unit testing if you're constrained on time.

jasonlotito 2y ago

Go ahead. Don't mock that external service that you rely on for an API. Now you need to have multiple keys, one for each developer, or share keys separate from various environments? Does it not offer dev/test/staging/prod keys? Well, now you need to share those keys. Does it only offer Prod keys? Now you are stuck sharing that. API request limits? Now you are eating through that just to run tests. And let's not forget that testing things locally means you are mocking the network, or lack-thereof. "Mocking is an anti-pattern" is a great sentiment if you ignore costs or restrictions in the real world.

bottlepalm 2y ago
Also the intermittent failures of your tests relying on unstable dependencies.
- t43562 2y ago
  If your dependencies are unstable then that is very important to know! If it means you have to add forms of resilience then that's good for your code perhaps?
  - bottlepalm 2y ago
    The point is it makes you tests nondeterministic which is a huge problem with large fluid teams and thousands of tests. They need to be deterministic.
t43562 2y ago
That is a fairly good reason for trying to use external systems/tools that make testing easy/cheap to do. So a good approach would be to have tests where you can run with the mock and then run the same tests with the real system. Anything you catch with the mock saves you from using the costly system but you still get real testing.

time0ut 2y ago

Maybe I am missing something, but how else would I test various exception handling paths? There is a whole world of errors that can occur during IO. What happens if I get a 500 from that web service call? How does my code handle a timeout? What if the file isn't found? It is often only possible to simulate these scenarios using a mock or similar. These are also code paths you really want to understand.

lonelyasacloud 2y ago
I don't think there's a disagreement; the author states "Whenever I look at mocks, they mostly have the same problem as all unit tests that I see, they only model the happy path". So by corollary their opinion of the correct usage of mocking would also include modelling brokenness.
dgb23 2y ago
Put a small data interface around your IO, have it return DATA | NOT_FOUND etc. Then your tests don't need behavioral mocks or DI, they just need the different shapes of data and you test your own code instead of whatever your IO dependency is or some simulation thereof.
- myfavoritetings 2y ago
  isn't that just mocking with extra steps?
  - dgb23 2y ago
    You could call the data you generate for the tests "mocks". But they really aren't "mocks" in the sense of behavioral mocks via IoC/DI and you don't need to manipulate them via some kind of interface in order to put them into the right state for your particular tests. There are some extra steps, but you get extremely simple and reliable tests in return. In many(!) cases you already have a data interface, especially with HTTP/REST APIs. All you need to do is simply not bury the IO call down the stack and maybe describe the failure conditions as plain data in your signature and voila. (This is not a replacement for higher order testing like, manual, E2E or integration tests. But it certainly beats unit testing with mocks IMO.)

cryptonector 2y ago

Mocking is useful for testing small parts of your programs/libraries. For full-scale testing you really need to not emulate because any emulation will be woefully incomplete, so you're going to have to spin up a virtual network with all the services you need including DNS.

skybrian 2y ago

Mocks can be useful when there is a standard protocol and you want to document and test that your code follows the protocol exactly, doing the same steps, independently of whether some other component also follows the protocol. It tests something different from whether or not two components work together after you change both of them. It takes time to come up with good protocols that will remain stable and it might not be worth the effort to test it when the protocol design is new and still in flux, and you don’t have alternative implementations anyway. This is often the case for two internal modules in the same system. If you ever want to change the interface, you can change both of them, so an integration test will be a better way to ensure that functionality survives protocol changes. Database access tends to be a bad thing to mock because the interface is very wide: “you can run any SQL transaction here.” You don’t want to make changing the SQL harder to do. Any equivalent SQL transaction should be allowed if it reads or writes the same data. Compare with testing serialization: do you want to make sure the format remains stable and you can load old saves, or do you just want a round trip test? It would be premature to test backwards compatibility when you haven’t shipped and don’t have any data you want to preserve yet.

rglover 2y ago

When I implemented the test suite for my JS framework [1], I realized that there was a ton of cruft and noise in most test set ups. The solution? Just start a mirror of the app [2] and its database(s) on different ports and run the tests against that. Do away with mocks/stubs in favor of just calling the code you're testing, intentionally using a test-only settings file (e.g., so you can use a dev account for third-party APIs). You can easily write clean up code in your test this way and be certain what you've built works. [1] https://cheatcode.co/joystick https://cheatcode.co/joystick [2] A mirror of the app/db creates a worry-free test env that can easily be reset without messing up your dev env.

jeppester 2y ago

Anti-Pattern is a word that sounds smart and educated, but is rarely used against something that does not have a legit use case. This article did not change my opinion on the subject. The word anti-pattern is confusing in itself. "Anti" is usually a prefix for something that battles or goes against the word that it prefixes. In my opinion a better word be a hostile or adverse pattern.

lowbloodsugar 2y ago

“Mocks only test the happy path.” This is a problem with the test authors, not mocks. “All the bugs are when talking to an actual database.” Databases have rules that need to be fillowed, and a lot of those can be tested very quickly with mocks. The combined system can have bugs, so don’t only use mocks. Mocks and unit tests are not a substitute for all the other tests you need to do. How this person can claim to be a CTO I have no idea.

evnix 2y ago
He probably meant it takes more effort to create mocks for all the negative cases. In most cases you won't have the time or the bandwidth to do this. Try mocking DB triggers, views, access rules etc in mocks and you will know why most teams don't bother mocking but use the real thing instead. And about the comment about him being a CTO. Well he is a CTO and you?
- lowbloodsugar 2y ago
  Then he should have said that. Is not clear communication a requirement for CTO these days? Everything you are describing is about actually testing the database. A database is a complex server and things like db triggers and store procedures should be tested isolation too. And then you have integration tests too. My team just found a bug that wasn’t covered in a unit test. We found it in a long running API test. And so we added a unit test for the specific low level miss, and a quick integration test too.

bni 2y ago

Most IO nowadays in my context is to call some REST API. I prefer to use nock (https://github.com/nock/nock https://github.com/nock/nock) With that I can create an environment for my test to run in without changing anything about the implementation. The article does not seem to bring up this way to do it.

mbonnet 2y ago

I agree with this in principle, but some things can only be mocked, like AWS interactions in a test/CI environment.

exabrial 2y ago

Sigh, no no no, no, no it's not. In fact, Mocking is an essential tool for writing _unit_ tests; you know, testing exactly one thing (a 'unit') at a time. In Java for instance, a 'unit' would be a single static method, or a single class. Other languages will have different definitions of these terms, but the essential point would be "smallest reasonable grouping of code that can be executed, preferably deterministically" The problem is people conflate the various levels of integration tests. You actually should* have both: Full unit test coverage + an integration test to prove all of the pieces work together successfully. Small unit tests with mocks will point you _very quickly_ to exactly where a problem is a codebase by pointing out the effects of contract changes. Large integration tests prove your product meets requirements, and also that individual components (often written by different teams) work together. They are two different things with two different goals. * Important Caveat on the word 'should': Testing de-risks a build. However, if your business product is a risk itself (lets say you're hedging a startup on NFTs going wild), then your testing should reflect the amount of risk you're underwilling to spend money on. Unit testing in general speeds up development cycles, but takes time to develop. A good Software Engineering leader recognizes the risks in both the business side and development side and finds a balance. As a product matures, so should the thoroughness of it's testing.

brushfoot 2y ago

Bad article. Some of the advice is good, like decoupling I/O and logic where that makes sense. But the general idea of mocking being an anti-pattern is overreach. This kind of thinking is overly rigid/idealistic: > And with Postgres you can easily copy a test database with a random name from a template for each test. So there is your easy setup. > You need to test reality. Instead of mocking, invest in end-to-end (E2E) testing. "Easily" is like "just." The ease or difficulty is relative to skill, time, team size, infrastructure, and so on. As for testing reality, sure. But there's also a place for unit tests and partial integration tests. In some situations, mocking makes sense. In others, full E2E testing is better. Sometimes both might make sense in the same project. Use the right tool for the job.

CuriouslyC 2y ago
I've worked with a lot of pro-mocking engineers, and the time they spent on mocks easily outstripped the time a good build engineer would have spent creating a fast reusable test framework using real databases/dummy services/etc. The mocks won not because they were better or more efficient, but because of lack of deeper engineering skill and cargo culting.
- mvdtnz 2y ago
  That engineer may have spent a couple of dozen hours on their mock. But the engineers who spent time on a test framework that uses real databases will soak up thousands of developer hours in CI time over the next decade.
  - erik_seaberg 2y ago
    Each mock needs to be maintained, sanity checking against the latest behavior of the actual dependency. And CI costs hardware, not developer time if he has anything else to work on.
  - remixff2400 2y ago
    Spinning up DB instances is a lot faster now than it used to be in the past. There are even modules for in-memory instances of certain databases. The speed of a unit test vs. that of one that uses an actual database is small enough for it to be a real consideration. That being said, of course, "it depends" on your use case. But I've found setting up this sort of test environment quite a bit easier now than writing database mocks, a lot less time-and-maintenance intensive, and relatively quick to run in any environment. (Also, in a decade, I'm pretty confident this gap will get even smaller, while the time to maintain mocks will stay constant)

DotaFan 2y ago

I've mocked a lot in my past. Last 2 years I've been using fakes explicitly, although it has an overhead, I like it as there is less maintenance and refactoring with tests.

aleksiy123 2y ago

If the normal way of testing is passing some parameters to test some code is what I'm going to call "outside-in" testing. Then mocking is "inside-out" testing. You check that your code is passing the right params/request to some dependency and reacting correctly to the output/response. Its really the same thing and you can flip between them by "inverting". Sometimes mocking just makes much more sense, and sometimes just passing paramaters to a function directly does. The end goal is the same: test some unit of codes behaviour against some specific state/situation. They have their place but like all testing should be layered with other types of test to "test in depth".

kyrofa 2y ago

Wow, some great examples in here for how to use mocks wrong. I get the impression the author has just never seen tests that use mocks properly, honestly. The various refactorings contained in here are fine, of course, but I see no reason to call the entire use of mocks an anti-pattern. They're a tool, and they need to be used properly. Let's not throw the baby out with the bath water.

imabotbeep2937 2y ago

Anyone who is overly zealous about anything is always wrong in the end. Including testing. "Why would people mock everything? Why not stand up a real test db and test on it?" Because the test zealous have explicitly declared that EACH test should be atomic. Yes you can find these people at major tech conferences. Each test should mock its own db, web service, etc. Every single time. And it should do that in no more than a few milliseconds, so that the entire project compiles in no more than 2mins, even for the largest and most complex corporate projects. And these tests should be fully end-to-end, even for complex microservices across complex networking architecture. Some of you may be rolling on the floor laughing at how naive and time-consuming such a project would be. We all agree such testing is a noble goal. But you need a team of absolute geniuses who do nothing but write "clever" code all day to get there in any sizeable project. My organization won't hire or pay those people, no matter what they say about having 100% coverage. We just do the best we can, cheat, and lower the targets as necessary.

cwbrandsma 2y ago
Lets not forget how long it would take to spin up an enterprise database, even in memory, there are hundreds (or thousands) of tables. Also there can be multiple databases with their own schema, and each require a fair amount of data in some of those tables just to do anything..

alganet 2y ago

Argument by exaggeration: For car crash tests, we should always use full humans. A test dummy might have a lot of sensors and be constructed to behave like a human in a crash, but you'll never get the full crash details with a doll. Notice the problem here? This argument does not consider the costs and risks associated with each approach. For testing, IO is very expensive. It leads to huge CI setups and testsuites that take multiple hours to run. There is no way around this except using some kind of test double.

deleted 2y ago

[deleted]

sethammons 2y ago

This is a solid article. So many mocks that, at the end, verify you set up your mock amd that 1=1. One paragraph I think is missing: error handling. You want units to be able to error so you can validate error handling which is _very_ hard on E2E tests. You can simulate disk full or db errors and make sure things fall back or log as expected. This can be done with fakes. Mocks are a specific type of test double that I have very little use of.

CuriouslyC 2y ago

I prefer dependency injection instead of mocking. Not only is injecting a "mock" service better than monkey patch mocks in pretty much all cases, but it's an actually useful architectural feature beyond testing.

leni536 2y ago
That's the only way to mock in some languages/testing frameworks. In C++ monkey patching would be quite difficult, but DI is simple. googlemock works this way.

jiehong 2y ago

Many comments are about the danger of over mocking, which is right. But, I’ve also suffered the opposite: having to use a lib that assumes it only runs in production, and always initialises some context no matter what (up to assuming only a specific VM would be used, never ever elsewhere, especially not in local) In the wild, I’ve rarely (if ever) saw code that was too testable. Too complex for no reason? Yes.

pmarreck 2y ago

What if we put the people in charge of an interface, also in charge of the mock for that interface that others can then use in their tests?

nineplay 2y ago

For some reason this article gives me flashbacks to the new CTO who comes in and declares 'micro-services' or 'containers' as the perfect solution for some problem that no one has actually run into. The article's author has had their pain points, but it doesn't mean all mocking is bad everywhere in every use case. I wrote some code recently that detects cycle errors in objects with inheritance and I mocked the DB calls. - Did I test for DB failures? No, but that's not the goal of the tests. - Could I have refactored the code to not rely on DB calls? Yes, but every refactor risks the introduction of more bugs. - Could I have launched a temporary DB instance and used that instead? Yes, but there's no obvious reason that would have been easier and cleaner than mocking DB calls. In python it wasn't hard to implement. It was the first time I'd used the mock library so naturally there was learning overhead but that's unavoidable - any solution would have learning overhead.

pmarreck 2y ago

> When you add UI-driven tests (and you should have some), I disagree. If you want to send your test suite into the toilet, add a headless browser driver and nondeterministic assertions based on it. Most output that becomes UI can be tested; the rest can be checked by a quick QA.

w10-1 2y ago

How can one write an article about testing that doesn't even mention the invariants you're trying to validate (by construction or testing)? That's the minimum context for addressing any QA solution. The GoF pattern book did list patterns, but it primarily argued for a simple language about patterns: context, problem, solution, limitations. It's clear. The blog-o-sphere recipe of click-bait, straw-man, glib advice designed not to guide practice but to project authority (and promise career advancement) is the exact opposite, because it obfuscates. The point of writing is to give people tools they can apply in their proximal situation. Are you really testing if your solutions start by refactoring the code to be more testable? That's more like design if not architecture -- excellent, but well beyond scope (and clearly in the CTO's bailiwick). And as for mocks: they're typically designed to represent subsystems at integration points (not responses to functions or IO/persistence subsystems). How hard is that to say? The CTO's way is not to win the argument but to lead organizations by teaching applicable principles, providing guardrails, and motivating people to do the right thing. Sorry to be exasperated and formulaic, but I think we can do better.

jweir 2y ago

I tell my developers that each mock you use costs you $100. Maybe it is worth it, but probably not. No I don't really charge em – but it gets the idea across that mocks have costs that you don't always see up front.

alexfromapex 2y ago
Maybe you’re writing them incorrectly then? I’ve written several that were for core app features used in 30ish test cases on a team with 7 engineers and they’ve worked flawlessly for over two years.
- jweir 2y ago
  No, it is that mocks can hide interface changes. So if you have a mock, then you need to test that the interface works without the mock. And if you are doing that, why not just skip the mock? foo calls x(user, date) foo mock # tests pass x changes to x(user, time) but the tests for foo do not change, tests still pass, runtime errors. If you have static/strong typing the compiler will pick this up – but for dynamic languages you have a problem.

bunderbunder 2y ago

I used to love mocks, once upon a time. Nowadays, though, I internally sigh when I see them. I've come to the opinion that test doubles of any kind should be used as a last resort. They're a very useful tool for hacking testability into legacy code that's not particularly testable. But in a newer codebase they should be treated as a code smell. Code that needs mocks to be tested tends to be code that is overly stateful (read: temporally coupled), or that doesn't obey the Law of Demeter, or that does a poor job of pushing I/O to the edge where it belongs. And those are all design elements that make code brittle in ways that mocking can't actually fix; it can only sweep it under the carpet.

dionian 2y ago
Seconded. I shudder when I think back to the "test driven development" days when I wrote so much throwaway test code. Later, you try to refactor the app and it's another 50% of effort to update all the tests. The solution is to avoid it in the way you described.
- bunderbunder 2y ago
  I'm curious if you've seen Ian Cooper's excellent talk, "TDD, Where Did It All Go Wrong?" https://www.youtube.com/watch?v=EZ05e7EMOLM https://www.youtube.com/watch?v=EZ05e7EMOLM One of his key points is that throwaway test code is only a problem if you neglect to throw it away.
throwaway894345 2y ago
I think the answer, like most things, is "it depends". Specifically it depends on the complexity of the thing you're mocking. Mocking a database is a bad idea because there's a ton of complexity incumbent in Postgres that your mocks are masking, so a test that mocks a database isn't actually giving you much confidence that your thing works, but if your interface is a "FooStore" (even one that is backed by a database), you can probably mock that just fine so long as your concrete implementation has "unit tests" with the database in the loop. Additionally, mocking/faking is often the only way to simulate error conditions. If you are testing a client that calls to a remote service, you will have to handle I/O errors or unexpected responses, and that requires mocking or faking the remote service (or rather, the client side transport stack). But yeah, I definitely think mocks should be used judiciously, and I _really_ think monkeypatch-based mocking is a travesty (one of the best parts about testing is that it pushes you toward writing maintainable, composable code, and monkey patching removes that incentive--it's also just a lot harder to do correctly).

serverlessmom 2y ago

oooooh I love this, I was writing against mocks a few months ago, and love to see others talking about more sophisticated testing methods: https://www.signadot.com/blog/why-developers-shouldnt-write-mocks-a-guide-to-modern-testing https://www.signadot.com/blog/why-developers-shouldnt-write-...

choeger 2y ago

A radical point of view. And as such it is of course wrong ;). First of all, there are languages where dry-running your code with all parameters mocked is still a valid test run. Python, js, and Perl for instance make it very simple to have a stupid error in the routine that crashes every run. But more importantly, a unit test usually executes inside the same process as the code. That gives you tremendous introspection capabilities and control over the execution flow. Testing for a specific path or scenario is exactly what you should do there. Finally, what if not mocks, are in-memory filesystems or databases? They, too, won't show all the behaviors that the real thing will do. And so so test containers or even full dedicated environments. It's all going to be an approximation.

projektfu 2y ago
I can foresee the "not what I mean" answers to everything. Oh, sure it's a fake DB but that's not a mock. Oh, yeah, you need to test with something that always makes an error but that's not a mock. Eventually, what they mean is that if it sucks, it's what they're talking about, and you should never do that. If it was really useful, it's not a mock.

RonnieOwnsLexus 2y ago

i dont get it. I if am taking a dependency on database or another class and i mock it using its interface, what is the harm in it? Essentially i have tested that given my dependencies working correctly my class would also work as expected.

rkeene2 2y ago
Almost -- you're testing that given your mocking implementation perfectly mirrors what the dependency would do given the inputs tested with that your functions produce the correct outputs (and hopefully you also verified the side-effects). The article is stating that almost nobody goes through the trouble of implementing a mock database perfectly, they just do something like make a single call return some hard-coded data. While this works a bit, it means that if the database ever changes its interface you have to remember to notice and implement that change as well.

looperhacks 2y ago

As a Java (mostly Spring) dev, I use mocks a lot to separate different components from each other, if I only want to test one of them. If your code only contains tests that mock other things, you're missing something, as others have pointed out. But just because you have a good coverage of integration testing, doesn't mean that writing isolated unit tests are bad. I find it much easier to diagnose a problem in a tight unit test than in a integration test that covers half the project. Some criticism to the article: The "more unit testing" section reminds me of junior devs asking why they can't test private methods in Java. If I'm testing a unit, I want to test the contract it promises (in this case, a method that does some checks and then sends something). That the behavior is split between multiple methods is an implementation detail, and writing tests around that makes changes harder (now I can't refactor the methods without also having to update the tests, even if the contract doesn't change) and it doesn't even test the contract! (There's nothing that makes sure that the mail is actually sent - we could be testing methods that aren't used by anything but the test code) For the "easier to test IO" section: just don't. Your tests now depend on some in-memory implementation that will behave differently than the real thing. That's just mocking with extra steps, you still don't know whether your application will work. If you want to do io, do the real io "Separation of logic and IO": this is in general the right thing to do, but the way it's described is weird. First, it does the same as in the "more unit testing" section with the same problems. Then, the code is refactored until it's barely understandable and the article even admits it with the Greenspan quote. In the end, the production code is worse, just to ... Not test whether there's actually some code doing the IO. I actually think there are some good ideas in there: separating the logic from the IO (and treating them as separate units) is important, not just for better testability, but also for easier refactoring and (if done with care) to be easier to reason about. In the end, you will need both unit and integration tests (and if your system is large enough, e2e tests). Whether you're using mocks for your unit tests or not, doesn't make much of a difference in the grand picture. Just don't mock stuff in integration or e2e if you absolutely can't prevent it.

0xbadcafebee 2y ago

Mocks aren't an anti-pattern. Anti-patterns are a "common response to a recurring problem, usually ineffective, risking being highly counterproductive". On the contrary, mocks are a common response to a recurring problem which are often effective and have no greater risk than a great many alternative testing methodologies. They do solve problems and they are useful. But like literally anything else in the universe: it depends, and they don't solve every problem. You wanna know how to test without mocking? Use any kind of test. Seriously, just make a test. I don't care what kind of test it is, just have one. When you notice a problem your testing doesn't catch, improve your testing. Rinse, repeat. I don't care what kind of 10x rockstar uber-genius you think you are, you're going to be doing this anyway no matter what super amazing testing strategy you come up with, so just start on it now. Are there some ways of testing that are more effective than others? Yes, but it depends. If testing were simple, easy, straightforward and universal we wouldn't be debating how to do it. (about 99% of the time I'm disappointed in these clickbait blog posts upvoted on HN. they are shallow and brief (it's a blog post, not a book), yet quite often dismissive of perfectly reasonable alternatives, and in the absence of any other information, misleading. it would be better to just describe the problem and how the author solved it, and leave out the clickbaity sweeping generalizations and proclamations)

mvdtnz 2y ago

Frightening that there is someone out there calling themselves a CTO and offering CTO coaching who doesn't understand what unit testing is.

sevensor 2y ago

If you take the article's advice to move everything that's not IO into pure, testable code (which is good), what's left is code that does IO. What are you even testing when you call such a procedure? At that point, it's mostly calls into other people's code. Maybe that's a good place to draw the line on testing things?

binary132 2y ago

Never understood what was so hard about dependency inversion

kevwil 2y ago

"What to do instead of Mocking" should be more like "Enhance your use of Mocks with better unit tests and integration tests". The listed complaints sound more like problems with sloppy/lazy coding practices than actual problems with mocks.

worik 2y ago

Some testing frameworks (thinking of my experience with Flutter) insist on mocking From memory the HTTP client 404s every request in testing mode

alexfromapex 2y ago

It is so cringe to see bad advice like this being given. Yes, you can write mocks incorrectly. You should not model them after the "happy path" but you should make sure they cover the most use-cases both good and bad. I have been a senior or principal engineer on teams that did both of these approaches and the non-mocking approach is terrible because you end up with separate tests that have colliding data. It's slower using a real database back-end and becomes a mess and leads to issues where your database is heavily coupled to your test code which is the real anti-pattern. Then a year or two later when you want to change databases or database architectures you're screwed because you have to go into a bunch of tests manually and change things. The whole point of the mocks is it makes everything modular.

gchaincl 2y ago

> This often happens with an in-memory database, which is compatible to your main database you do use in production, but is faster. Not sure how this will solve edge cases problems described at the beginning of the article

nerder92 2y ago

Everytime I read about this kind of argument against mock is usually due to a misunderstanding on why unit tests with mocking exists in the first place. The underlying assumption is that tests are a quality assurance tool but I think that this is true only for E2E (possibly in production). In outside in TDD unit test are used as a design tool not a QA one, and mocking is a convenient way to quickly do that without the need of implementing the next layer, mock usually don’t replace an implemented service that does IO they implemented a noop that triggers an exception (so they your E2E won’t pass until you implement that) The problem is in the name, unit test should be called implementation spec or in-code documentation. Each layer of testing has its roles and serves a different purpose.

tpoacher 2y ago

Mocking is an indeed an anti pattern ... when dealing with tests that pretend to be unit tests but are not actually unit tests (e.g. needing to be aware if IO edge-cases, to quote the article). But tests that are not actually unit tests masquerading as unit tests and vice versa is arguably the bigger problem here. Not mocking per se.