9 ms·
Why I don't mock
- clubhi 12y agoMocking helps with feature isolation. If a test failed you can be sure it was because of the use case of the test. Mocking helps when you are calling into code that you can't control. E.g, when calling into an API that is broken and sometimes returns incorrect results. I don't write very many tests anymore. I haven't mocked an object in years. I also have a lot of bugs in my code.
- mpweiher 12y agoMocking helps with feature isolation. By enabling isolation in the test, mocking reduces the pressure to design more isolated code. Mocking helps when you are calling into code that you can't control. Pervasive mocking helps isolate you from the pressure to remove (deep) dependencies on code that you can't control. Redesign your code so it only interacts with such code at the edges. At those places, create APIs that are independent of the code you can't control, and adapt that code to fit your APIs.
- pswenson 12y ago"Redesign your code so it only interacts with such code at the edges." yeah, and what if you don't have time to do this? sometimes mocks are appropriate because of schedules and other priorities. Much better to mock than to not write tests because mocking is not pure.
- eropple 12y agoQuit? Before it makes you quit? If your schedules and other priorities are against fixing broken code, I'd be leery of your organization's long-term health. If you're in for the long haul--and at my current gig I certainly am--then eliminating bad code (and any code as tightly coupled as what you describe) is the only sane option. Because it'll kill you eventually.
- SideburnsOfDoom 12y ago> By enabling isolation in the test, mocking reduces the pressure to design more isolated code. Having just written some tests (with mocks) today, I have read your comment several times and failed to extract any meaning whatsoever from it. It seems like the kind of comment that's counter-intuitive because it's just nonsense. Mocking reduces isolation by enabling isolation? Um. > Redesign your code so it only interacts with such code at the edges. Yes, and those edges can then be mocked when you want to test the rest of the code. Apis, interfaces etc. are a good fit to mocking. Mocking helps isolation. TFA's argument seems to be that they ended up not needing a database, and if they'd have "used mocking" they would have started by mocking the database, therefor mocking encourages bad design. Nope, starting with things that you don't need yet encourages bad design. It doesn't matter if you're mocking those things or not.
- the_af 12y ago> Pervasive mocking helps isolate you from the pressure to remove (deep) dependencies on code that you can't control. Redesign your code so it only interacts with such code at the edges. At those places, create APIs that are independent of the code you can't control, and adapt that code to fit your APIs. Yes... and that process you describe can be accomplished easily using mocks or stubs (depending on what you wish to test).
- mpdehaan2 12y agoSo I do a lot of systems programming type development. In these cases, whether you're PXE booting a server or talking to 500 different APIs that all range between the latest database or cloud API, flashing firmware on a RAID controller (shudder), whatever, there's a point of diminishing returns. There's also most of the bugs that you won't be able to simulate, because they come from external factors. The magic lies in automated integration tests. QA automation is AMAZING stuff. Set up real servers and real test environments when you can. Test real database upgrades. Mock testing can have it's place, but often you find yourself making internal API contracts more rigid than you want, which can limit refactoring.
- humanrebar 12y agoHow do you test your system's response to error conditions? Do you have hooks in your integrated system to force errors (throw exceptions, violate contracts, etc.)?
- mpdehaan2 12y agoIn some cases, yes. You can do things like set up scenarios you know won't work, such as connecting to machines that don't exist or sending the wrong parameters. The main thing is if the external inputs can't be made to "violate a contract", the internals aren't the problem. Ultimately, it depends on what you are building. Simple CRUD web applications are different from systems development type applications in many cases. Many of these classes can be more easily mocked out. Often the breaking change you'll see is an upstream API or SaaS endpoint changing in ways you didn't expect (or the difference between a Linux utility on the latest Ubuntu, an older RHEL, and Solaris), so it's often better to concentrate on testing the real thing. And these tests are expensive, so an automated test matrix that deploys a VM fleet with lots of permutations is usually necessary.
- pjc50 12y agoI too do systemsish programming in C++. All the places I've ever done this have "unit" tests that link in almost all of the executable and inject test data through the normal frontend. There's very little mocking; sometimes there will be a setFooForUnitTests() function to short-circuit some normal configuration path.
- InclinedPlane 12y agoA big danger with mocks is just shoving the problems under the rug rather than fixing them. Mocks represent assumptions, typically untested assumptions. There is generally nothing enforcing a mock's behaviors to be in accordance with whatever it is mocking, so that leaves it as a free parameter which can easily be incorrect if someone makes an error (which obviously can and does happen).
- tomstuart 12y agoThis isn’t a danger, it’s a benefit: your tests of one unit (whatever you consider a “unit” to be) are isolated from the behaviour of the rest of the system. For example, if you break one part of your implementation, you get one failing unit test, not a thousand. Checking these assumptions is what integration and acceptance tests are for.
- meric 12y agoI don't use mocks much, except with regard to the "now" method. There's some code that relies on the current time, I can use mock to mock many values of 'current time' instead of being forced to only test the case where the current time is when I run the unit test. This is especially important when dealing with code involving time zone and day light savings.
- mpweiher 12y agoMy experience with time is that whenever you build a system that is time-dependent in any meaningful way, you want to have "synthetic time" that you control, which you only feed with real time in a single, controllable place. Otherwise you get nasty effects, such as time moving on while your system is processing etc.
- meric 12y agoI'm guessing you mean like: def has_expired(now=None): #.. implementation instead of: def has_expired(): #.. implementation But at some point you need to call that method somewhere else... def get_request_handler(request): #...code now = timezone.now() if has_expired(now=now): raise Error403. And `get_request_handler` still needs to be tested, which I 'stub' the API for getting the current time in the correct timezone within the test for the request. (Taking into account the other comment), using a library called 'mock'. (https://pypi.python.org/pypi/mock https://pypi.python.org/pypi/mock)
- nahname 12y agoIf you are only changing the value of what an API returns, that is not a mock. It is a stub.
- blt 12y agoI agree. But another option is to make all the methods that would call now() take an extra parameter of the current time. Talk about making your dependencies explicit...
- 12y ago
- nope_42 12y agoI had a random idea the other day to create a program that monitors/logs system level calls and lets you replay them for a program. This would effectively let you use your program as normal, log any actual pieces that use external resources, and then automatically use the saved results as part of your test framework by overriding those system calls at the appropriate time. Is anyone aware of a solution like this? I certainly don't have time to create it but I would definitely find it useful.
- ap22213 12y agoIIRC, SoapUI does something similar.
- AlexanderDhoore 12y agoWe have something like this at the company I work for. We have a "simulation layer" which wraps ALL third party code. This lets us simulate the failure of file system calls, network drivers... And we use it to write our tests.
- lucaspiller 12y agoAlthough not quite the same, VCR for Ruby does this for network requests.
- phaedrus 12y agoI did this for a gdb wrapper I wrote. It logs all the stdin/stdout between itself and gdb and between itself and the user. You can then take that log file and replay it to simulate both the user input and the gdb output.
- giulianob 12y agoI really want to see people's complex projects and example code. There have been way too much opinion on this topic lately without any concrete evidence to show the strengths/weaknesses of using or not using mocks.
- pling 12y agoI can only give you an anecdote as our product is closed source. Mocks went like this, in a bell curve fashion: 1. Mocks are great, we'll use them everywhere. 2. Mocks are preventing our tests from being refactored. 3. Mocks are preventing our code form being refactored. 4. Mocks are preventing progress on the platform. 5. Code is being modified and tests and mocks are being removed to keep productivity up. 6. There are no mocks and few tests. 7. Tests increase again as defects are found. No mocks required. This is over the space of 8 years and our problem was the amount of coupling they caused.
- emsy 12y agoMocks don't cause coupling. Programmers do.
- revscat 12y agoCute, but even as terse as this comment is its sentiment is incorrect. Mocks, almost by their very nature, encourage coupling. Ask yourself the question "What tends to get mocked? " Answer: Objects which are difficult and/or heavy to instantiate, but which are outside of your direct control and cannot be changed, e.g.: database connections, network sockets, etc. So when you want to write tests around this you are almost forced to mock them out. So you instantiate those classes (as mocks) in your test class, so that the code under test. There are now two references to that class: one in the test, and one in the code being tested. This is coupling. QED.
- tomstuart 12y ago> Ask yourself the question "What tends to get mocked?" Answer: every collaborator of the object under test, so that only the tested object is actually having its implementation exercised. All other objects that it needs to talk to — database connections, network sockets, or whatever — are replaced with mocks, and the test subject’s interactions with those mocks are verified, so that a) we have confidence that the tested object is interacting with its collaborators in the expected way, and b) the test itself doesn’t depend at all on the actual behaviour of those collaborators’ implementations, which behaviour should be explicitly exercised elsewhere. That’s the opposite of coupling. I don’t understand where your answer comes from or what it means.
- lmm 12y agoMocks shouldn't be the first resort; they're there as a way to allow you to test code that you can't improve right now. I wouldn't want to use them on new code, but as a stepping-stone for testing and improving a large existing codebase they have their place.
- tomstuart 12y agoThat’s not what they’re for! Mocks allow you to design the interface of a new object by implementing that object’s collaborators before you implement the object itself. This is a very high-fidelity source of information about what the interface should look like; anything else is just (educated?) guessing.
- mattdeboard 12y agoThat may not be what they're "for" (wtf does that even mean?) but they're a damn good use of mocks. You can't fix everything right now.
- tomstuart 12y ago> That may not be what they're "for" (wtf does that even mean?) It means that “[mocks are] a way to allow you to test code that you can't improve right now” and “[mocks are] a stepping-stone for testing and improving a large existing codebase” is a mischaracterisation of the role and benefits of mocks. If using them like that is useful for you then that’s great, but it’s misleading to say that that’s why “they’re there”. Steve Freeman & Nat Pryce’s book [0], for example, is an excellent illustration of what mocks are for. [0] http://www.growing-object-oriented-software.com/ http://www.growing-object-oriented-software.com/
- lmm 12y agoA mock is overkill for that case; a stub or a dummy implementation (e.g. in-memory datastore) suffices when the interface with the collaborators is small and well-defined, as it should be when making a new design. The case where you need a full-on mock object is, IME, only when the collaborator is not decoupled enough from the primary object.
- ChrisGaudreau 12y agoMocking is the primary problem with TDD as it exists right now. "Isolated testing" (or isolated TDD) is advocated by people like Uncle Bob. Many developers listen to Uncle Bob and end up believing that unit testing without complete isolation is not "real" unit testing. Isolated unit testing is supposed to improve your designs. But when you mock everything, you end up with tests that know everything about your implementation (white box testing). The result is that refactoring breaks your tests. Your tests no longer document how to use the system; they are merely a mirror of its implementation. Kent Beck's central point in his book is that TDD mitigates fear, allows refactoring, and gives you immediate feedback. Extensive mocking makes developers fearful to refactor (thus hurting design) and reduces the quality of feedback. Not to mention it makes your tests hard to refactor too. Martin Fowler explains the difference between "classical" and "mockist" testing: http://martinfowler.com/articles/mocksArentStubs.html#ClassicalAndMockistTesting http://martinfowler.com/articles/mocksArentStubs.html#Classi...
- CMCDragonkai 12y agoI spent more time building my mocks instead of building my tests. Sometimes a file might have 90% mocks and 10% tests. Or maybe that's just my mocking framework...
- glenjamin 12y agoThe idea is that if you end up with code like this, it shows you have too many collaborators and should refactor them away.
- oelmekki 12y agoExactly. An other reason could be it's a main entry point for code execution, a class that only purpose is the glue others. In that case, having a lot of mocks is a code smell pointing that you should probably test this on the integration level rather than on the unit one (and thus, discard mocks and use "real" everything). (but beware not to be too much self indulgent on what is a gluing class, most of the time, they are not and it's indeed a god class problem).
- pswenson 12y agoGood points... but what happens when you walk into a legacy code base with few tests that wasn't designed to be tested and is tightly coupled to a component such as a database?
- mattdeboard 12y agoThere's a time and place for various tools. Mocking isn't all good, it's not all bad. Sometimes it's a good tool for accomplishing an immediate goal. Sometimes it's a crutch that encourages poor design. Keep an open mind and make evidence-based decisions. Only Siths deal in absolutes.
- crashedsnow 12y ago"Do or do not. There is no try". Absolutely.
- emsy 12y agoI found the article disappointing. Instead of giving reasonable arguments against mocking, the author offers an example where mocking was used wrong and therefore propagates that it shouldn't be used at all. The problem is not mocking. I found it to be absolutely helpful if used right (and right often means sparsely). The problem is this "all or nothing" mentality that many programmers have. Either they mock the hell out of the codebase or they don't use it at all. I don't use mock objects, unless it's message based business logic, often located at the topmost layer of abstraction.
- programminggeek 12y agoI think OOP is exposing serious flaws in how humans write software and solve problems. I'm not sure, but it seems there is an impedance mismatch in how we write code, and how we'd like to test code, but there is a reluctance or outright defiance toward the idea of writing testable code. The FP community doesn't seem to have a problem so much with testing simply because they mostly write pure functions that are data in -> data out and that is the easiest thing in the world to test. It occurs to me that most developers aren't writing testable OO code, and thus we have a never-ending argument about the most clever way to test (or not test) code that is not easily tested in isolation. If we never get around to writing testable code, we aren't going to have systems that are easy to test. If we don't have easy to test systems, testing is going to suck and people won't do it. I don't know that FP is the answer, but just asking people to write better OO systems isn't working. The problem certainly is more fundamental than simply TDD or mocking or whatever language or tool we are using. The problem is in how we look at solving problems in code and how that relates to system maintenance over time. It's a people problem, not a tooling one.
- keeperofdakeys 12y agoI don't think there is enough of an FP community doing large, heavily tested systems to allow accurate generalisations like that. I can think of plenty "We used FP, and everything went better than expected!" articles, but you probably don't hear from the people who tried, but came across problems.
- programminggeek 12y agoI agree, and I don't know that FP is the solution. I just know that the OO design approach isn't working super well with the people who are using it, and we now have a couple decades where design hasn't progressed significantly on the average project. So, how to we change the human behavior that is leading to less than ideal outcomes?
- camus2 12y agoSo, a lot of ideas are thrown here without any real use cases.People say "mocks are bad,whitebox testing is bad".But how can I unit test anything without mocks? let's take javascript.I cant generate mocks from my code,i have to write them.I have this data access object that takes a connnection. var dao=new Dao(connection); dao.getById(1,function(err,res){ console.log(res) }); Tell me how do I unit test getById in isolation without mocking connection,without doing whitebox testing,without knowing what actually happens in my getById. the obvious thing here is to spy on a hypothetic connection.query, fake a db call and return a predictable result.I cant do anything without a mock.
- john2x 12y agoI'm new to mocking (to serious unit testing, really), is mocking functions of 3rd-party libraries and/or web API's recommended? It sounds like a good idea, especially for functions that need to hit the network for a web API, but maybe I'm missing something?
- michaelmior 12y agoI would say it's generally a bad idea to mock 3rd party libraries. It's code that's not under your control and testing with a mock which may not match the actual behaviour could cause you trouble. In the case of network requests, something like VCR[1] helps a lot. [1] https://github.com/vcr/vcr https://github.com/vcr/vcr
- SideburnsOfDoom 12y agoI would disagree. If you want to unit-test your code that depends on those 3rd party libraries, then your choices are to: 1) Use the real thing. This can be slower and a lot less reliable. 2) Capture some typical output of the 3rd party api, and make a mock that emits it. Which technique is useful? Both are. But the second one gives you fast-running fine-grained repeatable unit tests so therefore is your first line of testing. So you discover in an integration test (#1) that the live api has a behaviour that the mock doesn't, maybe an occasional malformed response due to an error at the other end. Great, capture it and add it to the mock. Now you can reproduce it at will and write fast tests on how you handle it.
- derefr 12y ago3) Define a Gateway interface (e.g. DataStoreGateway) that presents exactly the API you want to consume; write one implementation of it that speaks to your third-party API (e.g. ODBCDataStoreGateway), and another that doesn't do much (e.g. MemoryDataStoreGateway). Note that a trivial gateway-implementation, like MemoryDataStoreGateway, isn't a mock! It's a fully-functional component that works perfectly well from its consumers' perspectives. But, unlike the version that consumes a third-party component, it doesn't have nearly any failure modes that would confuse your tests. It just does what it does, simply, and lets your tests test what you're trying to test. And note that when you stop mocking, and limit yourself to switching out fully-fledged gateway implementations, "dependency injection" stops being this huge pain with Factories and heavily-parameterized initializers. Instead, you can just make your objects discover their collaborators through a service registry. Your unit tests just register the trivial implementations in the service registry on setup(). --- † Don't get a bad taste in your mouth thinking about Spring here. A simpler, healthier example is Erlang's "global" module (http://erlang.org/doc/man/global.html http://erlang.org/doc/man/global.html).
- SideburnsOfDoom 12y agoTFA says "For example, when building SportStats v2 at the BBC we thought we needed a database for persistence. But we didn't build it in until we needed it, and we didn't mock it out either. We waited until the code told us that we now needed a database. It never did" So they seem to be saying "if you write code upfront before you need it, you might not need it, therefor mocking is bad" IMHO, that is not a valid conclusion to draw.
- the_af 12y agoI agree with you, that seems to be what the author is saying, and his conclusion doesn't follow. Also, I'm not sure if he really means mocking or any kind of test doubles. I thought everyone agreed that writing code upfront before you need it is always a bad idea, especially specific implementations like a database.
- deleted 12y ago[deleted]
- ytechie 12y agoI don't understand how you can be 100% against mocking. When I want to visualize testing, I often find it useful to think about how it's applied in the real world. For example, when building a car, the parts are tested in isolation (for the most part). Individual parts are tested - size, shape, stress, etc. Groups of parts are also tested in isolation. For example, the engine is actually connected to a mock drivetrain, mock computer, mock user inputs, etc. I see that as useful, and I think the same goes for software. It can simply be too costly in terms of time, bandwidth, etc to test against real resources.
- stinos 12y agoWhile I think I get where the OP is going, this seems just too easy to claim without any real-life code. So let's try this then. The last time I used mocks: we're interfacing with motor control hardware. In code we have interfaces for axis configuration, axis movement and so on. All implemented for different controllers. On top of that there's a layer defining specific movements. Say you want to go from point a to b while avoiding c, there's a piece of code that calculates a route (so it needs axes configurations for max velocity/range/...) and another piece that applies this route to a set of axes (so it needs axes to drive). Apart from dragging around a bunch of motors everywhere I code, how am I supposed to properly test those two pieces without any mocks? Telling me the design is all wrong to start with is fine, but then you also have to come up with a better one of course :). Telling me that route calculation shouldn't depend directly on axis configuration but should use it's own configuration type and I should then add a conversion between the two is also no solution as it a) introduces an extra layer and b) still requires a mock to test the conversion, d'uh :)
- deleted 12y ago[deleted]
- trustfundbaby 12y agoAs always theres a middle ground between the extremes of MOCK-ALL-TEH-THINGS or mock nothing!! Since reading Sandi Metz's very fantastic "Practical Object-oriented design in ruby" it clear to me that mocking is great for checking that objects other the one under test receive messages sent by the methods being tested (makes tests easier to setup and write usually). If I find that I'm doing too much mocking, then its usually because the method I'm using is doing way too much. FWIW.
- krazydad 12y agoWas hoping to read an admonition against excessive snark and bullying in Hacker News comments. Ah well.
- deleted 12y ago[deleted]
- webhat 12y agoI dislike the "that's not supposed to throw an exception" developer statement, if something is not supposed to throw an exception under normal operating you can almost guarantee that it will throw exceptions in the demo. So one of the few cases where I use lots of mock is in Exception Handling. Some exceptions are almost impossible to generate consistently in tests, or can't be simulated easily next to test that are supposed to pass when that exception state doesn't occur - think API failures, databases being yanked mid-query, etc.