4 ms·

When you test with mocks, most of the time you're not testing reality, you're testing your theory of reality. Think of it like this: How many times have you d

When you test with mocks, most of the time you're not testing reality, you're testing your theory of reality. Think of it like this: How many times have you deployed an app in production, or against production data, only to have it crash and burn because something somewhere in the data wasn't what you expected? No one would really be surprised by it happening, I think most of us have experienced it a time or two in our careers. Here's the thing: No seasoned developer is going to recommend that you write a migration process, create a DB with purely dev created test data, and if it passes, immediately pass it into production/into the hands of customers. Most experienced dev's would tell you that's dangerous, and you really need to test it against actual production data first. In this case, the dev created test data is a 'mock', and the production data is 'reality'. In this context, no one would disagree that the dev created test data is not enough in order to ensure proper functioning. But many people will tell you that a mock object is enough to ensure proper functioning. This is not true, and the investment required in order to make it true tends to be a lot higher than people expect for the same reason that battle tested code tends to be more complicated than most people would expect is necessary to solve the problem. That doesn't mean that mocking doesn't bring value, but because the cost of making that mock accurately mimic reality, it means you should reserve it for those times where the alternatives tend to be even more prohibitive. This is why you'll see a lot of recommendations for using mocks to do things like simulate failed connections at specific points in a protocol handshake. Getting the connection to break at the correct time using an actual network is prohibitively difficult. Getting a mock to do it consistently at the right time is a lot easier to do. These are the sorts of situations that mocks were originally created to solve. The issue is that somewhere along the line people started abusing mocks in situations that did not warrant them either because the cost of actually getting it correct is too high, or because they didn't actually pay the cost resulting in a false sense of security. I don't think any reasonable developer will tell you that mocks are worthless, but I also don't think any reasonable developer will tell you that you should be mocking a whole lot.

tunesmith 12y ago
Well, you're more explaining why not to mock, rather than what they mean by not mocking... but I have a hard time agreeing with this, for properly designed code. Referring to my earlier example of B calling C, what if B's business logic is complex, but makes no outbound calls except to C, and C (for simplicity's sake) is also complex, but returns only a boolean result? Mocking has real value there because you can control what C returns when testing B. You might write tests for if C returns true, if C returns false, if C returns null, or if C returns some general exception. There, you're testing that B behaves correctly, given all possible responses from C. At that point, you've controlled the interface between B and C, and limited the ways that C can respond. Because of that, you can be sure that if your test for B fails, it's because of a problem with B. If you instead don't mock C, then an integration test that tests both B and C (since B is actually calling C) becomes a lot more complex - probably more than twice as hard to nail down a bug. The trick there is that by focusing on the unit test, you've written a clear, simple interface between B and C, when you might not have otherwise. I think that's an example of how mocking can really help, and how it doesn't lead to the nightmare of mocks-calling-mocks that a lot of people seem to criticize. And rather than being only useful in cases like protocol handshakes, it seems frequently useful, such as in wanting to mock a dao layer when testing a business logic layer. At any rate, that makes it sound like by not mocking, they're merely doing multi-level integration tests. I just would have expected them to be more focused on unit tests, which to my mind requires mocking if you have a multi-layer application. Jim W. made the same point about "not mocking" in his youtube video though... and in the same breath said that he does, however, mock at the boundaries. Which makes me wonder if this is all just a terminology confusion. Because for instance, if I'm testing a public method in business logic, I'm not going to mock the private methods that it calls in that same business logic layer - that just seems crazy to me. And yet I would still call that a unit test, since it's still limited to one layer, if not to one single-layer-of-abstraction method. I will mock the dao layer in those cases, though, because that strikes me as a completely separate layer/concern. But I don't know, maybe someone else would say that I "don't mock all that often".
- mreiland 12y ago
  I didn't post that to give you something concrete to attack, I posted it because you claimed to not understand what they meant when they said they don't mock that often. I will respond to 1 point you made because I feel it's the point you're not understanding. > Referring to my earlier example of B calling C, what if B's business logic is complex, but makes no outbound calls except to C, and C (for simplicity's sake) is also complex, but returns only a boolean result? Theory and practice. In theory, the only thing you need to rely on is the return value. In practice, that code also relies on the runtime characteristics of the code it's calling, and this includes things like unexpected slowness, as well as unexpected exceptions (depending on the technology you're working in). This is the entire point about a mock being a theory of reality, and getting that right is difficult.

tunesmith 12y ago

Well, you're more explaining why not to mock, rather than what they mean by not mocking... but I have a hard time agreeing with this, for properly designed code. Referring to my earlier example of B calling C, what if B's business logic is complex, but makes no outbound calls except to C, and C (for simplicity's sake) is also complex, but returns only a boolean result? Mocking has real value there because you can control what C returns when testing B. You might write tests for if C returns true, if C returns false, if C returns null, or if C returns some general exception. There, you're testing that B behaves correctly, given all possible responses from C. At that point, you've controlled the interface between B and C, and limited the ways that C can respond. Because of that, you can be sure that if your test for B fails, it's because of a problem with B. If you instead don't mock C, then an integration test that tests both B and C (since B is actually calling C) becomes a lot more complex - probably more than twice as hard to nail down a bug. The trick there is that by focusing on the unit test, you've written a clear, simple interface between B and C, when you might not have otherwise. I think that's an example of how mocking can really help, and how it doesn't lead to the nightmare of mocks-calling-mocks that a lot of people seem to criticize. And rather than being only useful in cases like protocol handshakes, it seems frequently useful, such as in wanting to mock a dao layer when testing a business logic layer. At any rate, that makes it sound like by not mocking, they're merely doing multi-level integration tests. I just would have expected them to be more focused on unit tests, which to my mind requires mocking if you have a multi-layer application. Jim W. made the same point about "not mocking" in his youtube video though... and in the same breath said that he does, however, mock at the boundaries. Which makes me wonder if this is all just a terminology confusion. Because for instance, if I'm testing a public method in business logic, I'm not going to mock the private methods that it calls in that same business logic layer - that just seems crazy to me. And yet I would still call that a unit test, since it's still limited to one layer, if not to one single-layer-of-abstraction method. I will mock the dao layer in those cases, though, because that strikes me as a completely separate layer/concern. But I don't know, maybe someone else would say that I "don't mock all that often".

mreiland 12y ago
I didn't post that to give you something concrete to attack, I posted it because you claimed to not understand what they meant when they said they don't mock that often. I will respond to 1 point you made because I feel it's the point you're not understanding. > Referring to my earlier example of B calling C, what if B's business logic is complex, but makes no outbound calls except to C, and C (for simplicity's sake) is also complex, but returns only a boolean result? Theory and practice. In theory, the only thing you need to rely on is the return value. In practice, that code also relies on the runtime characteristics of the code it's calling, and this includes things like unexpected slowness, as well as unexpected exceptions (depending on the technology you're working in). This is the entire point about a mock being a theory of reality, and getting that right is difficult.