4 ms·

Mocks Aren't Stubs (2007)

40 points by backslash_16 10y ago

born_of_dust 10y ago
Love this article.
taeric 10y ago
Every validation you do on a mock is an assertion on outside behavior that is in no way coupled with your test. If that behavior changes, your test is now false. Not failing. False. That concerns me.
- freshhawk 10y ago
  Yeah, maybe there is an approach that works well with mocks but this has always concerned me as well. It's rare (for me anyway) that I want to test which calls a function makes rather than if the function returns the correct result. I can imagine scenarios where that is useful and I know how to use mocks when that comes up ... but the examples are always a system where mock based testing is strangely coupled to implementation, very fragile and have the scary failure mode you describe.
  - shados 10y ago
    When you're testing functions in an impure language (pretty much any language, short of extremes like Elm, maybe Haskell, etc), functions are used not only for their return value but also their side effects. Side effects can be things like making an http request or writing to a file, it can be changing some global variable, but it can also be something like calling a public method of another public object. Since that function/method could also be impure, the very act of calling it is actually, in a way, part of the contract of the function under test: it returns a value, and it creates a side effect in the form of calling another function. The only real way to test for those side effects is to pass in a stub or mock and to check that the function was called correctly. That makes your tests brittle, but it's the only way to make sure it does what you think it does (even if you only call the function for an intermediate return value, the fact that changing the intermediate function could change the result of the function under test). Those tests are brittle because functions with side effects are, in fact, brittle, and favoring pure functions (where the only thing that matter is the return value, and dependencies are composed or passed as arguments) makes both your tests and your code better.
  - lojack 10y ago
    Typically, I find that I'm concerned with what calls a function makes when those calls interface things external to my program. I.e. I'm asserting that a request to a given http endpoint is made, or that an smtp client sends an email. I assume the external libraries work and am testing my interface to them.
    - taeric 10y ago
      This is where I find myself having to use these. Though, I greatly prefer to have the test merely check for the result of the call, if I can. e.g., if I have to post to a url to delete something, check to see if said something exists. In the end, it is not the call that I care about, but what the call was supposed to do. This way can lead to brittle tests, clearly. But the reality seems to be that brittle systems lead to brittle tests. There is no testing panacea.
- gautamdivgi 10y ago
  Why would it be of concern? I think mocks have a good place in unit tests. I generally believe the premise of unit tests is to test your code against specific pre-conditions. If those pre-conditions cannot be created in a unit testing environment then mocks are justified. I would, however, be very way of mocks moving past unit testing and into integration or system testing (or other varieties like failure injection testing, performance testing, etc.). Mocks have their uses & their limitations. Understanding that is key to using them well.
  - gohrt 10y ago
    > If those pre-conditions cannot be created in a unit testing environment then you have technical debt to repay, by making those pre-conditions satisfiable
  - wahnfrieden 10y ago
    It means that you lose out on a lot of value from your test suite: besides verifying behavior on new code, tests also allow you to confidently refactor existing code. If your tests test implementation via mocking, and break when you refactor your code in ways that preserve the intended behaviors, then you're unable to confidently refactor. Mocks lead you down a path where developers will eventually feel too scared or fatigued to refactor anything substantial.
cessor 10y ago
I often found that these terms are not very usefull when explaining the concepts of IoC in testing (which is essentially what they do) to colleagues who were unfamiliar with it. Programming in .NET I usually tried to use frameworks that avoided the terms, such as NSubstitute where you create an object by substitution: var format = Substitute.For<IFormatCode>(); I haven't used it in a while but I remember that Rhino Mocks asked for ridiculous code like var format = MockRepository.GenerateStub<IFormatCode>(); (Which one is it, a Mock or a Stub? And: Does it matter?) I recommend Gerard Meszaros excellent book XUnit Test Patterns, he makes more fine grained distinctions between different aspects of testing. I liked his notion that you simply create objects that produce or react to "indirect input and output" (i.e. exceptions, opcodes, sideeffects) rather than "direct input and output" (which is what "arrange" and "assert" are checking). Mocks and stubs are only realizations of different patterns. http://xunitpatterns.com/ http://xunitpatterns.com/
- freshhawk 10y ago
  That's an outstanding book. I'm not a Java programmer and that still might be the best book on testing, how to design tests, what to test and how think about testing at a high level that I've read.
dpark 10y ago
I feel like this distinction is arbitrary and unintuitive. A mock is literally just something that mimics something else. All "doubles" can be considered mocks. Claiming that mocks insist on call verification seems like a very forced categorization. If there's value in this distinction, maybe we need new terminology instead of asserting that everyone who's conflated "stubs" and "mocks" for two decades is wrong. As a side note, verifying call patterns makes for terrible, brittle tests that cost far too much to maintain. This pattern is great for enforcing call patterns that truly that matter for, e.g, performance reasons (no unexpected cascade of DB calls), but is typically just a giant pain that provides little to no value, because instead of testing that the code does the right thing, you're testing that it doesn't change.
- aikah 10y ago
  I think the difference is mocks carry tests while Stubs are dumb fakes. I think the problem is the choice of words. Maybe mocks should actually be called "spies" instead and stubs "fakes".
  - dpark 10y ago
    That terminology makes much more sense, especially the "spies" part. Personally, I tend to think of stubs as being super basic. A stub class for me returns default values or throws NotImplemented or something as simple as possible. It doesn't maintain meaningful state. Mocks for me are generally simplified implementations like an in-memory store that replaces a DB. Frameworks like Moq have muddied my definition though because they call things mocks when they only return canned data. Building a real mock (fake) with real state using Moq tends to be a pain.
  - gary_bernhardt 10y ago
    Spies are a different thing: they allow any interaction, recording it all so that you can inspect it layer. Python's "mock" library, for example, is mostly a spy library.
    - dpark 10y ago
backslash_16 10y ago
How do you test your code that interacts with external components, like sending emails to a mail server or making an external API call without using some sort of test fake or stub? (using the definition of them from the article) I'm new to TDD and refactoring legacy code for testability. With the code architecture I have it looks and feels like I need either a mock, fake, or stub. I'm using an IExternalComponent interface as a parameter and for the concrete implementation I can either pass the real thing (it's non deterministic, this works if the service is up) or I can pass a test double that allows me to throw exceptions at will and force failure conditions to be tested. Is there another way of doing this that I am missing?
ternaryoperator 10y ago
This article is almost universally cited when a book or article on testing or TDD begins the inevitable discussion of mocks/fakes/stubs. But in real life, this is a distinction without difference. I have never once heard someone use the term 'mock' and be misunderstood because he/she should've used 'stub'.

40 points by backslash_16 10y ago

Love this article.

Every validation you do on a mock is an assertion on outside behavior that is in no way coupled with your test. If that behavior changes, your test is now false. Not failing. False. That concerns me.

freshhawk 10y ago
Yeah, maybe there is an approach that works well with mocks but this has always concerned me as well. It's rare (for me anyway) that I want to test which calls a function makes rather than if the function returns the correct result. I can imagine scenarios where that is useful and I know how to use mocks when that comes up ... but the examples are always a system where mock based testing is strangely coupled to implementation, very fragile and have the scary failure mode you describe.
- shados 10y ago
  When you're testing functions in an impure language (pretty much any language, short of extremes like Elm, maybe Haskell, etc), functions are used not only for their return value but also their side effects. Side effects can be things like making an http request or writing to a file, it can be changing some global variable, but it can also be something like calling a public method of another public object. Since that function/method could also be impure, the very act of calling it is actually, in a way, part of the contract of the function under test: it returns a value, and it creates a side effect in the form of calling another function. The only real way to test for those side effects is to pass in a stub or mock and to check that the function was called correctly. That makes your tests brittle, but it's the only way to make sure it does what you think it does (even if you only call the function for an intermediate return value, the fact that changing the intermediate function could change the result of the function under test). Those tests are brittle because functions with side effects are, in fact, brittle, and favoring pure functions (where the only thing that matter is the return value, and dependencies are composed or passed as arguments) makes both your tests and your code better.
- lojack 10y ago
  Typically, I find that I'm concerned with what calls a function makes when those calls interface things external to my program. I.e. I'm asserting that a request to a given http endpoint is made, or that an smtp client sends an email. I assume the external libraries work and am testing my interface to them.
  - taeric 10y ago
    This is where I find myself having to use these. Though, I greatly prefer to have the test merely check for the result of the call, if I can. e.g., if I have to post to a url to delete something, check to see if said something exists. In the end, it is not the call that I care about, but what the call was supposed to do. This way can lead to brittle tests, clearly. But the reality seems to be that brittle systems lead to brittle tests. There is no testing panacea.
gautamdivgi 10y ago
Why would it be of concern? I think mocks have a good place in unit tests. I generally believe the premise of unit tests is to test your code against specific pre-conditions. If those pre-conditions cannot be created in a unit testing environment then mocks are justified. I would, however, be very way of mocks moving past unit testing and into integration or system testing (or other varieties like failure injection testing, performance testing, etc.). Mocks have their uses & their limitations. Understanding that is key to using them well.
- gohrt 10y ago
  > If those pre-conditions cannot be created in a unit testing environment then you have technical debt to repay, by making those pre-conditions satisfiable
- wahnfrieden 10y ago
  It means that you lose out on a lot of value from your test suite: besides verifying behavior on new code, tests also allow you to confidently refactor existing code. If your tests test implementation via mocking, and break when you refactor your code in ways that preserve the intended behaviors, then you're unable to confidently refactor. Mocks lead you down a path where developers will eventually feel too scared or fatigued to refactor anything substantial.

cessor 10y ago

I often found that these terms are not very usefull when explaining the concepts of IoC in testing (which is essentially what they do) to colleagues who were unfamiliar with it. Programming in .NET I usually tried to use frameworks that avoided the terms, such as NSubstitute where you create an object by substitution: var format = Substitute.For<IFormatCode>(); I haven't used it in a while but I remember that Rhino Mocks asked for ridiculous code like var format = MockRepository.GenerateStub<IFormatCode>(); (Which one is it, a Mock or a Stub? And: Does it matter?) I recommend Gerard Meszaros excellent book XUnit Test Patterns, he makes more fine grained distinctions between different aspects of testing. I liked his notion that you simply create objects that produce or react to "indirect input and output" (i.e. exceptions, opcodes, sideeffects) rather than "direct input and output" (which is what "arrange" and "assert" are checking). Mocks and stubs are only realizations of different patterns. http://xunitpatterns.com/ http://xunitpatterns.com/

freshhawk 10y ago
That's an outstanding book. I'm not a Java programmer and that still might be the best book on testing, how to design tests, what to test and how think about testing at a high level that I've read.

dpark 10y ago

I feel like this distinction is arbitrary and unintuitive. A mock is literally just something that mimics something else. All "doubles" can be considered mocks. Claiming that mocks insist on call verification seems like a very forced categorization. If there's value in this distinction, maybe we need new terminology instead of asserting that everyone who's conflated "stubs" and "mocks" for two decades is wrong. As a side note, verifying call patterns makes for terrible, brittle tests that cost far too much to maintain. This pattern is great for enforcing call patterns that truly that matter for, e.g, performance reasons (no unexpected cascade of DB calls), but is typically just a giant pain that provides little to no value, because instead of testing that the code does the right thing, you're testing that it doesn't change.

aikah 10y ago
I think the difference is mocks carry tests while Stubs are dumb fakes. I think the problem is the choice of words. Maybe mocks should actually be called "spies" instead and stubs "fakes".
- dpark 10y ago
  That terminology makes much more sense, especially the "spies" part. Personally, I tend to think of stubs as being super basic. A stub class for me returns default values or throws NotImplemented or something as simple as possible. It doesn't maintain meaningful state. Mocks for me are generally simplified implementations like an in-memory store that replaces a DB. Frameworks like Moq have muddied my definition though because they call things mocks when they only return canned data. Building a real mock (fake) with real state using Moq tends to be a pain.
- gary_bernhardt 10y ago
  Spies are a different thing: they allow any interaction, recording it all so that you can inspect it layer. Python's "mock" library, for example, is mostly a spy library.
  - dpark 10y ago

backslash_16 10y ago

How do you test your code that interacts with external components, like sending emails to a mail server or making an external API call without using some sort of test fake or stub? (using the definition of them from the article) I'm new to TDD and refactoring legacy code for testability. With the code architecture I have it looks and feels like I need either a mock, fake, or stub. I'm using an IExternalComponent interface as a parameter and for the concrete implementation I can either pass the real thing (it's non deterministic, this works if the service is up) or I can pass a test double that allows me to throw exceptions at will and force failure conditions to be tested. Is there another way of doing this that I am missing?

ternaryoperator 10y ago

This article is almost universally cited when a book or article on testing or TDD begins the inevitable discussion of mocks/fakes/stubs. But in real life, this is a distinction without difference. I have never once heard someone use the term 'mock' and be misunderstood because he/she should've used 'stub'.