8 ms·

Testing Without Mocks: A Pattern Language (2018)

55 points by etamponi 5y ago

tyingq 5y ago
"Test-specific production code. Some code needed for the tests is written as tested production code, particularly for infrastructure classes. It requires extra time to write and adds noise to class APIs." I don't see this much in real world code. Is there a space/niche that does this extensively? Does it raise a bunch of arguments about anti-patterns, etc?
- ravenstine 5y ago
  I've seen it (and, ahem, have written it...), but it's a rare sort of antipattern that I think mostly happens with inexperienced developers. I haven't had to do such a thing in several years except when there was some weird framework/library limitation where I was semi-forced to write a little test-specific code. Test-specific production code usually is a crutch to work within weird situations ultimately caused by dogmatic adherence to object oriented principles where there's the potential for inaccessible state all over the place; if something is getting in the way of tests that you only have coarse-grain control over then having `if (env == "test")` acting as a glorified `jmp` instruction might result in either making something testable or at least faster to test. Experienced developers that are at least halfway decent, rather than use such a crutch, instinctively consider whether the situation they are in is caused by stupid design and correct for the issue at a higher level; if they've always managed to think this way (i.e. like an engineer) then it may never occur to them why anyone would write test-specific code other than perhaps feature flags.
  - omegalulw 5y ago
    It's not always an antipattern. Another example would be a public API that exposes more state/info than it's users need, just for better tests. It's still kind of not the best practice, I will concede, since almost always you want to focus on testing the contract, but sometimes, exposing some extremely critical state info is ok to improve tests.
- geewee 5y ago
  I think it depends on where you draw the line. In C# it's very common to use interfaces even though you only ever expect to have a single class of the given type, just so you can stub it out in tests. It's unfortunate, but some languages require you to do at least some re-architecting so it's easy to test your code.
  - BurningFrog 5y ago
    WIthout reading the article, I'm on the "testability is a first class design criteria". Ease of testing should absolutely affect all your production code design choices. That said, there are terrible ways to do that, as always.
  - mosdl 5y ago
    Interfaces have other advantages as well, that is an added benefit. Implementing a No Op version of an interface for tests or a simple version that uses a map instead of db calles has a lot of benefits.
- bluGill 5y ago
  It is a lot eaiser to make an interface and throw a mocking framework at it. I've been thinking of similar ideas for a while, but implementing them is hard so I never tried it.
- eyelidlessness 5y ago
  In my personal experience I’ve been more prone to expose test specific code in implementation than my teammates, but even so: - it’s usually to export things which are project internal (and create references where they’re magic numbers etc otherwise) specifically to decouple tests - otherwise it’s been basically some lighter weight version of the null wrapper And no one has complained. Perhaps they should but I think it hasn’t been an issue because my devotion to automated testing has been more thorough and thoughtful than many of those who’d complain, and they’re happy to benefit from the safety that provides. Honestly I think test and implementation code should be more tightly coupled not less. At least in terms of being able to reference things that are clearly project-/package-private. I know this would bother a lot of people depending on discipline, but it’s pretty much exactly what people do in many Rust projects and seems pretty successful.
loevborg 5y ago
Enjoyed this much more than I thought I would • Shows (almost) real-world examples to demonstrate the patterns • Uses JavaScript, which is notoriously hard to test because of complex UI frameworks and difficult-to-mock APIs • Set of patterns that refer to one another, providing mutual support Would love to read more—any other resource along similar lines appreciated
- omegalulw 5y ago
  I found this to have many questionable claims for example: > Ensure that all Logic classes can be constructed without providing any parameters (and without using a DI framework). In practice, this means that most objects instantiate their dependencies in their constructor by default, although they may also accept them as optional parameters. When you do the latter, it kind of defeats the purpose of dependency injection and what you are testing won't be what you use in prod. And when you don't do this, you have to set up dependencies properly - at that point it's almost becomes an integration test. IMO the right thing here is to mock dependencies in unit tests so you can properly test the class in question. Fundamentally, it seems like the author hasn't worked on or considered complex distributed systems where stuff like dependency management, sequencing (back and forth interactions), etc really matter - you can't test those properly in unit tests, you need integration tests. Unit tests and integration tests complement each other, you can't reasonably test and update complex systems without both.
- loevborg 5y ago
  I found that James has recorded a series of 22 TDD sessions that demonstrate the techniques outlined in the blog post in JS: https://www.youtube.com/watch?v=nlGSDUuK7C4&list=PLD-LK0HSm0Hpp-OspFpZ32uY766YntGVQ&index=1&ab_channel=JamesShore https://www.youtube.com/watch?v=nlGSDUuK7C4&list=PLD-LK0HSm0...
- eyelidlessness 5y ago
  > Uses JavaScript, which is notoriously hard to test because of complex UI frameworks and difficult-to-mock APIs It really depends. I’ve found testing JS/TS to range from “damn this is not worth automating” to “this is how it should be”. Obviously UI frameworks may fall in the former category, but IME this is a place where JSX really shines: it’s an unopinionated DSL, perfectly fit to stub however you see fit. Much more complex is testing DOM and event loop behavior. Server-side (non-presentational) JS/TS may be difficult to test, but that’s really more an accident of history than anything. It almost all depends on the Express API or something like it, and it’s damn near impossible to reason about at a unit level or as a statically analyzable system. But it doesn’t have to be that way, and I’ve developed complex systems without this problem, to the point unit tests provide enough confidence that few if any integration tests were of any value. Which in the article’s parlance means nearly all tests other than internal library code were testing pure logic functions or infrastructure functions. (I’m still sad I wasn’t able to open source this library code but a new implementation is on my side project list.) > Set of patterns that refer to one another, providing mutual support I first skimmed the article then re-read it, because I found this challenging as a reader trying to keep track of my cross-reference stack. It was better on a second pass, but I do think there’s an opportunity to inline some linked information at the expense of redundancy to minimize that, and to treat this format as more of a reference backing the article.
ed-209 5y ago
Spoiler - expososing test data from the implementation: https://github.com/jamesshore/testing-without-mocks-example/blob/6e5e137e25933308b0146cc844aa01b561cd6986/src/command_line.js#L28 https://github.com/jamesshore/testing-without-mocks-example/...
- eyelidlessness 5y ago
  This and other cases of test-specific implementation code is also specifically called out under tradeoffs in the article FWIW.
karlmdavis 5y ago
I’m really not clear what’s going on with the null implementations… aren’t those just very boring mocks?
- eyelidlessness 5y ago
  That’s my impression as well. Almost always when people advocate narrow tests without mocks they end up using something that’s not called a mock but ultimately isn’t distinguishable from a mock. Which I don’t think necessarily invalidates the article (which I’ve only skimmed so far, but already strikes me as having a lot going for it), just some terminology used. It’s perfectly reasonable, regardless of terminology, to say: here are some techniques that encourage testability and high confidence in those tests. And it appears to me there’s a lot of that in the article. So with that said, I’m gonna go give it a more thorough read :)
- xmcqdpt2 5y ago
  Same, it seems to me that the article is based on the idea that val app = new AppClass(new MockServer) assert(...) is bad but val app = new AppClass(Server.createMock()) assert(...) is good. I for one see no difference.
- etamponi 5y ago
  (Poster here) Agreed. Perhaps what differentiate those from "normal" mocks is: 1) They only apply to "infrastructure" (so databases, external services etc). Basically the article says: "It's okay to use mocks sparingly, to avoid costly and flaky tests involving external services". 2) They are defined along with the production code, and _tested accordingly_. Which I assume means that you test the "null implementations"/"infrastructure mocks" by making sure they are always aligned with the real interactions. This avoids the danger of "normal mocks" to diverge from the real interactions. What I would like to understand better, instead, is if there is a way to avoid all of this system to break once the infrastructure is two layers below the code you're testing. Because you can inject a null implementation into ApplicationClass, but what you have a UpperApplicationClass which depends on ApplicationClass? You either have to unravel the dependencies yourself to inject NullImplementation inside ApplicationClass inside UpperApplicationClass (which defies the point of the article, in a way), or you have to resort to a DI framework, which I personally abhor (I've still to find a single case in which it is a net benefit and not just a magic way of obfuscating the code).
bcbrown 5y ago
This is almost exactly my approach to testing. I'm going to have to save this for sharing with teammates.
chezzwizz 5y ago
I felt too compelled to stick with existing design pattern theories and maintain a sort of persistent stubbornness with traditional xUnit style tests and TDD rather than follow the rabbit on this article.
jdlshore 5y ago
Hey all, author here. I'm a bit late to the party but happy to answer any questions you have. The main question seems to be: what's the difference between null implementations and mocks? There's a practical answer to that question and a technical answer. Practically speaking, the reason to use nullable infrastructure wrappers is that they enable sociable tests. A flaw of mock-based testing is that you only check that your system under test (SUT) calls its dependencies, but not that your code's expectations of its dependencies are correct. Sociable tests check that the SUT uses its dependencies correctly. This is valuable when refactoring, because it's easy to inadvertently introduce changes to behavior that break callers. Mock-based tests ("solitary tests") can't detect those errors. Technically speaking, the nullable infrastructure wrappers convert your tests from interaction-based ("did I call the right methods") to state-based ("did my system return the correct results or make the appropriate state changes"). They do this by adding state inspection capabilities to production code, allowing you to assert on the state of the production code rather than asserting which methods were called. They also implement a tiny hidden stub to enable you to "turn off" interactions with the outside world. My approach and mocks/spies are superficially similar, in that they both make use of test doubles (stubs, mocks, and spies are all test doubles) but they lead to significantly different testing approaches. Mocks/spies lead to interaction-based solitary tests, and nullable infrastructure leads to state-based sociable tests. I also find the tests are easier to write and read, and they're orders of magnitude faster¹ than using a mocking framework, too. ¹In a head-to-head comparison, nullable infrastructure ran 1,075,268 tests/sec, testdouble.js ran 12,210 tests/sec, and sinon.js ran 2,793 tests/sec. https://github.com/jamesshore/livestream/blob/2020-05-26-end/src/_comparisons_test.js https://github.com/jamesshore/livestream/blob/2020-05-26-end...
- etamponi 5y ago
  Thank you for your clarification (and for the article). Could you elaborate a bit more? I see the point of sociable tests, but I don't see how nullable infrastructure does not break the pattern itself. Or, actually, I only see one way in which it does not break sociable tests: if the null implementation is tested against the _real_ implementation, guaranteeing that it does not drift away. At this point, using the null version of a wrapper in another test is guaranteed to test the correct interaction with the infrastructure. Is this it, or is there something else that I am missing?

55 points by etamponi 5y ago

"Test-specific production code. Some code needed for the tests is written as tested production code, particularly for infrastructure classes. It requires extra time to write and adds noise to class APIs." I don't see this much in real world code. Is there a space/niche that does this extensively? Does it raise a bunch of arguments about anti-patterns, etc?

ravenstine 5y ago
I've seen it (and, ahem, have written it...), but it's a rare sort of antipattern that I think mostly happens with inexperienced developers. I haven't had to do such a thing in several years except when there was some weird framework/library limitation where I was semi-forced to write a little test-specific code. Test-specific production code usually is a crutch to work within weird situations ultimately caused by dogmatic adherence to object oriented principles where there's the potential for inaccessible state all over the place; if something is getting in the way of tests that you only have coarse-grain control over then having `if (env == "test")` acting as a glorified `jmp` instruction might result in either making something testable or at least faster to test. Experienced developers that are at least halfway decent, rather than use such a crutch, instinctively consider whether the situation they are in is caused by stupid design and correct for the issue at a higher level; if they've always managed to think this way (i.e. like an engineer) then it may never occur to them why anyone would write test-specific code other than perhaps feature flags.
- omegalulw 5y ago
  It's not always an antipattern. Another example would be a public API that exposes more state/info than it's users need, just for better tests. It's still kind of not the best practice, I will concede, since almost always you want to focus on testing the contract, but sometimes, exposing some extremely critical state info is ok to improve tests.
geewee 5y ago
I think it depends on where you draw the line. In C# it's very common to use interfaces even though you only ever expect to have a single class of the given type, just so you can stub it out in tests. It's unfortunate, but some languages require you to do at least some re-architecting so it's easy to test your code.
- BurningFrog 5y ago
  WIthout reading the article, I'm on the "testability is a first class design criteria". Ease of testing should absolutely affect all your production code design choices. That said, there are terrible ways to do that, as always.
- mosdl 5y ago
  Interfaces have other advantages as well, that is an added benefit. Implementing a No Op version of an interface for tests or a simple version that uses a map instead of db calles has a lot of benefits.
bluGill 5y ago
It is a lot eaiser to make an interface and throw a mocking framework at it. I've been thinking of similar ideas for a while, but implementing them is hard so I never tried it.
eyelidlessness 5y ago
In my personal experience I’ve been more prone to expose test specific code in implementation than my teammates, but even so: - it’s usually to export things which are project internal (and create references where they’re magic numbers etc otherwise) specifically to decouple tests - otherwise it’s been basically some lighter weight version of the null wrapper And no one has complained. Perhaps they should but I think it hasn’t been an issue because my devotion to automated testing has been more thorough and thoughtful than many of those who’d complain, and they’re happy to benefit from the safety that provides. Honestly I think test and implementation code should be more tightly coupled not less. At least in terms of being able to reference things that are clearly project-/package-private. I know this would bother a lot of people depending on discipline, but it’s pretty much exactly what people do in many Rust projects and seems pretty successful.

loevborg 5y ago

Enjoyed this much more than I thought I would • Shows (almost) real-world examples to demonstrate the patterns • Uses JavaScript, which is notoriously hard to test because of complex UI frameworks and difficult-to-mock APIs • Set of patterns that refer to one another, providing mutual support Would love to read more—any other resource along similar lines appreciated

omegalulw 5y ago
I found this to have many questionable claims for example: > Ensure that all Logic classes can be constructed without providing any parameters (and without using a DI framework). In practice, this means that most objects instantiate their dependencies in their constructor by default, although they may also accept them as optional parameters. When you do the latter, it kind of defeats the purpose of dependency injection and what you are testing won't be what you use in prod. And when you don't do this, you have to set up dependencies properly - at that point it's almost becomes an integration test. IMO the right thing here is to mock dependencies in unit tests so you can properly test the class in question. Fundamentally, it seems like the author hasn't worked on or considered complex distributed systems where stuff like dependency management, sequencing (back and forth interactions), etc really matter - you can't test those properly in unit tests, you need integration tests. Unit tests and integration tests complement each other, you can't reasonably test and update complex systems without both.
loevborg 5y ago
I found that James has recorded a series of 22 TDD sessions that demonstrate the techniques outlined in the blog post in JS: https://www.youtube.com/watch?v=nlGSDUuK7C4&list=PLD-LK0HSm0Hpp-OspFpZ32uY766YntGVQ&index=1&ab_channel=JamesShore https://www.youtube.com/watch?v=nlGSDUuK7C4&list=PLD-LK0HSm0...
eyelidlessness 5y ago
> Uses JavaScript, which is notoriously hard to test because of complex UI frameworks and difficult-to-mock APIs It really depends. I’ve found testing JS/TS to range from “damn this is not worth automating” to “this is how it should be”. Obviously UI frameworks may fall in the former category, but IME this is a place where JSX really shines: it’s an unopinionated DSL, perfectly fit to stub however you see fit. Much more complex is testing DOM and event loop behavior. Server-side (non-presentational) JS/TS may be difficult to test, but that’s really more an accident of history than anything. It almost all depends on the Express API or something like it, and it’s damn near impossible to reason about at a unit level or as a statically analyzable system. But it doesn’t have to be that way, and I’ve developed complex systems without this problem, to the point unit tests provide enough confidence that few if any integration tests were of any value. Which in the article’s parlance means nearly all tests other than internal library code were testing pure logic functions or infrastructure functions. (I’m still sad I wasn’t able to open source this library code but a new implementation is on my side project list.) > Set of patterns that refer to one another, providing mutual support I first skimmed the article then re-read it, because I found this challenging as a reader trying to keep track of my cross-reference stack. It was better on a second pass, but I do think there’s an opportunity to inline some linked information at the expense of redundancy to minimize that, and to treat this format as more of a reference backing the article.

ed-209 5y ago

Spoiler - expososing test data from the implementation: https://github.com/jamesshore/testing-without-mocks-example/blob/6e5e137e25933308b0146cc844aa01b561cd6986/src/command_line.js#L28 https://github.com/jamesshore/testing-without-mocks-example/...

eyelidlessness 5y ago
This and other cases of test-specific implementation code is also specifically called out under tradeoffs in the article FWIW.

karlmdavis 5y ago

I’m really not clear what’s going on with the null implementations… aren’t those just very boring mocks?

eyelidlessness 5y ago
That’s my impression as well. Almost always when people advocate narrow tests without mocks they end up using something that’s not called a mock but ultimately isn’t distinguishable from a mock. Which I don’t think necessarily invalidates the article (which I’ve only skimmed so far, but already strikes me as having a lot going for it), just some terminology used. It’s perfectly reasonable, regardless of terminology, to say: here are some techniques that encourage testability and high confidence in those tests. And it appears to me there’s a lot of that in the article. So with that said, I’m gonna go give it a more thorough read :)
xmcqdpt2 5y ago
Same, it seems to me that the article is based on the idea that val app = new AppClass(new MockServer) assert(...) is bad but val app = new AppClass(Server.createMock()) assert(...) is good. I for one see no difference.
etamponi 5y ago
(Poster here) Agreed. Perhaps what differentiate those from "normal" mocks is: 1) They only apply to "infrastructure" (so databases, external services etc). Basically the article says: "It's okay to use mocks sparingly, to avoid costly and flaky tests involving external services". 2) They are defined along with the production code, and _tested accordingly_. Which I assume means that you test the "null implementations"/"infrastructure mocks" by making sure they are always aligned with the real interactions. This avoids the danger of "normal mocks" to diverge from the real interactions. What I would like to understand better, instead, is if there is a way to avoid all of this system to break once the infrastructure is two layers below the code you're testing. Because you can inject a null implementation into ApplicationClass, but what you have a UpperApplicationClass which depends on ApplicationClass? You either have to unravel the dependencies yourself to inject NullImplementation inside ApplicationClass inside UpperApplicationClass (which defies the point of the article, in a way), or you have to resort to a DI framework, which I personally abhor (I've still to find a single case in which it is a net benefit and not just a magic way of obfuscating the code).

bcbrown 5y ago

This is almost exactly my approach to testing. I'm going to have to save this for sharing with teammates.

chezzwizz 5y ago

I felt too compelled to stick with existing design pattern theories and maintain a sort of persistent stubbornness with traditional xUnit style tests and TDD rather than follow the rabbit on this article.

jdlshore 5y ago

Hey all, author here. I'm a bit late to the party but happy to answer any questions you have. The main question seems to be: what's the difference between null implementations and mocks? There's a practical answer to that question and a technical answer. Practically speaking, the reason to use nullable infrastructure wrappers is that they enable sociable tests. A flaw of mock-based testing is that you only check that your system under test (SUT) calls its dependencies, but not that your code's expectations of its dependencies are correct. Sociable tests check that the SUT uses its dependencies correctly. This is valuable when refactoring, because it's easy to inadvertently introduce changes to behavior that break callers. Mock-based tests ("solitary tests") can't detect those errors. Technically speaking, the nullable infrastructure wrappers convert your tests from interaction-based ("did I call the right methods") to state-based ("did my system return the correct results or make the appropriate state changes"). They do this by adding state inspection capabilities to production code, allowing you to assert on the state of the production code rather than asserting which methods were called. They also implement a tiny hidden stub to enable you to "turn off" interactions with the outside world. My approach and mocks/spies are superficially similar, in that they both make use of test doubles (stubs, mocks, and spies are all test doubles) but they lead to significantly different testing approaches. Mocks/spies lead to interaction-based solitary tests, and nullable infrastructure leads to state-based sociable tests. I also find the tests are easier to write and read, and they're orders of magnitude faster¹ than using a mocking framework, too. ¹In a head-to-head comparison, nullable infrastructure ran 1,075,268 tests/sec, testdouble.js ran 12,210 tests/sec, and sinon.js ran 2,793 tests/sec. https://github.com/jamesshore/livestream/blob/2020-05-26-end/src/_comparisons_test.js https://github.com/jamesshore/livestream/blob/2020-05-26-end...

etamponi 5y ago
Thank you for your clarification (and for the article). Could you elaborate a bit more? I see the point of sociable tests, but I don't see how nullable infrastructure does not break the pattern itself. Or, actually, I only see one way in which it does not break sociable tests: if the null implementation is tested against the _real_ implementation, guaranteeing that it does not drift away. At this point, using the null version of a wrapper in another test is guaranteed to test the correct interaction with the infrastructure. Is this it, or is there something else that I am missing?