3 ms·

I’ve recently learned that some actually do consider mocking harmful. Where I work right now, there’s a really outdated and unfashionable fight over the benefi

by _ix 7y ago

I’ve recently learned that some actually do consider mocking harmful. Where I work right now, there’s a really outdated and unfashionable fight over the benefit of unit testing in general. Existing engineers don’t see value in test driven development, exhaustive testing, unit testing, and mocks/spies are thrown in... and we’re a python shop. I’m utterly confused. I, too, grew concerned after reading–will I be hearing this cited/twisted as further evidence against investing in our dreadful testing situation?

mikekchar 7y ago
I'm a died in the wool TTDer, but I actually think that mocking unnecessarily is usually harmful. I use it as a technique of last resort. I should define my terms before I explain, because many people use the term "mock" to mean things it didn't originally mean. Test objects that are used in place of domain level objects were traditionally known as "fakes". A fake that represented a fixed known value was called a "stub". A fake that included an assertion that a function was called (or that collected data on function calling) was called a "mock". It's a bit confusing for me that many people use the term "mock" to mean "fake". I found it weird that the original article pointed to an article on faking and then used the term "mock" without referred to the original meaning of the that word (which makes me wonder what they mean when they say "fake"). Anyway, usually you want a fake when you don't have access to some part of the system to test it directly. Sometimes that's because it's a completely different service. You can fake out that service so that you can see if the code that interacts with that service is working, without having to actually set up the service. A stub is useful in situations where you need to know that your code is working with specific values of data inputs. So you might have an object that you pass to a function and you want to know what happens if one of the properties on the object is null. It might be hard to set that up, so you can stub it out. A mock on the other hand, basically tests if a function is called. A good example where you might legitimately need a mock is where you pass an object to a function and you are expecting that a callback on that object will be called. It's really hard to test that without a mock. Where mocks can be dangerous is when you completely mock out any interfaces and stub the return values. You pass a fake object as a collaborator to your function and you test that your function works. The problem is that your fake object may not necessarily represent a real object in the system. If you ever want to refactor the code, your tests will no longer tell you that a property is missing, or that a function is missing because all of your test code is using fake objects with mocked and stubbed methods. Ideally a unit test that uses an interface should fail when you change that interface. This allows you simply to change an interface somewhere and have your tests tell you exactly what you need to do to make that change work. Where you end up getting a lot of conflict WRT testing strategy is that some people believe very strongly that unit tests should test things in isolation. Secondly people believe that unit testing should be a black box testing strategy. So you should test through your public interfaces only and any collaborators that adheres to the interface contract should work as expected. In this style of testing, you are often encouraged to mock anything and everything at the interface boundary. This has many advantages. First, it means that your test objects can be very simple, so writing tests is very quick -- even if the code in the system is complex (because you aren't using any of that code). Second, because you are testing the public interface only and there is minimal setup, your tests become documentation of the interface contracts. Third, because it is black box, if you change the implementation of your "unit", you don't have to change your tests. Despite these benefits, I'm not a big fan of this style. I like white box testing using real collaborators. My goal is not to define interfaces and nail them up -- quite the opposite. I want to be able to change interfaces fluidly. I value ease of refactoring over just about anything else. Second, I want to use real collaborators almost because it is painful. If your collaborator is awkward and brittle to set up in tests, it is also awkward and brittle to set up in production code. My goal is to remove that and to simply the code. Again, my highest value is my ability to refactor the code. I want the code to become easier to work with over time, not harder and more complex. Finally, I want to code to break at a "white box" level, not a "black box" level when I change behaviour. Ideally, I want my tests to say, "On the third line of that function, we're going to have a problem because that function is different now". I don't want to be aware of problems at a larger scope "Somewhere in function A that calls function B which calls function C and D there is something wrong because it does something weird". In the end I write small functions that are tested directly with real collaborators. I avoid private functions because it hides my implementation details. I test at a low level so that I avoid test complexity from excess branching. I get incredible specificity from failing tests, when end up essentially giving me a TODO list for what I need to do when refactoring code. Hope that gives you some idea of at least why one person avoids mocking -- although, you do need it sometimes. And to be fair, sometimes I'll do a London School, outside in, mock the world implementation if I'm not sure what I'm building. However, I throw away all my mocks and re-TDD once I know what I'm building.
- chriswarbo 7y ago
  I agree with basically all of this. > some people believe very strongly that unit tests should test things in isolation. Secondly people believe that unit testing should be a black box testing strategy. So you should test through your public interfaces only and any collaborators that adheres to the interface contract should work as expected. I think much of the confusion and talking-past-each-other comes from ambiguous language. I actually agree with all the things in the above quote (isolation, black-box, public-only, relying only on specified interfaces). Where I've differed from co-workers is that I consider the appropriate "unit" to be a feature/piece-of-functionality (e.g. "logging in"), whereas they consider the appropriate "unit" to be a piece of code (e.g. a method or class). I had a bit of a rant about this at http://chriswarbo.net/blog/2017-11-10-unit_testing_terminology.html http://chriswarbo.net/blog/2017-11-10-unit_testing_terminolo...
  - blacksmythe 7y ago
    The talk that you link to: https://www.infoq.com/presentations/tdd-original/ https://www.infoq.com/presentations/tdd-original/ Ian Cooper reminds what was Kent's original proposition on TDD, what misunderstandings occurred along the way, and suggests a better approach to TDD, one that supports development rather impeding it.
  - mikekchar 7y ago
    I think Michael Feathers explained it the best. He likened unit testing to clamping a piece of woodwork while you are working on it. The bits you are working on need to be in motion because you are working on them. The bits you are not working on them need to be clamped in place -- you don't want those things moving while you are working on some other bit. A "unit" is anything you might want to be clamped in place. It can be a function. It can be an object. It can be a subsystem. You want to unit test at different levels of abstraction so that you can "clamp" those levels of abstraction down. One of the things I've found people get confused with is that they see unit testing and integration testing as orthogonal. They think a unit test should exercise a small piece of code in isolation and an integration test should test examples of real collaborators. Frequently they mock out all their unit tests and write a few integration tests. Then their unit tests become brittle and annoying and so they delete them, leaving only a few integration tests. This leads a lot of people with the impression that only integration tests are useful. If we can back up and redefine "unit test", then the problem disappears. I read your "rant". Don't even get me started on BDD :-) Originally people had problems understanding the purpose of TDD because the word "test" had them confused. They would think, "I need to write tests to ensure that this is working". They didn't think about it in terms of clamping the behaviour so that it doesn't change when you are working on another part of the system. For that reason, a lot of people discussed changing the word "test" to something else that truly embodied what TDD was all about. Many people hit on the word "behavior" -- you want to document the current behaviour of your "units" (at different levels of abstraction). Somehow this got totally confused with automated acceptance testing! Now we have things like cucumber (which I don't actually hate, but it accomplishes a completely different goal than TDD!) What really frustrates me is when I talk to people about this stuff and they think I'm a complete lunatic :-)

by _ix 7y ago

mikekchar 7y ago

I'm a died in the wool TTDer, but I actually think that mocking unnecessarily is usually harmful. I use it as a technique of last resort. I should define my terms before I explain, because many people use the term "mock" to mean things it didn't originally mean. Test objects that are used in place of domain level objects were traditionally known as "fakes". A fake that represented a fixed known value was called a "stub". A fake that included an assertion that a function was called (or that collected data on function calling) was called a "mock". It's a bit confusing for me that many people use the term "mock" to mean "fake". I found it weird that the original article pointed to an article on faking and then used the term "mock" without referred to the original meaning of the that word (which makes me wonder what they mean when they say "fake"). Anyway, usually you want a fake when you don't have access to some part of the system to test it directly. Sometimes that's because it's a completely different service. You can fake out that service so that you can see if the code that interacts with that service is working, without having to actually set up the service. A stub is useful in situations where you need to know that your code is working with specific values of data inputs. So you might have an object that you pass to a function and you want to know what happens if one of the properties on the object is null. It might be hard to set that up, so you can stub it out. A mock on the other hand, basically tests if a function is called. A good example where you might legitimately need a mock is where you pass an object to a function and you are expecting that a callback on that object will be called. It's really hard to test that without a mock. Where mocks can be dangerous is when you completely mock out any interfaces and stub the return values. You pass a fake object as a collaborator to your function and you test that your function works. The problem is that your fake object may not necessarily represent a real object in the system. If you ever want to refactor the code, your tests will no longer tell you that a property is missing, or that a function is missing because all of your test code is using fake objects with mocked and stubbed methods. Ideally a unit test that uses an interface should fail when you change that interface. This allows you simply to change an interface somewhere and have your tests tell you exactly what you need to do to make that change work. Where you end up getting a lot of conflict WRT testing strategy is that some people believe very strongly that unit tests should test things in isolation. Secondly people believe that unit testing should be a black box testing strategy. So you should test through your public interfaces only and any collaborators that adheres to the interface contract should work as expected. In this style of testing, you are often encouraged to mock anything and everything at the interface boundary. This has many advantages. First, it means that your test objects can be very simple, so writing tests is very quick -- even if the code in the system is complex (because you aren't using any of that code). Second, because you are testing the public interface only and there is minimal setup, your tests become documentation of the interface contracts. Third, because it is black box, if you change the implementation of your "unit", you don't have to change your tests. Despite these benefits, I'm not a big fan of this style. I like white box testing using real collaborators. My goal is not to define interfaces and nail them up -- quite the opposite. I want to be able to change interfaces fluidly. I value ease of refactoring over just about anything else. Second, I want to use real collaborators almost because it is painful. If your collaborator is awkward and brittle to set up in tests, it is also awkward and brittle to set up in production code. My goal is to remove that and to simply the code. Again, my highest value is my ability to refactor the code. I want the code to become easier to work with over time, not harder and more complex. Finally, I want to code to break at a "white box" level, not a "black box" level when I change behaviour. Ideally, I want my tests to say, "On the third line of that function, we're going to have a problem because that function is different now". I don't want to be aware of problems at a larger scope "Somewhere in function A that calls function B which calls function C and D there is something wrong because it does something weird". In the end I write small functions that are tested directly with real collaborators. I avoid private functions because it hides my implementation details. I test at a low level so that I avoid test complexity from excess branching. I get incredible specificity from failing tests, when end up essentially giving me a TODO list for what I need to do when refactoring code. Hope that gives you some idea of at least why one person avoids mocking -- although, you do need it sometimes. And to be fair, sometimes I'll do a London School, outside in, mock the world implementation if I'm not sure what I'm building. However, I throw away all my mocks and re-TDD once I know what I'm building.

chriswarbo 7y ago
I agree with basically all of this. > some people believe very strongly that unit tests should test things in isolation. Secondly people believe that unit testing should be a black box testing strategy. So you should test through your public interfaces only and any collaborators that adheres to the interface contract should work as expected. I think much of the confusion and talking-past-each-other comes from ambiguous language. I actually agree with all the things in the above quote (isolation, black-box, public-only, relying only on specified interfaces). Where I've differed from co-workers is that I consider the appropriate "unit" to be a feature/piece-of-functionality (e.g. "logging in"), whereas they consider the appropriate "unit" to be a piece of code (e.g. a method or class). I had a bit of a rant about this at http://chriswarbo.net/blog/2017-11-10-unit_testing_terminology.html http://chriswarbo.net/blog/2017-11-10-unit_testing_terminolo...
- blacksmythe 7y ago
  The talk that you link to: https://www.infoq.com/presentations/tdd-original/ https://www.infoq.com/presentations/tdd-original/ Ian Cooper reminds what was Kent's original proposition on TDD, what misunderstandings occurred along the way, and suggests a better approach to TDD, one that supports development rather impeding it.
- mikekchar 7y ago
  I think Michael Feathers explained it the best. He likened unit testing to clamping a piece of woodwork while you are working on it. The bits you are working on need to be in motion because you are working on them. The bits you are not working on them need to be clamped in place -- you don't want those things moving while you are working on some other bit. A "unit" is anything you might want to be clamped in place. It can be a function. It can be an object. It can be a subsystem. You want to unit test at different levels of abstraction so that you can "clamp" those levels of abstraction down. One of the things I've found people get confused with is that they see unit testing and integration testing as orthogonal. They think a unit test should exercise a small piece of code in isolation and an integration test should test examples of real collaborators. Frequently they mock out all their unit tests and write a few integration tests. Then their unit tests become brittle and annoying and so they delete them, leaving only a few integration tests. This leads a lot of people with the impression that only integration tests are useful. If we can back up and redefine "unit test", then the problem disappears. I read your "rant". Don't even get me started on BDD :-) Originally people had problems understanding the purpose of TDD because the word "test" had them confused. They would think, "I need to write tests to ensure that this is working". They didn't think about it in terms of clamping the behaviour so that it doesn't change when you are working on another part of the system. For that reason, a lot of people discussed changing the word "test" to something else that truly embodied what TDD was all about. Many people hit on the word "behavior" -- you want to document the current behaviour of your "units" (at different levels of abstraction). Somehow this got totally confused with automated acceptance testing! Now we have things like cucumber (which I don't actually hate, but it accomplishes a completely different goal than TDD!) What really frustrates me is when I talk to people about this stuff and they think I'm a complete lunatic :-)