4 ms·

"... stubs and mocks are bad. Your tests are only as good as your mocks. They hide the true behavior of your system, drift away from reality, and take a lot of

"... stubs and mocks are bad. Your tests are only as good as your mocks. They hide the true behavior of your system, drift away from reality, and take a lot of effort to maintain." I've often felt guilty for leaning toward broader-scope integration tests, but I've long felt this too— that they're less effort to write and maintain and are much better value in terms of actually catching issues. Not that unit tests don't have their place, but they're best for mostly-pure functions or classes where you're intentionally trying to cover a bunch of branches or corner cases.

jamesgreenleaf 5y ago
I've felt the same way - as though not being able to shoehorn everything into pure unit tests somehow means that I'm not doing testing right. In most cases though, what we really care about is verifying the full functionality of an app. Maybe integration & e2e tests should be considered as the primary form of testing, with unit tests applying only in the specific cases where they make sense, like you mentioned. Maybe the ulimate form of testing is something along the lines of Netflix's chaos monkey (now apparently an entire "simian army" suite of tools). Continuously test the real application, live in production, where it's not allowed to fail.
- mekane8 5y ago
  I believe pretty strongly in the "testing pyramid" - that you should have a broad base of unit tests, with fewer integration tests and even fewer e2e tests. Source (also from Google): https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html https://testing.googleblog.com/2015/04/just-say-no-to-more-e... I think good unit test coverage is essential for good maintenance, but you can't cover _everything_ with unit tests. You definitely need integration and e2e tests to ensure bigger picture correctness. I have found that value one gets out of the different kinds of tests as you proceed up the pyramid get higher the closer to release you are, and are most valuable post release. During development they become obsolete too quickly due to changes in how things work. But locking in an already-released version to avoid regressions is great.
  - mikepurvis 5y ago
    I think the real question for me is, suppose you come into maintainership of a legacy application with 100k lines of untested code, where do you start? I think the classical TDD view is "every new change, you add a test for, and so test coverage grows over time." And that's fine, but I think it's much more valuable before making any functional changes at all, to impose some high level smoke tests that run the entire thing, and do a handful of common, end to end user flows.
    - disgruntledphd2 5y ago
      Yeah, I'm a big fan of the Outside-In approach (I think it's from Michael Feathers's work). The idea is that you test something at the very end of whatever you're working on, and ensure that it's identical to the original. You can then make larger changes to actually test the code properly while confirming that the overall system still works. Super, super useful with ML things where running stuff can take a very looooonnnnnngggg time.
    - mekane8 5y ago
      Yes, that gels with what I was saying - use e2e tests for "big picture" assurance that the system is correct. Use unit tests for super fain grained assurance of individual business rules, etc. All of the legacy projects I've ever inherited have lacked unit tests, and are theoretically functioning correctly, so the e2e tests make sense there.
mekane8 5y ago
I personally like and get value out of stubs and mocks in my unit tests, but I think they suffer some of the same issues as code without tests - it's hard to know the original intent and it's hard to constrain later contributors into doing the correct thing. I can mock up an external part of the system for my unit tests and be confident I'm only testing the behavior in my module, but someone who comes along later won't have the same picture in their head and will come to different assumptions. Integration tests are definitely the way to go here, when you are wanting to ensure your unit works correctly in conjunction with external systems. But that doesn't make the unit tests bad, just insufficient. I'd probably phrase it as "Stubs and Mocks do not make sufficient tests", not that they necessarily make bad tests.
benlivengood 5y ago
Mocks (and fakes) have their place; verifying the behavior in exceptional circumstances in a deterministic way (avoiding flaky tests) and unit-testing code exhaustively that would otherwise be too slow if written as full integration tests. Filesystem, network, and database dependencies are good places to mock/fake because errors are rare (and would be flaky) in integration tests and maintaining backend state for a lot of unit tests can slow the development cycle. Yes, keeping mocks in sync with reality is hard but it should be up to the library owner to maintain either a good set of mocks or provide a fast enough library that integration tests are the better choice. This works in a company with uniform testing goals as a core value, not so we'll when dependencies and users have different testing philosophies.
- mikepurvis 5y ago
  I think the distinction I'm seeing here is that a mock is a pretend implementation of whatever the interface is, either at a code or service level, whereas as a fake is that actual service in whatever its minimally-viable state is. So like— not mocking out the database interface with cooked responses, but instead using an actual instance of the database.
kyrra 5y ago
Googler, opinions are my own. My corner of the world has ~100 people working on it and we do a mix of unit tests and integration tests. We definitely use mocks, but prefer fakes where possible. I really like our unit tests to poke and prod the corners of a given small section of code. Our integration tests are just that, and they serve their purpose as well. But being able to stress all underlying parts of the system with all possible inputs at the integration test level is really really hard (if not impossible). Many times, the way a system currently is can't push the corners of some helper class deep down, and you won't find those until someone makes some change in an intermediary layer. Making sure things are working at both the micro and macro level seems like the best way to go. But there is the obvious tradeoff of time it takes to do this. Other teams we work with are write only integration tests. While it mostly works, I would say that it's harder for someone else to come into the code base and make changes. Unit tests help people understand how the code is expected to work today and how their changes may impact that. Integration tests won't always catch those kinds of things (plus tend to be way longer to run, which makes iteration time longer).
- mikepurvis 5y ago
  Yeah, I hear that. I wonder if there's a cart/horse issue here where the greatest value for low level unit tests is when you have a well-documented, thoughtfully architected system. Then the tests are verifying intentional behaviours and helping ensure the longevity of the system as designed. I don't think I have this experience first-hand, but I could certainly imagine a scenario where a well-intentioned person added a bunch of unit tests to a badly designed system, and they ultimately ended up just getting in the way and causing confusion when it was time to rethink it.

jamesgreenleaf 5y ago

I've felt the same way - as though not being able to shoehorn everything into pure unit tests somehow means that I'm not doing testing right. In most cases though, what we really care about is verifying the full functionality of an app. Maybe integration & e2e tests should be considered as the primary form of testing, with unit tests applying only in the specific cases where they make sense, like you mentioned. Maybe the ulimate form of testing is something along the lines of Netflix's chaos monkey (now apparently an entire "simian army" suite of tools). Continuously test the real application, live in production, where it's not allowed to fail.

mekane8 5y ago
I believe pretty strongly in the "testing pyramid" - that you should have a broad base of unit tests, with fewer integration tests and even fewer e2e tests. Source (also from Google): https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html https://testing.googleblog.com/2015/04/just-say-no-to-more-e... I think good unit test coverage is essential for good maintenance, but you can't cover _everything_ with unit tests. You definitely need integration and e2e tests to ensure bigger picture correctness. I have found that value one gets out of the different kinds of tests as you proceed up the pyramid get higher the closer to release you are, and are most valuable post release. During development they become obsolete too quickly due to changes in how things work. But locking in an already-released version to avoid regressions is great.
- mikepurvis 5y ago
  I think the real question for me is, suppose you come into maintainership of a legacy application with 100k lines of untested code, where do you start? I think the classical TDD view is "every new change, you add a test for, and so test coverage grows over time." And that's fine, but I think it's much more valuable before making any functional changes at all, to impose some high level smoke tests that run the entire thing, and do a handful of common, end to end user flows.
  - disgruntledphd2 5y ago
    Yeah, I'm a big fan of the Outside-In approach (I think it's from Michael Feathers's work). The idea is that you test something at the very end of whatever you're working on, and ensure that it's identical to the original. You can then make larger changes to actually test the code properly while confirming that the overall system still works. Super, super useful with ML things where running stuff can take a very looooonnnnnngggg time.
  - mekane8 5y ago
    Yes, that gels with what I was saying - use e2e tests for "big picture" assurance that the system is correct. Use unit tests for super fain grained assurance of individual business rules, etc. All of the legacy projects I've ever inherited have lacked unit tests, and are theoretically functioning correctly, so the e2e tests make sense there.

mekane8 5y ago

I personally like and get value out of stubs and mocks in my unit tests, but I think they suffer some of the same issues as code without tests - it's hard to know the original intent and it's hard to constrain later contributors into doing the correct thing. I can mock up an external part of the system for my unit tests and be confident I'm only testing the behavior in my module, but someone who comes along later won't have the same picture in their head and will come to different assumptions. Integration tests are definitely the way to go here, when you are wanting to ensure your unit works correctly in conjunction with external systems. But that doesn't make the unit tests bad, just insufficient. I'd probably phrase it as "Stubs and Mocks do not make sufficient tests", not that they necessarily make bad tests.

benlivengood 5y ago

Mocks (and fakes) have their place; verifying the behavior in exceptional circumstances in a deterministic way (avoiding flaky tests) and unit-testing code exhaustively that would otherwise be too slow if written as full integration tests. Filesystem, network, and database dependencies are good places to mock/fake because errors are rare (and would be flaky) in integration tests and maintaining backend state for a lot of unit tests can slow the development cycle. Yes, keeping mocks in sync with reality is hard but it should be up to the library owner to maintain either a good set of mocks or provide a fast enough library that integration tests are the better choice. This works in a company with uniform testing goals as a core value, not so we'll when dependencies and users have different testing philosophies.

mikepurvis 5y ago
I think the distinction I'm seeing here is that a mock is a pretend implementation of whatever the interface is, either at a code or service level, whereas as a fake is that actual service in whatever its minimally-viable state is. So like— not mocking out the database interface with cooked responses, but instead using an actual instance of the database.

kyrra 5y ago

Googler, opinions are my own. My corner of the world has ~100 people working on it and we do a mix of unit tests and integration tests. We definitely use mocks, but prefer fakes where possible. I really like our unit tests to poke and prod the corners of a given small section of code. Our integration tests are just that, and they serve their purpose as well. But being able to stress all underlying parts of the system with all possible inputs at the integration test level is really really hard (if not impossible). Many times, the way a system currently is can't push the corners of some helper class deep down, and you won't find those until someone makes some change in an intermediary layer. Making sure things are working at both the micro and macro level seems like the best way to go. But there is the obvious tradeoff of time it takes to do this. Other teams we work with are write only integration tests. While it mostly works, I would say that it's harder for someone else to come into the code base and make changes. Unit tests help people understand how the code is expected to work today and how their changes may impact that. Integration tests won't always catch those kinds of things (plus tend to be way longer to run, which makes iteration time longer).

mikepurvis 5y ago
Yeah, I hear that. I wonder if there's a cart/horse issue here where the greatest value for low level unit tests is when you have a well-documented, thoughtfully architected system. Then the tests are verifying intentional behaviours and helping ensure the longevity of the system as designed. I don't think I have this experience first-hand, but I could certainly imagine a scenario where a well-intentioned person added a bunch of unit tests to a badly designed system, and they ultimately ended up just getting in the way and causing confusion when it was time to rethink it.