5 ms·

I have moved into thinking that mocks don't really have much value. Testing real things is the only way to verify real things. If you can, set up a real DB, ca

I have moved into thinking that mocks don't really have much value. Testing real things is the only way to verify real things. If you can, set up a real DB, cache, whatever, and run your services in tests with real instances of their dependencies. Obviously this falls through for external APIs and things that are very expensive, but your savings in "But the tests passed" confusion time will be huge.

JamesBarney 6y ago
I've come to this conclusion. Organizations will spend many years writing and maintaining dubious unit tests that break with the smallest change, when they'd get an order of magnitude more value spending that time on an automation suite. One of the biggest reasons is testing at the same unit of granularity as the spec means you're tests break when the spec changes, and usually unit tests are far too granular. *I still think unit tests are great for tricky logic, but a lot of coding is just gluing together different systems where unit tests don't have a lot of bang for their buck.
- bird_monster 6y ago
  I might-should've rephrased my original post to suggest that mocks for non-code-you-wrote deps are a waste, but being able to inject different implementations of code you wrote for the sake of testing still has value. If you want to test how your code behaves when getting/setting data from a database, you fundamentally cannot mock the database. If you want to test how your code forms arguments to send to a dependency, you probably can, as long as you do both. If you're just testing that your args are correct without actually using them, you'll never really know. A specific example stands out in my mind when I watched a TDD-minded developer suggest that their code was complete because they injected a mocked version of DynamoDB into a service and then verified their tests ran. They committed and pushed without actually testing against DynamoDB. When tasked with _actually running it_, they found that there was a fundamental flaw in the way that developer perceived Dynamo's behavior, which obviously meant their mock was totally useless. If you want to test that you're pulling the right ID off an object to send to a database you can mock. If you want to test how your code actually behaves with a database, including failure scenarios and invalid arguments, you cannot. As usual, solutions are much more nuanced than I previously stated. I've moved away from mocks, not banished them entirely.
  - chihuahua 6y ago
    Yes, there's a pretty fundamental problem: does your mock match the behavior of the real component? You can never be sure, unless you write extensive tests that compare your mock to the real component. And then those tests need the real component. But mocks could be useful for testing that your code correctly handles errors that are returned by an external component.
  - matthias509 6y ago
    I’ve seen this over and over. The engineer building the mock is the same that wrote the code, so the same incorrect assumptions get baked into both. This is what makes mocks of limited utility. In the end, they don’t prove the code works and they are expensive to create and maintain.
    - bird_monster 6y ago
      Yep. I kinda also think that the dev that wrote the code shouldn't be the dev that writes any of the tests, but that's much more of a philosophically sound but in-practice poor idea that I think about sometimes. I think it has an array of benefits that go totally unrealized if a person writes both. Things like, if your code is too complex for another engineer to write tests for, it's too complex for your team to maintain. If code standards aren't consistent enough in your codebase that other engineers feel the need to refactor/redo blocks of code in the name of style/readability/whatever, your codebase's style isn't automated well enough. Things like that. I just think it's so much of a better strategy, unfortunately up front it seems like a big time sink for (at surface level) little game. I also refuse to measure test coverage in my codebases. "How frequently do bugs show up in production" and "how frequently are bugs fixed without adding tests" are metrics I find valuable but are underrepresented in the testing space. If bugs don't make it to prod very often, and when they do they are fixed, your testing strategy is probably sufficient. There's no reason to write thousands of null checks and formatting validators if you don't need to. Tests require as much or more maintenance as code. There's no reason to write more of them than you need.
      - raducu 6y ago
- cle 6y ago
  One of the big reasons for this that I've seen is that it's easier to measure "coverage" with unit tests. Many managers and engineers like things that are easier to measure, even if they're measuring the wrong thing. It can also be a cover-your-ass strategy--if a major catastrophe occurs, you don't want to be defending your decisions without data. From that perspective, "we feel like we had sufficient integration test coverage" is much worse than "we have 95% unit test coverage". You don't want to make it easy to become a scapegoat, and mountains of data are good protection, whether they're relevant data or not. I too generally only write unit tests for tricky pieces of code, especially when it's dealing with concurrency which tends to be difficult to test deterministically at the API level. There are other good reasons to write unit tests, such as execution speed, complexity of setting up a testing environment, faster development feedback loops, etc.
  - karottenreibe 6y ago
    It's actually not that hard in most non-embedded software testing to get code coverage for any test stage. Even up to manual testing. The tooling exists for all major languages. I recently wrote an article on the subject which links to tutorials for Java and C#: https://docs.teamscale.com/howto/recording-coverage-for-manual-tests/#the-principles-behind-manual-test-coverage https://docs.teamscale.com/howto/recording-coverage-for-manu... Maybe that'll help you convince your managers to invest in the kinds of tests that make sense for your product.
commandlinefan 6y ago
> set up a real DB, cache, whatever You should shoot for both, actually. If your tests require a live DB to run, they can't be automated (or they can be automated, but failures won't tell you much useful). There's a non-unit-test value of having good code coverage, too: being able to run any function independent of the rest of the app, with arbitrary inputs. When you have a problem that's difficult to reproduce in a controlled environment, it's really useful to be able to run a function that's normally ten function calls deep all by itself over and over again while trying to isolate a problem. If you have to run the entire app to run any of it, that becomes impossible.
- bird_monster 6y ago
  > You should shoot for both, actually. If your tests require a live DB to run, they can't be automated (or they can be automated, but failures won't tell you much useful). Can you define a scenario in which hitting a real DB wouldn't present useful information while a mocked DB would?
  - commandlinefan 6y ago
    Sure, if the DB is down. Or if somebody else deleted the data that your test relied on. It'll happen often enough that you'll stop paying attention to failing "unit" tests.
    - bird_monster 6y ago
      > Or if somebody else deleted the data that your test relied on If you're relying on a shared database you're already not really testing effectively. I don't really think this is a scenario to worry about (meaning, if you're in this scenario, you have bigger things to worry about). > if the DB is down. An interesting suggestion.
  - raducu 6y ago
    When you have multiple teams pushing changes and a limited mumber or DB licenses and you want your CI pipeline to run faster, mocks are a very good compromise. DB bugs rarely happen if you don't change any DB stuff after a couple of years in a project, at that point you are really confident in your DAOs and ORM services and you can mock them. You write integration tests for the DAOs and ORM, but you rely on the mocked version in 95% of the test cases.

JamesBarney 6y ago

I've come to this conclusion. Organizations will spend many years writing and maintaining dubious unit tests that break with the smallest change, when they'd get an order of magnitude more value spending that time on an automation suite. One of the biggest reasons is testing at the same unit of granularity as the spec means you're tests break when the spec changes, and usually unit tests are far too granular. *I still think unit tests are great for tricky logic, but a lot of coding is just gluing together different systems where unit tests don't have a lot of bang for their buck.

bird_monster 6y ago
I might-should've rephrased my original post to suggest that mocks for non-code-you-wrote deps are a waste, but being able to inject different implementations of code you wrote for the sake of testing still has value. If you want to test how your code behaves when getting/setting data from a database, you fundamentally cannot mock the database. If you want to test how your code forms arguments to send to a dependency, you probably can, as long as you do both. If you're just testing that your args are correct without actually using them, you'll never really know. A specific example stands out in my mind when I watched a TDD-minded developer suggest that their code was complete because they injected a mocked version of DynamoDB into a service and then verified their tests ran. They committed and pushed without actually testing against DynamoDB. When tasked with _actually running it_, they found that there was a fundamental flaw in the way that developer perceived Dynamo's behavior, which obviously meant their mock was totally useless. If you want to test that you're pulling the right ID off an object to send to a database you can mock. If you want to test how your code actually behaves with a database, including failure scenarios and invalid arguments, you cannot. As usual, solutions are much more nuanced than I previously stated. I've moved away from mocks, not banished them entirely.
- chihuahua 6y ago
  Yes, there's a pretty fundamental problem: does your mock match the behavior of the real component? You can never be sure, unless you write extensive tests that compare your mock to the real component. And then those tests need the real component. But mocks could be useful for testing that your code correctly handles errors that are returned by an external component.
- matthias509 6y ago
  I’ve seen this over and over. The engineer building the mock is the same that wrote the code, so the same incorrect assumptions get baked into both. This is what makes mocks of limited utility. In the end, they don’t prove the code works and they are expensive to create and maintain.
  - bird_monster 6y ago
    Yep. I kinda also think that the dev that wrote the code shouldn't be the dev that writes any of the tests, but that's much more of a philosophically sound but in-practice poor idea that I think about sometimes. I think it has an array of benefits that go totally unrealized if a person writes both. Things like, if your code is too complex for another engineer to write tests for, it's too complex for your team to maintain. If code standards aren't consistent enough in your codebase that other engineers feel the need to refactor/redo blocks of code in the name of style/readability/whatever, your codebase's style isn't automated well enough. Things like that. I just think it's so much of a better strategy, unfortunately up front it seems like a big time sink for (at surface level) little game. I also refuse to measure test coverage in my codebases. "How frequently do bugs show up in production" and "how frequently are bugs fixed without adding tests" are metrics I find valuable but are underrepresented in the testing space. If bugs don't make it to prod very often, and when they do they are fixed, your testing strategy is probably sufficient. There's no reason to write thousands of null checks and formatting validators if you don't need to. Tests require as much or more maintenance as code. There's no reason to write more of them than you need.
    - raducu 6y ago
cle 6y ago
One of the big reasons for this that I've seen is that it's easier to measure "coverage" with unit tests. Many managers and engineers like things that are easier to measure, even if they're measuring the wrong thing. It can also be a cover-your-ass strategy--if a major catastrophe occurs, you don't want to be defending your decisions without data. From that perspective, "we feel like we had sufficient integration test coverage" is much worse than "we have 95% unit test coverage". You don't want to make it easy to become a scapegoat, and mountains of data are good protection, whether they're relevant data or not. I too generally only write unit tests for tricky pieces of code, especially when it's dealing with concurrency which tends to be difficult to test deterministically at the API level. There are other good reasons to write unit tests, such as execution speed, complexity of setting up a testing environment, faster development feedback loops, etc.
- karottenreibe 6y ago
  It's actually not that hard in most non-embedded software testing to get code coverage for any test stage. Even up to manual testing. The tooling exists for all major languages. I recently wrote an article on the subject which links to tutorials for Java and C#: https://docs.teamscale.com/howto/recording-coverage-for-manual-tests/#the-principles-behind-manual-test-coverage https://docs.teamscale.com/howto/recording-coverage-for-manu... Maybe that'll help you convince your managers to invest in the kinds of tests that make sense for your product.

commandlinefan 6y ago

> set up a real DB, cache, whatever You should shoot for both, actually. If your tests require a live DB to run, they can't be automated (or they can be automated, but failures won't tell you much useful). There's a non-unit-test value of having good code coverage, too: being able to run any function independent of the rest of the app, with arbitrary inputs. When you have a problem that's difficult to reproduce in a controlled environment, it's really useful to be able to run a function that's normally ten function calls deep all by itself over and over again while trying to isolate a problem. If you have to run the entire app to run any of it, that becomes impossible.

bird_monster 6y ago
> You should shoot for both, actually. If your tests require a live DB to run, they can't be automated (or they can be automated, but failures won't tell you much useful). Can you define a scenario in which hitting a real DB wouldn't present useful information while a mocked DB would?
- commandlinefan 6y ago
  Sure, if the DB is down. Or if somebody else deleted the data that your test relied on. It'll happen often enough that you'll stop paying attention to failing "unit" tests.
  - bird_monster 6y ago
    > Or if somebody else deleted the data that your test relied on If you're relying on a shared database you're already not really testing effectively. I don't really think this is a scenario to worry about (meaning, if you're in this scenario, you have bigger things to worry about). > if the DB is down. An interesting suggestion.
- raducu 6y ago
  When you have multiple teams pushing changes and a limited mumber or DB licenses and you want your CI pipeline to run faster, mocks are a very good compromise. DB bugs rarely happen if you don't change any DB stuff after a couple of years in a project, at that point you are really confident in your DAOs and ORM services and you can mock them. You write integration tests for the DAOs and ORM, but you rely on the mocked version in 95% of the test cases.