4 ms·

I've come to this conclusion. Organizations will spend many years writing and maintaining dubious unit tests that break with the smallest change, when they'd g

I've come to this conclusion. Organizations will spend many years writing and maintaining dubious unit tests that break with the smallest change, when they'd get an order of magnitude more value spending that time on an automation suite. One of the biggest reasons is testing at the same unit of granularity as the spec means you're tests break when the spec changes, and usually unit tests are far too granular. *I still think unit tests are great for tricky logic, but a lot of coding is just gluing together different systems where unit tests don't have a lot of bang for their buck.

bird_monster 6y ago
I might-should've rephrased my original post to suggest that mocks for non-code-you-wrote deps are a waste, but being able to inject different implementations of code you wrote for the sake of testing still has value. If you want to test how your code behaves when getting/setting data from a database, you fundamentally cannot mock the database. If you want to test how your code forms arguments to send to a dependency, you probably can, as long as you do both. If you're just testing that your args are correct without actually using them, you'll never really know. A specific example stands out in my mind when I watched a TDD-minded developer suggest that their code was complete because they injected a mocked version of DynamoDB into a service and then verified their tests ran. They committed and pushed without actually testing against DynamoDB. When tasked with _actually running it_, they found that there was a fundamental flaw in the way that developer perceived Dynamo's behavior, which obviously meant their mock was totally useless. If you want to test that you're pulling the right ID off an object to send to a database you can mock. If you want to test how your code actually behaves with a database, including failure scenarios and invalid arguments, you cannot. As usual, solutions are much more nuanced than I previously stated. I've moved away from mocks, not banished them entirely.
- chihuahua 6y ago
  Yes, there's a pretty fundamental problem: does your mock match the behavior of the real component? You can never be sure, unless you write extensive tests that compare your mock to the real component. And then those tests need the real component. But mocks could be useful for testing that your code correctly handles errors that are returned by an external component.
- matthias509 6y ago
  I’ve seen this over and over. The engineer building the mock is the same that wrote the code, so the same incorrect assumptions get baked into both. This is what makes mocks of limited utility. In the end, they don’t prove the code works and they are expensive to create and maintain.
  - bird_monster 6y ago
    Yep. I kinda also think that the dev that wrote the code shouldn't be the dev that writes any of the tests, but that's much more of a philosophically sound but in-practice poor idea that I think about sometimes. I think it has an array of benefits that go totally unrealized if a person writes both. Things like, if your code is too complex for another engineer to write tests for, it's too complex for your team to maintain. If code standards aren't consistent enough in your codebase that other engineers feel the need to refactor/redo blocks of code in the name of style/readability/whatever, your codebase's style isn't automated well enough. Things like that. I just think it's so much of a better strategy, unfortunately up front it seems like a big time sink for (at surface level) little game. I also refuse to measure test coverage in my codebases. "How frequently do bugs show up in production" and "how frequently are bugs fixed without adding tests" are metrics I find valuable but are underrepresented in the testing space. If bugs don't make it to prod very often, and when they do they are fixed, your testing strategy is probably sufficient. There's no reason to write thousands of null checks and formatting validators if you don't need to. Tests require as much or more maintenance as code. There's no reason to write more of them than you need.
    - raducu 6y ago
      Some things can be mocked quite successfuly -- like ORM stuff. You can also write an integration test for the mocked component, let the rest of the code blaze away with the mocked version, it is usually a very good compromise.
cle 6y ago
One of the big reasons for this that I've seen is that it's easier to measure "coverage" with unit tests. Many managers and engineers like things that are easier to measure, even if they're measuring the wrong thing. It can also be a cover-your-ass strategy--if a major catastrophe occurs, you don't want to be defending your decisions without data. From that perspective, "we feel like we had sufficient integration test coverage" is much worse than "we have 95% unit test coverage". You don't want to make it easy to become a scapegoat, and mountains of data are good protection, whether they're relevant data or not. I too generally only write unit tests for tricky pieces of code, especially when it's dealing with concurrency which tends to be difficult to test deterministically at the API level. There are other good reasons to write unit tests, such as execution speed, complexity of setting up a testing environment, faster development feedback loops, etc.
- karottenreibe 6y ago
  It's actually not that hard in most non-embedded software testing to get code coverage for any test stage. Even up to manual testing. The tooling exists for all major languages. I recently wrote an article on the subject which links to tutorials for Java and C#: https://docs.teamscale.com/howto/recording-coverage-for-manual-tests/#the-principles-behind-manual-test-coverage https://docs.teamscale.com/howto/recording-coverage-for-manu... Maybe that'll help you convince your managers to invest in the kinds of tests that make sense for your product.

bird_monster 6y ago

I might-should've rephrased my original post to suggest that mocks for non-code-you-wrote deps are a waste, but being able to inject different implementations of code you wrote for the sake of testing still has value. If you want to test how your code behaves when getting/setting data from a database, you fundamentally cannot mock the database. If you want to test how your code forms arguments to send to a dependency, you probably can, as long as you do both. If you're just testing that your args are correct without actually using them, you'll never really know. A specific example stands out in my mind when I watched a TDD-minded developer suggest that their code was complete because they injected a mocked version of DynamoDB into a service and then verified their tests ran. They committed and pushed without actually testing against DynamoDB. When tasked with _actually running it_, they found that there was a fundamental flaw in the way that developer perceived Dynamo's behavior, which obviously meant their mock was totally useless. If you want to test that you're pulling the right ID off an object to send to a database you can mock. If you want to test how your code actually behaves with a database, including failure scenarios and invalid arguments, you cannot. As usual, solutions are much more nuanced than I previously stated. I've moved away from mocks, not banished them entirely.

chihuahua 6y ago
Yes, there's a pretty fundamental problem: does your mock match the behavior of the real component? You can never be sure, unless you write extensive tests that compare your mock to the real component. And then those tests need the real component. But mocks could be useful for testing that your code correctly handles errors that are returned by an external component.
matthias509 6y ago
I’ve seen this over and over. The engineer building the mock is the same that wrote the code, so the same incorrect assumptions get baked into both. This is what makes mocks of limited utility. In the end, they don’t prove the code works and they are expensive to create and maintain.
- bird_monster 6y ago
  Yep. I kinda also think that the dev that wrote the code shouldn't be the dev that writes any of the tests, but that's much more of a philosophically sound but in-practice poor idea that I think about sometimes. I think it has an array of benefits that go totally unrealized if a person writes both. Things like, if your code is too complex for another engineer to write tests for, it's too complex for your team to maintain. If code standards aren't consistent enough in your codebase that other engineers feel the need to refactor/redo blocks of code in the name of style/readability/whatever, your codebase's style isn't automated well enough. Things like that. I just think it's so much of a better strategy, unfortunately up front it seems like a big time sink for (at surface level) little game. I also refuse to measure test coverage in my codebases. "How frequently do bugs show up in production" and "how frequently are bugs fixed without adding tests" are metrics I find valuable but are underrepresented in the testing space. If bugs don't make it to prod very often, and when they do they are fixed, your testing strategy is probably sufficient. There's no reason to write thousands of null checks and formatting validators if you don't need to. Tests require as much or more maintenance as code. There's no reason to write more of them than you need.
  - raducu 6y ago
    Some things can be mocked quite successfuly -- like ORM stuff. You can also write an integration test for the mocked component, let the rest of the code blaze away with the mocked version, it is usually a very good compromise.

cle 6y ago

One of the big reasons for this that I've seen is that it's easier to measure "coverage" with unit tests. Many managers and engineers like things that are easier to measure, even if they're measuring the wrong thing. It can also be a cover-your-ass strategy--if a major catastrophe occurs, you don't want to be defending your decisions without data. From that perspective, "we feel like we had sufficient integration test coverage" is much worse than "we have 95% unit test coverage". You don't want to make it easy to become a scapegoat, and mountains of data are good protection, whether they're relevant data or not. I too generally only write unit tests for tricky pieces of code, especially when it's dealing with concurrency which tends to be difficult to test deterministically at the API level. There are other good reasons to write unit tests, such as execution speed, complexity of setting up a testing environment, faster development feedback loops, etc.

karottenreibe 6y ago
It's actually not that hard in most non-embedded software testing to get code coverage for any test stage. Even up to manual testing. The tooling exists for all major languages. I recently wrote an article on the subject which links to tutorials for Java and C#: https://docs.teamscale.com/howto/recording-coverage-for-manual-tests/#the-principles-behind-manual-test-coverage https://docs.teamscale.com/howto/recording-coverage-for-manu... Maybe that'll help you convince your managers to invest in the kinds of tests that make sense for your product.