4 ms·

Something not mentioned but is a much bigger problem than performance for most is that if you write too many tests, you may be unable to easily upgrade/update t

Something not mentioned but is a much bigger problem than performance for most is that if you write too many tests, you may be unable to easily upgrade/update to newer versions without a significant amount of work to update all of the tests. You’re right to call out performance as a reason for testing. Unfortunately, though, it’s usually not that easy. A problem not found in testing may be the result of inexperience or lack of foresight in design, such as the assumption that another adapter service in front of that enterprise cloud service that only exists in production isn’t a bottleneck, or that your code that had 100% unit test coverage turned out to leave nasty half-updated data due to non-transactional services and was deadlock-prone with memory leaks.

stonemetal12 6y ago
Isn't that a bit of a red herring? Either I have automated tests that will take time to update every once in a while, or I have manual regression tests that I have to run for every code change. The automated test path seems to be the much less time intensive path, depending on how often you make big changes. On the other hand I really believe 100% unit test coverage is a bit of a waste of time. Unit tests should be reserved for algorithmic\calculative code, more mundane code logging\carting data from a to b type code should be covered by automated black box testing.
- klingon78 6y ago
  I meant upgrading application frameworks, OS, etc., so it’s not a red herring. Those can inflict a lot of changes if you have a lot of tests affected.
  - stonemetal12 6y ago
    Thus automated test time + automated test maintenance is usually less than manual test time + manual test maintenance. Mostly because execution time for automated tests is so much shorter than manual execution time that the higher cost of automated test maintenance is recouped and then some. The only way for framework\OS changes to impact that is if they happen often enough to make automated test maintenance "weigh" more than all the time saved by no longer having to manually run your test suite.
    - klingon78 6y ago
      I don’t think we’re speaking the same language. I’m talking about how much time it takes to change the code in every unit test, etc. because you upgraded something that requires changing the test framework, and this library and that one, so you end up not only having to fix all of your production code, but all of the tests with their mocks, stubs, injections, annotations, xml config, etc. If you have thousands of tests, this can keep people from upgrading.
- bluGill 6y ago
  That depends on the tests. I've seen tests that are the code is implemented like X. I've seen tests that are the code uses some API to do X. I've seen tests that are the program does X via some user interface. What you want are tests that the program does X, regardless of what computer it runs on, or what the interfaces are. However this is impossible long term. You MUST assume something that is subject to change in the future. Some changes are more subject to change than others. Automated tests are an assertion that "this will never again change". You can make an educated guess as to what will and will not change, if you guess right then you don't need to rework your tests. When (not if!) you guess wrong you need to rework tests.
shalmanese 6y ago
> you may be unable to easily upgrade/update to newer versions without a significant amount of work to update all of the tests. That's a code smell. Tests should only be updated to reflect a new understanding of the requirements of the system. If you're writing tests that are dependant on the implementation of code and not the behavior of code, then you're too tightly coupling your tests to your code.

stonemetal12 6y ago

Isn't that a bit of a red herring? Either I have automated tests that will take time to update every once in a while, or I have manual regression tests that I have to run for every code change. The automated test path seems to be the much less time intensive path, depending on how often you make big changes. On the other hand I really believe 100% unit test coverage is a bit of a waste of time. Unit tests should be reserved for algorithmic\calculative code, more mundane code logging\carting data from a to b type code should be covered by automated black box testing.

klingon78 6y ago
I meant upgrading application frameworks, OS, etc., so it’s not a red herring. Those can inflict a lot of changes if you have a lot of tests affected.
- stonemetal12 6y ago
  Thus automated test time + automated test maintenance is usually less than manual test time + manual test maintenance. Mostly because execution time for automated tests is so much shorter than manual execution time that the higher cost of automated test maintenance is recouped and then some. The only way for framework\OS changes to impact that is if they happen often enough to make automated test maintenance "weigh" more than all the time saved by no longer having to manually run your test suite.
  - klingon78 6y ago
    I don’t think we’re speaking the same language. I’m talking about how much time it takes to change the code in every unit test, etc. because you upgraded something that requires changing the test framework, and this library and that one, so you end up not only having to fix all of your production code, but all of the tests with their mocks, stubs, injections, annotations, xml config, etc. If you have thousands of tests, this can keep people from upgrading.
bluGill 6y ago
That depends on the tests. I've seen tests that are the code is implemented like X. I've seen tests that are the code uses some API to do X. I've seen tests that are the program does X via some user interface. What you want are tests that the program does X, regardless of what computer it runs on, or what the interfaces are. However this is impossible long term. You MUST assume something that is subject to change in the future. Some changes are more subject to change than others. Automated tests are an assertion that "this will never again change". You can make an educated guess as to what will and will not change, if you guess right then you don't need to rework your tests. When (not if!) you guess wrong you need to rework tests.

shalmanese 6y ago

> you may be unable to easily upgrade/update to newer versions without a significant amount of work to update all of the tests. That's a code smell. Tests should only be updated to reflect a new understanding of the requirements of the system. If you're writing tests that are dependant on the implementation of code and not the behavior of code, then you're too tightly coupling your tests to your code.