5 ms·

> If you want to reduce your test mass, the number one thing you should do is look at the tests that have never failed in a year and consider throwing them away

> If you want to reduce your test mass, the number one thing you should do is look at the tests that have never failed in a year and consider throwing them away. They are producing no information for you — or at least very little information. The value of the information they produce may not be worth the expense of maintaining and running the tests. This seems.. entirely unreasonable to me. Just because some area of code is not touched frequently doesn't mean that we should throw out tests pertaining to that section of code. Can someone justify this to me? This seems like a _terrible_ mistake.

inimino 10y ago
The context is trivial tests which only failed before the developer wrote the code and have never failed since. An example given is a method setting x to 5 and then testing that method. Because maintaining code has a cost, tests such as these should be removed. Regression tests are mentioned later and are not what the author is addressing here.
- cle 10y ago
  Depending on what language you're using, x=5 may not be as trivial as you think.
  - stcredzero 10y ago
    The most dangerous tools in the shop are often the deceptively simple and familiar ones.
cle 10y ago
I would argue that "producing information" is only one benefit of tests. Having worked with decent-sized legacy code bases, another big benefit of tests is preventing future loss of information. Deleting tests removes those guardrails. If what you write is successful, then it will become a legacy system, and your predecessors who inherit it (and ultimately your customers) will be glad you didn't delete tests.
- indolering 10y ago
  I worked at a company that didn't employ any unit testing and they were terrified of updating their aging software stack because it was constantly breaking as is. I was doing greenfield work and was paranoid about losing data, so I had extensive test-coverage. I undertook multiple complex refactors of the codebase and never had a problem thanks to my unit test coverage. My code was never the cause of any major data loss and I found plenty of very serious bugs in the legacy codebase.
ryanmarsh 10y ago
If you want to reduce your code mass, the number one thing you should do is look at the code paths that have not been reached in a year and consider throwing them away. Still make sense?
- spinningarrow 10y ago
  Code paths that have not been _reached_ are very different from code paths that have not been _changed_, though.
- plandis 10y ago
  So remove all code for handling unlikely failure scenarios? ;)
  - zachrose 10y ago
    This would depend on the kind of software you're making and the potential for an unhandled error to cause injury, expense or damage, but yeah. I would really like some system that can tell me how many times a line of production code has executed in the last year. For an ecommerce web app, I can't imagine how a line of code that hasn't run in the last two years would still need to be there.
  - jshevek 10y ago
    My immediate thoughts exactly, so I reread to confirm the parent said "consider" removing the code. I think this is reasonable. While considering the removal of unused code, we should should keep the code that handles unlikely failure scenarios.
- deleted 10y ago
  [deleted]
humanrebar 10y ago
Passing tests tell you a lot if you pay attention: """ Gregory (Scotland Yard detective): Is there any other point to which you would wish to draw my attention? Sherlock Holmes: To the curious incident of the dog in the night-time. Gregory: The dog did nothing in the night-time. Holmes: That was the curious incident. """
weberc2 10y ago
I don't get it either. In particular, if a test isn't failing, there is no reason to change it--i.e., no maintenance cost. Sometimes tests are written such that they test the implementation rather than the behavior (as is the case with egregious mocking), so changes to the implementation cause the test to report false negatives--but this is a matter of faulty testing practice adding unnecessary maintenance costs. Provided you're following good unit testing practice, I can't imagine an organization whose best investment is removing excess unit tests.
- erikpukinskis 10y ago
  Tests do have a maintenance cost: Sometimes they rely on interfaces other than the one being tested, and that code needs to be updated when those interfaces updated. Generally speaking, an interface will have 2x the consumers in a well tested codebase, which means 2x the refactoring. Tests take time to run, which slows the development loop. Test can be used as documentation. The more tests in a specification, the less useful that spec is as documentation. When refactoring, the number of tests increases the amount of behavior that needs to be pulled along. If the test is no longer important to the business, that slows down refactoring unnecessarily. "Has been passing for a long time" isn't necessarily a good heuristic for identifying these situations. I'm just saying there are lots of costs to having a passing test in the suite.
  - MaulingMonkey 10y ago
    > Tests do have a maintenance cost Agreed, but... > Sometimes they rely on interfaces other than the one being tested, and that code needs to be updated when those interfaces updated. Generally speaking, an interface will have 2x the consumers in a well tested codebase, which means 2x the refactoring. Only when refactoring the public interface, and only when tooling isn't capable of automating the refactoring. In other words, only when you're doing something complex enough that there's a high chance you'll break things and cause tests to fail! Hopefully one will catch said test failures before checking in - note that this will skew any CI metrics. Just because they didn't fail on your CI infrastructure within the past year, doesn't mean they didn't fail within the past year. > Tests take time to run, which slows the development loop. This is, perhaps, a good reason to make those tests run e.g. only for your overnight integration tests. But removal? Nervously auditing and reviewing changes (both your own, and other's in code reviews) for possible breaks because of reduced confidence that your unit testing will catch things also slows down the development loop. As does the debugging sessions when that isn't enough. > Test can be used as documentation. The more tests in a specification, the less useful that spec is as documentation. This is, perhaps, a good reason to relegate less informative tests to another directory. Perhaps the same directory that houses regression tests that your fuzzer found. > When refactoring, the number of tests increases the amount of behavior that needs to be pulled along. If the test is no longer important to the business, that slows down refactoring unnecessarily. The purist might cry that refactoring doesn't change (external) behavior. But I'll note that it's just as easy to delete unit tests when you're changing behavior, as it is to delete them ahead of time. Even easier! Bonus: Deferring their deletion until then makes the fact that you're actually changing behavior a lot more obvious and explicit. Ahh, a fibbed a little: It might be harder to get those deletions approved by your code reviewer if it's not obvious that changing behavior is OK. In my work environments, this is a feature.
ajmurmann 10y ago
I also am not sure how you would identify this with confidence. Sure, I can see the tests haven't failed on CI. However, maybe the tests fail all the time while the tests are being run during development.
MaulingMonkey 10y ago
> This seems like a _terrible_ mistake. I can at least confirm with certainty that it sometimes is a terrible mistake. I port things. One approach I sometimes use is to stub in noop implementations of platform specific things. I generally use #pragmas or the like to generate a warning to come back to (which I can always later crank up to an error to ensure I've gone back and implemented everything.) It's common that the original authors were also "kind enough" to leave some noop implementations of their own - although usually not with the helpful #pragmas. End result: Thousands of failing unit tests, most of which were passing for more than a year. Some are innocuous as minor string formatting changes caused by minor libc differences - but even that can be concerning if you want consistent, reliable results that don't vary by platform and threaten to cause platform specific edge cases. I've never seen a codebase where even 10% of the failures are that innocuous.

inimino 10y ago

The context is trivial tests which only failed before the developer wrote the code and have never failed since. An example given is a method setting x to 5 and then testing that method. Because maintaining code has a cost, tests such as these should be removed. Regression tests are mentioned later and are not what the author is addressing here.

cle 10y ago
Depending on what language you're using, x=5 may not be as trivial as you think.
- stcredzero 10y ago
  The most dangerous tools in the shop are often the deceptively simple and familiar ones.

cle 10y ago

I would argue that "producing information" is only one benefit of tests. Having worked with decent-sized legacy code bases, another big benefit of tests is preventing future loss of information. Deleting tests removes those guardrails. If what you write is successful, then it will become a legacy system, and your predecessors who inherit it (and ultimately your customers) will be glad you didn't delete tests.

indolering 10y ago
I worked at a company that didn't employ any unit testing and they were terrified of updating their aging software stack because it was constantly breaking as is. I was doing greenfield work and was paranoid about losing data, so I had extensive test-coverage. I undertook multiple complex refactors of the codebase and never had a problem thanks to my unit test coverage. My code was never the cause of any major data loss and I found plenty of very serious bugs in the legacy codebase.

ryanmarsh 10y ago

If you want to reduce your code mass, the number one thing you should do is look at the code paths that have not been reached in a year and consider throwing them away. Still make sense?

spinningarrow 10y ago
Code paths that have not been _reached_ are very different from code paths that have not been _changed_, though.
plandis 10y ago
So remove all code for handling unlikely failure scenarios? ;)
- zachrose 10y ago
  This would depend on the kind of software you're making and the potential for an unhandled error to cause injury, expense or damage, but yeah. I would really like some system that can tell me how many times a line of production code has executed in the last year. For an ecommerce web app, I can't imagine how a line of code that hasn't run in the last two years would still need to be there.
- jshevek 10y ago
  My immediate thoughts exactly, so I reread to confirm the parent said "consider" removing the code. I think this is reasonable. While considering the removal of unused code, we should should keep the code that handles unlikely failure scenarios.
deleted 10y ago
[deleted]

humanrebar 10y ago

Passing tests tell you a lot if you pay attention: """ Gregory (Scotland Yard detective): Is there any other point to which you would wish to draw my attention? Sherlock Holmes: To the curious incident of the dog in the night-time. Gregory: The dog did nothing in the night-time. Holmes: That was the curious incident. """

weberc2 10y ago

I don't get it either. In particular, if a test isn't failing, there is no reason to change it--i.e., no maintenance cost. Sometimes tests are written such that they test the implementation rather than the behavior (as is the case with egregious mocking), so changes to the implementation cause the test to report false negatives--but this is a matter of faulty testing practice adding unnecessary maintenance costs. Provided you're following good unit testing practice, I can't imagine an organization whose best investment is removing excess unit tests.

erikpukinskis 10y ago
Tests do have a maintenance cost: Sometimes they rely on interfaces other than the one being tested, and that code needs to be updated when those interfaces updated. Generally speaking, an interface will have 2x the consumers in a well tested codebase, which means 2x the refactoring. Tests take time to run, which slows the development loop. Test can be used as documentation. The more tests in a specification, the less useful that spec is as documentation. When refactoring, the number of tests increases the amount of behavior that needs to be pulled along. If the test is no longer important to the business, that slows down refactoring unnecessarily. "Has been passing for a long time" isn't necessarily a good heuristic for identifying these situations. I'm just saying there are lots of costs to having a passing test in the suite.
- MaulingMonkey 10y ago
  > Tests do have a maintenance cost Agreed, but... > Sometimes they rely on interfaces other than the one being tested, and that code needs to be updated when those interfaces updated. Generally speaking, an interface will have 2x the consumers in a well tested codebase, which means 2x the refactoring. Only when refactoring the public interface, and only when tooling isn't capable of automating the refactoring. In other words, only when you're doing something complex enough that there's a high chance you'll break things and cause tests to fail! Hopefully one will catch said test failures before checking in - note that this will skew any CI metrics. Just because they didn't fail on your CI infrastructure within the past year, doesn't mean they didn't fail within the past year. > Tests take time to run, which slows the development loop. This is, perhaps, a good reason to make those tests run e.g. only for your overnight integration tests. But removal? Nervously auditing and reviewing changes (both your own, and other's in code reviews) for possible breaks because of reduced confidence that your unit testing will catch things also slows down the development loop. As does the debugging sessions when that isn't enough. > Test can be used as documentation. The more tests in a specification, the less useful that spec is as documentation. This is, perhaps, a good reason to relegate less informative tests to another directory. Perhaps the same directory that houses regression tests that your fuzzer found. > When refactoring, the number of tests increases the amount of behavior that needs to be pulled along. If the test is no longer important to the business, that slows down refactoring unnecessarily. The purist might cry that refactoring doesn't change (external) behavior. But I'll note that it's just as easy to delete unit tests when you're changing behavior, as it is to delete them ahead of time. Even easier! Bonus: Deferring their deletion until then makes the fact that you're actually changing behavior a lot more obvious and explicit. Ahh, a fibbed a little: It might be harder to get those deletions approved by your code reviewer if it's not obvious that changing behavior is OK. In my work environments, this is a feature.

ajmurmann 10y ago

I also am not sure how you would identify this with confidence. Sure, I can see the tests haven't failed on CI. However, maybe the tests fail all the time while the tests are being run during development.

MaulingMonkey 10y ago

> This seems like a _terrible_ mistake. I can at least confirm with certainty that it sometimes is a terrible mistake. I port things. One approach I sometimes use is to stub in noop implementations of platform specific things. I generally use #pragmas or the like to generate a warning to come back to (which I can always later crank up to an error to ensure I've gone back and implemented everything.) It's common that the original authors were also "kind enough" to leave some noop implementations of their own - although usually not with the helpful #pragmas. End result: Thousands of failing unit tests, most of which were passing for more than a year. Some are innocuous as minor string formatting changes caused by minor libc differences - but even that can be concerning if you want consistent, reliable results that don't vary by platform and threaten to cause platform specific edge cases. I've never seen a codebase where even 10% of the failures are that innocuous.