3 ms·

> If you want to reduce your test mass, the number one thing you should do is look at the tests that have never failed in a year and consider throwing them away

> If you want to reduce your test mass, the number one thing you should do is look at the tests that have never failed in a year and consider throwing them away. They are producing no information for you — or at least very little information. I feel like there is some subtlety here that most will miss. Specifically, "and consider". That is the part we are really bad at.

alkonaut 7y ago
The tests that never failed for years are usually unit tests for fundamental things. Say your app has a math library at the bottom as in my example. It has thousands of years for things like vector additions, matrix multiplication and so on. And of course these never fail - they have no dependencies and we never modify it anymore. But we might modify it, say to plug in some other library to do math in 5 or 10 years. I don’t want to delete them for that reason. Luckily this kind of low level test usually takes very little time - we have a 15k test suite and the 5k “fundamental” tests take seconds while the whole integration level takes hours.
acchow 7y ago
Those tests are helpful when you do major architectural changes, which may not happen every year. Tests also help in general development. Make some changes, then fix the tests. If you are running tests locally on your machine (and offline) how can you be sure that every time a test fails locally the failure is logged?
- kazinator 7y ago
  ... and you don't always know what is a "major architectural change" that's going to break one of these tests, so you can't just relegate those tests into a category that is only run for major architectural changes.
- war1025 7y ago
  There's a difference between tests that haven't broken because they are in stable code, and tests that haven't failed because they are testing tautologies or even just completely failing to break when the underlying code is broken. I've lost count of how many times I've gone into an old test suite at work and found that the tests were still passing even though the code they were testing had completely changed or been removed. Sometimes tests are written very poorly. The codebase benefits from their removal.

alkonaut 7y ago

The tests that never failed for years are usually unit tests for fundamental things. Say your app has a math library at the bottom as in my example. It has thousands of years for things like vector additions, matrix multiplication and so on. And of course these never fail - they have no dependencies and we never modify it anymore. But we might modify it, say to plug in some other library to do math in 5 or 10 years. I don’t want to delete them for that reason. Luckily this kind of low level test usually takes very little time - we have a 15k test suite and the 5k “fundamental” tests take seconds while the whole integration level takes hours.

acchow 7y ago

Those tests are helpful when you do major architectural changes, which may not happen every year. Tests also help in general development. Make some changes, then fix the tests. If you are running tests locally on your machine (and offline) how can you be sure that every time a test fails locally the failure is logged?

kazinator 7y ago
... and you don't always know what is a "major architectural change" that's going to break one of these tests, so you can't just relegate those tests into a category that is only run for major architectural changes.
war1025 7y ago
There's a difference between tests that haven't broken because they are in stable code, and tests that haven't failed because they are testing tautologies or even just completely failing to break when the underlying code is broken. I've lost count of how many times I've gone into an old test suite at work and found that the tests were still passing even though the code they were testing had completely changed or been removed. Sometimes tests are written very poorly. The codebase benefits from their removal.