4 ms·
To be fair, I worked at a company that had CI for running unit tests so nothing could be merged into remote master until the build was green. The build asserte
by ThePadawan 6y ago
To be fair, I worked at a company that had CI for running unit tests so nothing could be merged into remote master until the build was green.
The build asserted that the number of failing unit tests was 0.
It turns out that "breaking the unit test library completely such that 0 tests run instead of all thousands of them" did not break the build.
When someone noticed a week later, that was not fun to clean up.
- rwbhn 6y agoRecently added an assertion that the # of tests succeeding was > 0 to cover a similar issue.
- the_af 6y agoTests that test nothing useful are surprisingly easy to write, and lead to a similar result -- even in cases where you do run them! That's why it makes some sense to test the tests (or at least, check their bug-detecting quality).
- Tainnor 6y agoThat's why you should IMHO always make sure you see your test fail at least once. TDD gives you that for free, but you don't have to use TDD: just manually breaking your implementation to see if the test breaks in the way it's supposed to should work equally well (bonus points if you make sure the error message is also precise enough to debug the issue).
- the_af 6y agoYes. But just like we automate tests instead of manually testing -- that's what computers are for after all: automation -- we can automate the tests of tests! Also, tests that once made sense might have become useless, so it seems relevant to have a way of automatically detecting obsolete "this doesn't test anything" tests even though they could have been useful once. Without relying too much on human intervention, that is.
- Tainnor 6y agothat's basically mutation testing but I've never seen a place that has been able to use it... maybe at some point the tools will get better