2 ms·
I disagree. Coverage and mutation measure completely different things, one is not "a way of doing" the other: - Coverage measures how much of your code is exer
by the_af 9y ago
I disagree. Coverage and mutation measure completely different things, one is not "a way of doing" the other:
- Coverage measures how much of your code is exercised by the tests. It does NOT measure test effectiveness (it's not like mutation, only "less/more brittle"). This can be trivially shown by writing tests that exercise all of your code but have no asserts (or trivial asserts such as 1 == 1). Unsurprisingly, this kind of obviously ineffective test code is often find in the wild (mostly written by junior devs), but less obvious cases of ineffective tests are also common, such as failing to test border conditions. This happens and it's more common than we'd like.
- Mutation testing is a way (but not the only one!) to measure how effective your tests are at actually finding bugs. That this technique exists shows that there is indeed a need to "test the tests", i.e. measure test quality & effectiveness. Production code is NOT "the test of the tests", as someone up this thread erroneously said.
This is like measuring altitude and airspeed: yes, they both measure something useful about your airplane, but they measure different things!
I confess I did not understand the rest of your post.
- Vinnl 9y agoI didn't mean coverage measurement is a way of doing mutation testing, I meant that measuring how much of your code is exercised by your tests is a way of measuring test effectiveness. Yes, it can be inaccurate, as your trivial example shows, but it does guide you towards unhandled test cases (i.e. where your tests are ineffective). Likewise, mutation testing guides you towards unhandled test cases (including ones that coverage can't detect).
- the_af 9y ago> I meant that measuring how much of your code is exercised by your tests is a way of measuring test effectiveness. The problem is that coverage alone is a very poor way of measuring effectiveness, which is why other techniques -- including, but not limited to, mutation testing -- are needed. This is nothing new: limitations of coverage are well known in software engineering.