6 ms·
This tool seems like it would encourage writing exactly the kind of tests I hate: change detectors. Tests that break every time you change the code regardless o
by modeless 8y ago
This tool seems like it would encourage writing exactly the kind of tests I hate: change detectors. Tests that break every time you change the code regardless of whether the change introduced a bug or not.
Change detector tests are worse than useless because they double the time it takes to make a code change, while not providing any useful information. I already know I changed the code, I don't need a test to tell me that.
We should aspire to write tests that aren't just change detectors, but catch real bugs. Perhaps a tool like this could be useful in an inverse way, to tell you that your test is sensitive to irrelevant details and should be made more general.
- forkerenok 8y agoFirst, it's worth noting that the output of the mutation test is also an heuristic. Secondly, I have really hard time imagining a case where, a change in branching logic in code, that doesn't trigger test failure, is not signalling a "coverage smell". Edit: And I do totally relate to what you are saying about change traps. And, I think, if such practice is already prevalent in the codebase, mutation testing will exacerbate it.
- systemtest 8y agoI agree, I have seen too many unit tests that copy the implementation of the method they test. TDD might solve this.
- Eridrus 8y agoThe one time this system complained about my tests it found a real bug in my test, though not an actual issue with the program under test.
- golergka 8y agoThat's exactly what I like about good tests: they serve as a double-entry bookkeeping system. Telling me that I changed something is very useful, because they help me verify that I changed only the things that I wanted to change.
- deleted 8y ago[deleted]
- rossjudson 8y agoEffective code tends to put a lot of functionality behind a relatively small interface, and tends to work for a long time without needing attention. A "change detector" is exactly what you want in this situation. Does the code still fulfill the spec (tests)? Tests that feel like they need to be changed every time the code changes are generally white box instead of black box, or the interface to the code being tested is too large.
- jjoonathan 8y ago> Effective code tends to put a lot of functionality behind a relatively small interface Do you have actual strategies for reducing high surface-area:volume requirements into more easily managed low surface-area:volume code or is this just wishful thinking? Nobody likes shim layers but they (and other horizontal strata) happen for a reason.
- stgpetrovic 8y agoI completely agree that change detector tests are worse than useless, they often have negative impact. When designing mutation testing, we have spent most of our time designing suppresion heuristics to avoid generating bad mutants. Some bad mutants fall into the category of change-detector tests, but there are many more categories (see the Appendinx for a taste). Finally, mutants don't have to be killed, they just point out a potential weakness in the test suite, and it's up to the developers to make that call.
- ur-whale 8y agoI understand exactly what you mean, and I used to have the same negative attitude towards "change detectors", but I have actually changed my mind about them and come to appreciate their value. I suspect the root cause of your headache with them is that you hate to see tests break when you change your code, probably because you view fixing the test as costly and a distraction. But I've come to see breaking "change detectors" as a source of comfort, sort of confirming what I expect my change to do. What you need to get there is to make test "fixing" more agile and less costly. I do understand that in a large shared codebase, that is easier said than done, but on smaller projects that's easier to do.
- nitwit005 8y agoIt depends on how the tests work. If they all treat the program as a black box, there should be no problem. If the tests are unit tests that use a lot of mocks, then yes, this sort of methodology might produce crappy results.