3 ms·
"Next, they selected random subsets of the project's original test suite of varying sizes, ran each against the buggy mutants, and counted how often the subsett
by ummonk 5y ago
"Next, they selected random subsets of the project's original test suite of varying sizes, ran each against the buggy mutants, and counted how often the subsetted test suite caught the bug."
I'm sorry what? Of course if you only run a random subset of tests, it won't matter whether the test suite achieves 100% line coverage. What a garbage study.
- TFortunato 5y agoNobody is arguing that usong a subset of tests won't catch all bugs. They are stating that having X% code coverage does not imply that you will be able to catch X% of bugs (at least, of the trivial type they introduced here) so you shouldn't use code coverage as a proxy for how good your test suite is. This may be obvious to you, but I've definitely seen teams fall into the trap of using code coverage as a primary measure of how good testing is.