3 ms·
Notice the result is a bit more nuanced if reading the paper, i.e. > Our results suggest that, for large Java programs, the correlation between coverage and ef
by riffraff 11y ago
Notice the result is a bit more nuanced if reading the paper, i.e.
> Our results suggest that, for large Java
programs, the correlation between coverage and effectiveness
drops when suite size is controlled for.
and
> While coverage
measures are useful for identifying under-tested parts of a
program, and low coverage may indicate that a test suite is
inadequate, high coverage does not indicate that a test suite
is effective.
They also propose an alternative as a "quality goal" for test suites:
> Of course, developers still want to measure the quality
of their test suites, meaning they need a metric that does
correlate with fault detection ability. While this is still an
open problem, we currently feel that mutation score may be
a good substitute for coverage in this context.
- daveguy 11y agoI wasn't familiar with mutation testing. This is also from the article: >We used the open source tool PIT [35] to generate faulty versions of our programs. To describe PIT’s operation, we must first give a brief description of mutation testing. >A mutant is a new version of a program that is created by making a small syntactic change to the original program. For example, a mutant could be created by modifying a constant, negating a branch condition, or removing a method call. The resulting mutant may produce the same output as the original program, in which case it is called an equivalent mutant. For example, if the equality test in the code snippet in Figure 1 were changed to if (index >= 10), the new program would be an equivalent mutant. Sounds like a resource intensive way to test -- # tests * loc * mutation options. However, if you need the fault tolerance it could be worth it. You can probably spot check to get similar quality with lower resources. Very interesting.
- julian37 11y agopitest has a few tricks up its sleeve to keep overhead reasonable. For example (from http://pitest.org/faq/ http://pitest.org/faq/): > Per test case line coverage information is first gathered and all tests that do not exercise the mutated line of code are disguarded. I've used pitest in real life and while it is quite a bit heavier than straightforward testing, it's definitely usable. I'm now using it on all Java projects, I wish there was something comparable for JavaScript!
- mordocai 11y agoI don't think it is quite as complete, but this looks usable https://github.com/jimivdw/grunt-mutation-testing https://github.com/jimivdw/grunt-mutation-testing. That's just from google, no experience with it.
- lmmi 11y agoComputing the mutation score using a randomly selected subset of the mutants seems to work well, though it hasn't been studied a lot. One paper is "Reducing the cost of mutation testing: An empirical study" by Wong and Mathur.
- fuhrysteve 11y ago>> for large Java programs I wonder if the results would be different at all for a less strongly typed language, like python for example.
- lmmi 11y agoPaper author here. We did a follow up study and found that the mutation score is, in fact, correlated with the suite's ability to detect real faults, at least for the programs we studied. That paper is also on my website: "Are Mutants a Valid Substitute for Real Faults in Software Testing?"