lmmi

3 ms·

1.
by lmmi 11y ago
What you're looking for is in Figure 3 (admittedly a bit hard to read because I had to squish it into the paper). Each panel in that figure shows the results for test suites of a fixed size. For example, the top left panel shows the
2.
by lmmi 11y ago
Paper author here. We measured "effectiveness" as the mutation score. We showed in a separate paper that the mutation score is a good way to measure a test suite's ability to detect real faults. Of course, using the mutati
3.
by lmmi 11y ago
Those are good ideas, thanks!
4.
by lmmi 11y ago
I'd definitely say that the absence of coverage is a problem. My view is that coverage is necessary but not sufficient for good testing. We talked a bit about classifying tests but didn't do it in the end because it's surpri
5.
by lmmi 11y ago
Paper author here. As someone already mentioned, the suites were made by picking a random subset of the test cases written by the developers, not by using a suite generation tool. Now, our method is arguably not 100% realistic either, beca
6.
by lmmi 11y ago
Computing the mutation score using a randomly selected subset of the mutants seems to work well, though it hasn't been studied a lot. One paper is "Reducing the cost of mutation testing: An empirical study" by Wong and Mathu
7.
by lmmi 11y ago
Paper author here. We did a follow up study and found that the mutation score is, in fact, correlated with the suite's ability to detect real faults, at least for the programs we studied. That paper is also on my website: "Are M
8.
by lmmi 11y ago
Paper author here. Scoring coverage for every pass is an interesting idea and something I'd like to look into. I'm not sure it would change the result as much as you think, though. The basic finding of the paper is that coverag