2 ms·
While I think it's well-agreed upon in this community that code coverage is not all that big a deal, I think we also should consider the methods used in this st
by mabbo 11y ago
While I think it's well-agreed upon in this community that code coverage is not all that big a deal, I think we also should consider the methods used in this study before we all pat ourselves on the back for having 'proof' of what we know.
> we generated 31,000 test suites for five systems consisting of up to 724,000 lines of source code
You auto-generated unit tests suites, and you're surprised they weren't very good at finding bugs? Well, no kidding, they were auto-generated! Would you trust your unit tests to be generated by a computer? Of course not.
Do a study of real-world software, and compare the unit test coverage to the Test Suite effectiveness. Then I'll be interested.
- jhpriestley 11y agoThe unit tests are written by the programmers on each project, not generated by a computer. The test suites are generated by selecting a subset of the project's real-world test suite.
- lmmi 11y agoPaper author here. As someone already mentioned, the suites were made by picking a random subset of the test cases written by the developers, not by using a suite generation tool. Now, our method is arguably not 100% realistic either, because in practice different parts of the program will have different coverage. Some kind of weighted random sampling might make suites that are a bit closer to "real" suites. I've been thinking about looking into that. I think our suites are realistic enough to make the results worthwhile, though.