4 ms·
I've always wondered about building a test-sensitivity metric to complement coverage (and thus make coverage more useful): - Pick a random line of code (skip c
by drodgers 5y ago
I've always wondered about building a test-sensitivity metric to complement coverage (and thus make coverage more useful):
- Pick a random line of code (skip comments and whitespace if you can)
- Delete/modify it in some way
- Run the test suite and see if it passes
Do this thousands of times on a codebase, then report on the test suite failure rate accross all of the random changes, broken down by file etc.
You could track it all in a long running database and focus each test on lines which have recently changed to make it less expensive.
I've never actually done this, but I'd guess that you could get a good sense of which files have good tests, and (combined with traditional coverage data) which files have bad tests vs just no tests.
- ericpauley 5y agoThis is known as mutation testing[1]. [1] https://en.m.wikipedia.org/wiki/Mutation_testing https://en.m.wikipedia.org/wiki/Mutation_testing
- anonymoushn 5y agoSome especially corectness-concerned or compatibility-concerned people use mutation testing, which changes each line of code and fails the test run if no tests fail as a result of a change.