3 ms·
I think it's easy to conclude code coverage is not a proxy for useful tests. But the lack of any significant degree of code coverage does tell you something. An
by drittich 5y ago
I think it's easy to conclude code coverage is not a proxy for useful tests. But the lack of any significant degree of code coverage does tell you something. And encountering zero code coverage is certainly a problem. So, there is a continuum here, and a balance to be found.
I have come to realize there are many reasons we test, and they are often in competition with each other for getting developer's time to work on them. (Yes, like the Selfish Gene, having us write more code is simply the way tests get us to write more tests, which is really the end goal of it all.)
Lately I have been toying with the idea of creating different suites of tests for each of these testing concerns, so I could track coverage across them independently. I'm thinking of a) basic proof-of-execution tests, which some people think don't tell you much, but in certain codebases, simply validating that you can build a string of SQL from a varied bunch of parameters and execute it without getting a parsing error provides a significant amount of value, regardless of the results returned. And, you can write that test knowing nothing of the business logic. Another level of tests might be concerned with b) the structure of the method: testing all branches or combinations of parameters. Another suite c) would focus on functional tests. And then you have d) mutation and e) characterization tests. For different parts of the codebase, you might choose to favour increasing coverage for one of these suites over the others. You also might wish to add a suite of security or performance tests.
You don't want to get too granular, but the reason code coverage as a metric is not super helpful is we don't necessarily know what those tests are doing, so the more we can constrain their focus, the more meaningful the code coverage results become. And as a bonus, some of these types of tests are very quick to write (a, d, and e), so you can get quickly get to some useful degree of coverage even on a code base you don't understand, or that has no coverage at all. The value of this is not to be discounted, as writing tests can be very time-consuming, and seem like an impossible task when faced with a large, legacy codebase.