4 ms·

The metric is meaningful; I think you’re misinterpreting it. To your point, 100% coverage doesn’t mean you’ve eliminated all bugs, but it does mean that your co

The metric is meaningful; I think you’re misinterpreting it. To your point, 100% coverage doesn’t mean you’ve eliminated all bugs, but it does mean that your code base almost certainly has a lower bug yield that the code base with 50% coverage (assuming no one has games the metric). If you really think that the metric is meaningless and useless for deriving confidence, then you are necessarily asserting that code bases with 100% coverage have indistinguishable bug yields compared to those with 50%, 5%, or even 0% coverage. A claim like this is too extraordinary to be believed without considerable evidence.

kbp 8y ago
I guess it's useful for deriving a baseline level of confidence, like a low coverage score is a red flag, and an increasing coverage score probably corresponds to increasing test coverage, but my issue is that 100% coverage doesn't mean anything about the correctness of the code in absolute terms (unless 'gaming the metric' includes not thinking of every edge case, ie, that we're assuming the existing test suite is perfect). If you're working on a poorly tested codebase, it's a useful relative metric of your progress in testing what already exists, but unless you're assuming the code is already correct, that doesn't mean anything more than that. If you wanted to derive, say, the confidence that you won't see 500's daily in production from a metric, then line coverage isn't an effective one to use for that; the tests you write that give you that kind of confidence don't really help your coverage score. The parts of the codebase that are in the most urgent need of tests for getting that kind of confidence in places will most of the time be ones that already have good coverage; think of how TDD works, even if you're not doing TDD. I could agree that a high coverage score is a prerequisite for having confidence that your test suite is comprehensive (in general), but that's such a low bar, it's like saying a full bath is a prerequisite for a nice house, just knowing that shouldn't do much to convince you it's a mansion.

kbp 8y ago

I guess it's useful for deriving a baseline level of confidence, like a low coverage score is a red flag, and an increasing coverage score probably corresponds to increasing test coverage, but my issue is that 100% coverage doesn't mean anything about the correctness of the code in absolute terms (unless 'gaming the metric' includes not thinking of every edge case, ie, that we're assuming the existing test suite is perfect). If you're working on a poorly tested codebase, it's a useful relative metric of your progress in testing what already exists, but unless you're assuming the code is already correct, that doesn't mean anything more than that. If you wanted to derive, say, the confidence that you won't see 500's daily in production from a metric, then line coverage isn't an effective one to use for that; the tests you write that give you that kind of confidence don't really help your coverage score. The parts of the codebase that are in the most urgent need of tests for getting that kind of confidence in places will most of the time be ones that already have good coverage; think of how TDD works, even if you're not doing TDD. I could agree that a high coverage score is a prerequisite for having confidence that your test suite is comprehensive (in general), but that's such a low bar, it's like saying a full bath is a prerequisite for a nice house, just knowing that shouldn't do much to convince you it's a mansion.