3 ms·

> In other words, writing another test for the hot section of your code seems to improve quality on average about as much as writing a test for some rarely run

> In other words, writing another test for the hot section of your code seems to improve quality on average about as much as writing a test for some rarely run code that you didn't bother to test until now. I'm not sure. I've given it some more thought. I think the problem with distinguishing between the two is that "coverage" is defined, so to speak, as an area. Once an LOC is executed by any test, its value ticks over once. It never gets counted again. If instead each LOC execution was summed up independently, each LOC gets a "height" -- the number of times it was exercised in a suite. Then the overall suite gets test coverage volume, rather than test coverage area. Once you track per-LOC volumes, you could start trying to tease out the differences between introducing a test at all and adding another test to critical code. And, I suspect, the correlation between coverage volume and bug yield would be much more robust than area when controlled for number of cases. Because area when controlling for number of cases is ... a rough proxy for test coverage volume. Edit: though with an interesting difference. Coverage area converges to 100%, but coverage volume is effectively unbounded. This would hopefully upset the abuse of coverage as a management metric instead of as a code smell.

marcosdumay 11y ago
My thoughts go that way too, but I expected some volumes to be more important than others. And because of it, I expected the article to measure a negative correlation. Hence my surprise. But reading the paper, the study is about random samples of the complete test suite, and the results are far too weak to get any conclusion about it, finding negative correlations some times, and weak positive other times. Anyway, they got a strong correlation, with good p-value (not very worthy on this context, but it's all we get) for your hypothesis about coverage volume. It's just my hypothesis about some code being more important than other that is inconclusive.
- jacques_chester 11y ago
  > It's just my hypothesis about some code being more important than other that is inconclusive. There's a lot of literature going back a long way showing that defects tend to cluster. A leading indicator might be defects found in a module divided by coverage volume. On the other hand, defects tend to cluster in the hard stuff, not at random.

marcosdumay 11y ago

My thoughts go that way too, but I expected some volumes to be more important than others. And because of it, I expected the article to measure a negative correlation. Hence my surprise. But reading the paper, the study is about random samples of the complete test suite, and the results are far too weak to get any conclusion about it, finding negative correlations some times, and weak positive other times. Anyway, they got a strong correlation, with good p-value (not very worthy on this context, but it's all we get) for your hypothesis about coverage volume. It's just my hypothesis about some code being more important than other that is inconclusive.

jacques_chester 11y ago
> It's just my hypothesis about some code being more important than other that is inconclusive. There's a lot of literature going back a long way showing that defects tend to cluster. A leading indicator might be defects found in a module divided by coverage volume. On the other hand, defects tend to cluster in the hard stuff, not at random.