6 ms·

I need to ask a stupid question. What does one do with code coverage data? Like, I know what it is. What I'm asking is, on what occasion would the presence of

I need to ask a stupid question. What does one do with code coverage data? Like, I know what it is. What I'm asking is, on what occasion would the presence of code coverage data be more useful than stepping through the code in a debugger. What is something you do differently having this data than not? Like, is there a code review gate that forbids the code coverage figure to go below 80%? Or do developers consult the reports in a self-motivated way for some reason? Regularly? What is the step between "collect code coverage reports" and "profit"? We recently got this feature in our compiler, and I guess it seems obvious to everyone why to use it, but I must have missed that day in class. So everybody's really excited to create the report, but I can't figure out what they do with it afterward that explains the whole exercise.

danpat 11y ago
Run your test suite, look at the code coverage report, find out which bits of your code doesn't get exercised by the test suite, become aware that changes in those bits of your code probably won't trigger any failing tests if they introduce breakage.
cornstalks 11y ago
Sometimes there's a disconnect between what you think you're testing and what you're actually testing. After writing a unit test, I like to look at the code coverage analysis for that unit test and see if it actually tests the code that I think it tests. After seeing the analysis, sometimes I realize I'm not testing for a weird corner case that I should be. It also slightly gamifies unit testing because there's something slightly satisfying in seeing 100% code coverage (though it's important to remember that 100% code coverage doesn't necessarily mean your tests are perfect or test all the conditions you should be).
wglb 11y ago
Code coverage is simple a measure of the amount of the code touched by your tests. It doesn't tell you the quality of effectiveness of the tests. Only what code got executed during your tests.
etwigg 11y ago
> is there a code review gate that forbids the code coverage figure to go below 80%? That's one usage pattern. Here are two others, more focused on individual productivity rather than team standards. 1. Write some code, write a test, run the test. Rather than stepping through the test with a debugger, you enable code coverage and can see the code path instantly. Same idea as a debugger, just faster and less precise. Occasionally a test passes for the wrong reason. Looking at coverage data is a fast way to find out that you didn't quite understand what your test did. 2. Write some code, and write a quickcheck-style test (generate random data, assert that properties hold on the output). Look at the code coverage, and see if the random data is covering all of the code paths. If it is, hurrah, you just built a great test with almost no effort. If the random data isn't triggering some code paths, you can now target your efforts on that unexplored branch.
rkday 11y ago
We actually hook it into our build process. Basically: * we have 100% code coverage except for some well-defined exclusions (e.g. // LCOV_EXCL_LINE comments to exclude code that is logically unhittable) * on each checkin, our continuous integration server (Jenkins) runs the tests and checks that coverage is still 100% * if not, it fails the build This means that _every new checkin_ either has unit tests that exercise all the code, or // LCOV_EXCL comments that make it obvious to a code reviewer what isn't covered (and they can then sensibly judge the risk). It's a good way to keep our code quality high, and avoid the possibility that we quietly skimp on unit tests when under deadline pressure. Also, if new members of the team don't realise how important good unit test coverage is, they'll find that out on their first checkin when the build breaks, rather than several weeks in.
- overgard 11y ago
  With all-due respect to other philosophies than mine, that sounds like a nightmare. And I like tests. But everything has a cost, and I can't imagine ever getting anything significant done with such a straight-jacket, I'd spend all day tracking down weird compiler quirks. I do a lot of cross platform stuff, and just having warnings as errors (a good idea) means a lot of build fixing because clang and MSVC disagree on things. If I were writing code in MSVC and I had to deal with code coverage reports from clang breaking the build, I would be paralyzed. "Code quality" isn't a thing that can be trivially measured, and in my experience people that fixate on certain metrics tend to miss the forest for the trees.
  - rkday 11y ago
    I can see that it wouldn't always be ideal, but on this (admittedly relatively niche) codebase - core telephone network infrastructure, which we only support on Ubuntu and which we build with GCC - the kind of clang/MSVC incompatibilities you talk about aren't a problem, and the reliability requirements are high enough that it's worth having CI check for 100% test coverage. I think it's a question of field rather than philosophy - someone else in this thread said they do the same thing in avionics.
    - 11y ago
CuriousSkeptic 11y ago
Another language. But I find this kind of usage pretty useful http://www.ncrunch.net http://www.ncrunch.net Basically instant feedback on performance and conformance.
skocznymroczny 11y ago
I saw it used in the case of limited machine hours, if the test content is so huge, you can't run it all per checkin, you can use code coverage to get the minimum subset of tests that covers the maximum amount of code.
lantastic 11y ago
One use case already pointed out is to check for gaps that your tests have not covered. Other use case is figuring out what tests to re-execute when you change something - you can skip tests that never touch any code you are changing. Can be a time-saver if you have a big test suite or lack finer-grained tests (component/unit).
- viraptor 11y ago
  This sounds a bit dangerous. Code coverage works on code, not on data, so if you're testing something like: // globals rules_t rules[] = {{1, OK}, {2, FAIL}, {3, THERMONUCLEAR_WAR}}; It sounds like you could change rules 1 and 2 to THERMONUCLEAR_WAR as well and none of the tests would rerun. Is that right?
  - stouset 11y ago
    You might do something like this for local development, keeping things fast with some confidence you're not grossly breaking things. But you run the full suite through CI.
WalterBright 11y ago
Think of it this way. What are the odds of the code working correctly when the first time it is run is when it is in the customer's hands? I've used code coverage analyzers on and off for 30 years. I know that merely executing code does not mean it is bug free. But I've found a very strong relationship between higher coverage and fewer bugs found in the field. There've been reports on reddit that this is not so in other companies, but I get the feeling they're doing something wrong, because it utterly is not my experience.
tjr 11y ago
It's required in avionics software. Ideally 100% of the code in an avionics component will be executed in testing, but whatever isn't needs to be explained as to why it's acceptable to have been missed.
whitenoice 11y ago
In the newer version of intellij once you run code coverage it highlights in the side bar if that particular line was covered or not, its a pretty neat feature.

danpat 11y ago

Run your test suite, look at the code coverage report, find out which bits of your code doesn't get exercised by the test suite, become aware that changes in those bits of your code probably won't trigger any failing tests if they introduce breakage.

cornstalks 11y ago

Sometimes there's a disconnect between what you think you're testing and what you're actually testing. After writing a unit test, I like to look at the code coverage analysis for that unit test and see if it actually tests the code that I think it tests. After seeing the analysis, sometimes I realize I'm not testing for a weird corner case that I should be. It also slightly gamifies unit testing because there's something slightly satisfying in seeing 100% code coverage (though it's important to remember that 100% code coverage doesn't necessarily mean your tests are perfect or test all the conditions you should be).

wglb 11y ago

Code coverage is simple a measure of the amount of the code touched by your tests. It doesn't tell you the quality of effectiveness of the tests. Only what code got executed during your tests.

etwigg 11y ago

> is there a code review gate that forbids the code coverage figure to go below 80%? That's one usage pattern. Here are two others, more focused on individual productivity rather than team standards. 1. Write some code, write a test, run the test. Rather than stepping through the test with a debugger, you enable code coverage and can see the code path instantly. Same idea as a debugger, just faster and less precise. Occasionally a test passes for the wrong reason. Looking at coverage data is a fast way to find out that you didn't quite understand what your test did. 2. Write some code, and write a quickcheck-style test (generate random data, assert that properties hold on the output). Look at the code coverage, and see if the random data is covering all of the code paths. If it is, hurrah, you just built a great test with almost no effort. If the random data isn't triggering some code paths, you can now target your efforts on that unexplored branch.

rkday 11y ago

We actually hook it into our build process. Basically: * we have 100% code coverage except for some well-defined exclusions (e.g. // LCOV_EXCL_LINE comments to exclude code that is logically unhittable) * on each checkin, our continuous integration server (Jenkins) runs the tests and checks that coverage is still 100% * if not, it fails the build This means that _every new checkin_ either has unit tests that exercise all the code, or // LCOV_EXCL comments that make it obvious to a code reviewer what isn't covered (and they can then sensibly judge the risk). It's a good way to keep our code quality high, and avoid the possibility that we quietly skimp on unit tests when under deadline pressure. Also, if new members of the team don't realise how important good unit test coverage is, they'll find that out on their first checkin when the build breaks, rather than several weeks in.

overgard 11y ago
With all-due respect to other philosophies than mine, that sounds like a nightmare. And I like tests. But everything has a cost, and I can't imagine ever getting anything significant done with such a straight-jacket, I'd spend all day tracking down weird compiler quirks. I do a lot of cross platform stuff, and just having warnings as errors (a good idea) means a lot of build fixing because clang and MSVC disagree on things. If I were writing code in MSVC and I had to deal with code coverage reports from clang breaking the build, I would be paralyzed. "Code quality" isn't a thing that can be trivially measured, and in my experience people that fixate on certain metrics tend to miss the forest for the trees.
- rkday 11y ago
  I can see that it wouldn't always be ideal, but on this (admittedly relatively niche) codebase - core telephone network infrastructure, which we only support on Ubuntu and which we build with GCC - the kind of clang/MSVC incompatibilities you talk about aren't a problem, and the reliability requirements are high enough that it's worth having CI check for 100% test coverage. I think it's a question of field rather than philosophy - someone else in this thread said they do the same thing in avionics.
  - 11y ago

CuriousSkeptic 11y ago

Another language. But I find this kind of usage pretty useful http://www.ncrunch.net http://www.ncrunch.net Basically instant feedback on performance and conformance.

skocznymroczny 11y ago

I saw it used in the case of limited machine hours, if the test content is so huge, you can't run it all per checkin, you can use code coverage to get the minimum subset of tests that covers the maximum amount of code.

lantastic 11y ago

One use case already pointed out is to check for gaps that your tests have not covered. Other use case is figuring out what tests to re-execute when you change something - you can skip tests that never touch any code you are changing. Can be a time-saver if you have a big test suite or lack finer-grained tests (component/unit).

viraptor 11y ago
This sounds a bit dangerous. Code coverage works on code, not on data, so if you're testing something like: // globals rules_t rules[] = {{1, OK}, {2, FAIL}, {3, THERMONUCLEAR_WAR}}; It sounds like you could change rules 1 and 2 to THERMONUCLEAR_WAR as well and none of the tests would rerun. Is that right?
- stouset 11y ago
  You might do something like this for local development, keeping things fast with some confidence you're not grossly breaking things. But you run the full suite through CI.

WalterBright 11y ago

Think of it this way. What are the odds of the code working correctly when the first time it is run is when it is in the customer's hands? I've used code coverage analyzers on and off for 30 years. I know that merely executing code does not mean it is bug free. But I've found a very strong relationship between higher coverage and fewer bugs found in the field. There've been reports on reddit that this is not so in other companies, but I get the feeling they're doing something wrong, because it utterly is not my experience.

tjr 11y ago

It's required in avionics software. Ideally 100% of the code in an avionics component will be executed in testing, but whatever isn't needs to be explained as to why it's acceptable to have been missed.

whitenoice 11y ago

In the newer version of intellij once you run code coverage it highlights in the side bar if that particular line was covered or not, its a pretty neat feature.