6 ms·

Just because code is touched during the test process doesn’t necessarily mean it has been tested. Coverage is more useful for finding chunks of code that aren’t

Just because code is touched during the test process doesn’t necessarily mean it has been tested. Coverage is more useful for finding chunks of code that aren’t exercised at all by tests. Branches that never get hit that probably deserve extra scrutiny. Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code. Likewise, there’s stuff that’s just not worth it to wrap in a test. The effort involved in mocking/wrapping/abstracting sometimes isn’t worth the effort for some code, like simple calls to common libraries, reading a file, CLI help strings, etc. This stuff generally is better served by end-to-end or integration tests. Or just gets exercised naturally as part of development. Or can crash without much consequence. My point is coverage is a tool that tells you where to look, not the one and only metric that tells you how reliable your code is.

g14i 6y ago
Totally agree. In one of our projects (around 200K Python LoC) we had 90% CodCov, but we frequently found some bugs. Recently we started to heavily use mutation testing and fuzzing to find edge cases on parts of the code that were already "covered" according to the CodCov report. It was definitely worth it. I highly recommend investing in mutation and fuzzy testing.
- whymarrh 6y ago
  I was just about to bring up mutation testing. I've had some pretty great success with PIT [1] when writing Java. Code coverage and mutation test coverage pair wonderfully together. [1]:https://pitest.org
  - g14i 6y ago
    whymarrh, great that you mentioned PIT. This is exactly what we use for Java as well. For Python we use mutmut [1]. [1] - https://pypi.org/project/mutmut/ https://pypi.org/project/mutmut/
    - boxed 6y ago
      Mutmut author here. Great to hear you're using mutmut! I think it's pretty important that we who know about MT and have used it keep talking about it. It's a pretty great tool that more people should have in their tool belt.
  - TeaVMFan 6y ago
    Mutation coverage definitely improves the testing process. It is much harder to accidentally write bad tests once you are using a tool like PIT: https://frequal.com/java/HighQualityTestsWithPitMutationTesting.html https://frequal.com/java/HighQualityTestsWithPitMutationTest...
- ay 6y ago
  test coverage is calculated as % of the locations seen by profiler during the test runs, right ? This metric completely ignores the fact that every independent conditional doubles the space of the behaviors of the program. Assuming we have three conditions and three sequential groups of basic blocks chosen corresponding to true (X) and false (~X) values of these conditions, testing A+B+C and ~A+~B+~C code paths gives 100% code coverage but it covers at the very best only 25% of the code behaviors.
  - dalke 6y ago
    There are many types of test coverage measurement. I believe path coverage ("Has every possible route through a given part of the code been executed?") is your latter example. Quoting https://en.wikipedia.org/wiki/Code_coverage https://en.wikipedia.org/wiki/Code_coverage which also says: > Full path coverage, of the type described above, is usually impractical or impossible. ... a general-purpose algorithm for identifying infeasible paths has been proven to be impossible (such an algorithm could be used to solve the halting problem).[14] Basis path testing is for instance a method of achieving complete branch coverage without achieving complete path coverage."
Mindless2112 6y ago
I like this parallelism between coverage and tests: Coverage can prove that code is not tested but cannot prove that code is tested. Tests can prove that bugs exist but cannot prove that bugs do not exist.
- boxed 6y ago
  Mutation testing can prove that the tests check all the behavior of the code, but it can't prove that the code shouldn't have more behavior, or that this behavior is correct.
xienze 6y ago
> Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code. Absolutely correct. But at the same time, <100% coverage means you’ve got code where you can’t be sure there are no bugs because it doesn’t even get tested. Strive for full coverage that hits every line AND tests the appropriate scenarios and edge cases.
- Tomis02 6y ago
  Really? You want to strive to test that logging logs and observability observes? You want to test constructors, getters and setters? Testing trivial code brings negative value, why would you do that?
  - nickcox 6y ago
    You probably don't need to test any of those things but I would still expect them to be exercised. You can't very well test a class without constructing it.
  - morelisp 6y ago
    > You want to strive to test that logging logs and observability observes? You're asking this rhetorically, but I often find interesting problems once I start really exercising the latter, like: - Non-global collectors we forgot to register, or fail to get registered on some code paths. - Collectors that get registered twice or overlap with other collectors only in some conditions (e.g. connection addresses that are sometimes the same and sometimes not depending on how your service gets distributed). - Collectors that are not really safe scraping concurrently with the rest of the program; our test suites include race detectors. - Metrics that don't follow idiomatic naming conventions (can be found via e.g. `promtool check metrics`). - Metrics with legitimate difficult-to-notice bugs. We had an underflow counting the size of a connection pool because `Close()`ing one twice was explicitly allowed. > Testing trivial code brings negative value Agreed in the abstract, but if we were good judges of what code was trivial or not, we'd write a lot less bugs in the first place! That being said, this is also a discussion about coverage, not assertions. Even if I'm not actually checking any of those things, those code paths (and logging) are still getting exercised via any decent E2E/functional/behavioral test suite.
jameson 6y ago
> Coverage is more useful for finding chunks of code that aren’t exercised at all by tests. Right on point. High CC% doesn't tell application is well tested but is an indicator of what isn't covered
breck 6y ago
It's a really helpful tool for as you say telling "you where to look". One trick I've learned: if you want to forecast where the bugs will popup next month, take a quick look at the code coverage report. #1 source will be the code that doesn't appear at all (because the test build system didn't even include it) #2 will be all those red lines.
1337shadow 6y ago
Another advantage that does not get the love it deserves in my opinion, is how useful coverage becomes when it comes to upgrading a dependency, or even the language version. But your project will have to stick around for a while to appreciate that.
closeparen 6y ago
When “hold people accountable to metrics” is not just your company’s management philosophy but the gospel of its entire surrounding culture, this is not an argument that gets anywhere. It is always better to just cover the damn lines than make your Director explain to the VP why he has worse coverage numbers than a sister org. The same way you should make an expected “no such object” result a 200 status code, so that she does not have to stand up and justify her error rate.
- capableweb 6y ago
  Note that you're now discussing corporate politics, not just creating software, even though creating software often has to deal with corporate politics. My point is rzimmerman never contextualized this as a "how to navigate the workspace" argument, they are arguing it from a pure technical perspective.
hinkley 6y ago
And that’s why you shouldn’t chase after code coverage percentages. While there are uncovered lines, you have a TODO list that has been generated by the machine, telling you what needs more attention. If you go to that section intending to test it well, you’ll get a lot of lines of test code for a modest increase in coverage. If you’re just trying to boost coverage, you’ll check off the todo without accomplishing much. The only place these two metrics line up at all is when you can keep the number of conditions in each method close to one, and the functions pure. Otherwise you have a bunch of little functions that accumulate state and you’ve just moved the combinatorics where it’s even harder to reason about. I think the most you can say with any certainty is that coverage percentage should not go down when adding code. But staying flat doesn’t prevent things from getting worse.
- zlynx 6y ago
  Except when handling an error condition from a hard to test system call that was previously ignored. Being forced into stupid tricks like adding five useless log statements so that the CI tool doesn't block your change is ... annoying.
voodootrucker 6y ago
You can hit 100% with zero assertions if you like. Probably not the test you want to have though.
- regularfry 6y ago
  Zero assertions means that you know there aren't any lurking exceptions triggered by just running the code. That's a good thing to know, but I'd hope like hell that it's not the only thing you want to know.
  - joshschreuder 6y ago
    It may not even prove that depending on your mock/fake strategy though.

g14i 6y ago

Totally agree. In one of our projects (around 200K Python LoC) we had 90% CodCov, but we frequently found some bugs. Recently we started to heavily use mutation testing and fuzzing to find edge cases on parts of the code that were already "covered" according to the CodCov report. It was definitely worth it. I highly recommend investing in mutation and fuzzy testing.

whymarrh 6y ago
I was just about to bring up mutation testing. I've had some pretty great success with PIT [1] when writing Java. Code coverage and mutation test coverage pair wonderfully together. [1]:https://pitest.org
- g14i 6y ago
  whymarrh, great that you mentioned PIT. This is exactly what we use for Java as well. For Python we use mutmut [1]. [1] - https://pypi.org/project/mutmut/ https://pypi.org/project/mutmut/
  - boxed 6y ago
    Mutmut author here. Great to hear you're using mutmut! I think it's pretty important that we who know about MT and have used it keep talking about it. It's a pretty great tool that more people should have in their tool belt.
- TeaVMFan 6y ago
  Mutation coverage definitely improves the testing process. It is much harder to accidentally write bad tests once you are using a tool like PIT: https://frequal.com/java/HighQualityTestsWithPitMutationTesting.html https://frequal.com/java/HighQualityTestsWithPitMutationTest...
ay 6y ago
test coverage is calculated as % of the locations seen by profiler during the test runs, right ? This metric completely ignores the fact that every independent conditional doubles the space of the behaviors of the program. Assuming we have three conditions and three sequential groups of basic blocks chosen corresponding to true (X) and false (~X) values of these conditions, testing A+B+C and ~A+~B+~C code paths gives 100% code coverage but it covers at the very best only 25% of the code behaviors.
- dalke 6y ago
  There are many types of test coverage measurement. I believe path coverage ("Has every possible route through a given part of the code been executed?") is your latter example. Quoting https://en.wikipedia.org/wiki/Code_coverage https://en.wikipedia.org/wiki/Code_coverage which also says: > Full path coverage, of the type described above, is usually impractical or impossible. ... a general-purpose algorithm for identifying infeasible paths has been proven to be impossible (such an algorithm could be used to solve the halting problem).[14] Basis path testing is for instance a method of achieving complete branch coverage without achieving complete path coverage."

Mindless2112 6y ago

I like this parallelism between coverage and tests: Coverage can prove that code is not tested but cannot prove that code is tested. Tests can prove that bugs exist but cannot prove that bugs do not exist.

boxed 6y ago
Mutation testing can prove that the tests check all the behavior of the code, but it can't prove that the code shouldn't have more behavior, or that this behavior is correct.

xienze 6y ago

> Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code. Absolutely correct. But at the same time, <100% coverage means you’ve got code where you can’t be sure there are no bugs because it doesn’t even get tested. Strive for full coverage that hits every line AND tests the appropriate scenarios and edge cases.

Tomis02 6y ago
Really? You want to strive to test that logging logs and observability observes? You want to test constructors, getters and setters? Testing trivial code brings negative value, why would you do that?
- nickcox 6y ago
  You probably don't need to test any of those things but I would still expect them to be exercised. You can't very well test a class without constructing it.
- morelisp 6y ago
  > You want to strive to test that logging logs and observability observes? You're asking this rhetorically, but I often find interesting problems once I start really exercising the latter, like: - Non-global collectors we forgot to register, or fail to get registered on some code paths. - Collectors that get registered twice or overlap with other collectors only in some conditions (e.g. connection addresses that are sometimes the same and sometimes not depending on how your service gets distributed). - Collectors that are not really safe scraping concurrently with the rest of the program; our test suites include race detectors. - Metrics that don't follow idiomatic naming conventions (can be found via e.g. `promtool check metrics`). - Metrics with legitimate difficult-to-notice bugs. We had an underflow counting the size of a connection pool because `Close()`ing one twice was explicitly allowed. > Testing trivial code brings negative value Agreed in the abstract, but if we were good judges of what code was trivial or not, we'd write a lot less bugs in the first place! That being said, this is also a discussion about coverage, not assertions. Even if I'm not actually checking any of those things, those code paths (and logging) are still getting exercised via any decent E2E/functional/behavioral test suite.

jameson 6y ago

> Coverage is more useful for finding chunks of code that aren’t exercised at all by tests. Right on point. High CC% doesn't tell application is well tested but is an indicator of what isn't covered

breck 6y ago

It's a really helpful tool for as you say telling "you where to look". One trick I've learned: if you want to forecast where the bugs will popup next month, take a quick look at the code coverage report. #1 source will be the code that doesn't appear at all (because the test build system didn't even include it) #2 will be all those red lines.

1337shadow 6y ago

Another advantage that does not get the love it deserves in my opinion, is how useful coverage becomes when it comes to upgrading a dependency, or even the language version. But your project will have to stick around for a while to appreciate that.

closeparen 6y ago

When “hold people accountable to metrics” is not just your company’s management philosophy but the gospel of its entire surrounding culture, this is not an argument that gets anywhere. It is always better to just cover the damn lines than make your Director explain to the VP why he has worse coverage numbers than a sister org. The same way you should make an expected “no such object” result a 200 status code, so that she does not have to stand up and justify her error rate.

capableweb 6y ago
Note that you're now discussing corporate politics, not just creating software, even though creating software often has to deal with corporate politics. My point is rzimmerman never contextualized this as a "how to navigate the workspace" argument, they are arguing it from a pure technical perspective.

hinkley 6y ago

And that’s why you shouldn’t chase after code coverage percentages. While there are uncovered lines, you have a TODO list that has been generated by the machine, telling you what needs more attention. If you go to that section intending to test it well, you’ll get a lot of lines of test code for a modest increase in coverage. If you’re just trying to boost coverage, you’ll check off the todo without accomplishing much. The only place these two metrics line up at all is when you can keep the number of conditions in each method close to one, and the functions pure. Otherwise you have a bunch of little functions that accumulate state and you’ve just moved the combinatorics where it’s even harder to reason about. I think the most you can say with any certainty is that coverage percentage should not go down when adding code. But staying flat doesn’t prevent things from getting worse.

zlynx 6y ago
Except when handling an error condition from a hard to test system call that was previously ignored. Being forced into stupid tricks like adding five useless log statements so that the CI tool doesn't block your change is ... annoying.

voodootrucker 6y ago

You can hit 100% with zero assertions if you like. Probably not the test you want to have though.

regularfry 6y ago
Zero assertions means that you know there aren't any lurking exceptions triggered by just running the code. That's a good thing to know, but I'd hope like hell that it's not the only thing you want to know.
- joshschreuder 6y ago
  It may not even prove that depending on your mock/fake strategy though.