5 ms·

> Knowing that software is well tested is even more difficult. One widely accepted measure is that of branch coverage, the ratio of basic code blocks that were

by rob74 3y ago

> Knowing that software is well tested is even more difficult. One widely accepted measure is that of branch coverage, the ratio of basic code blocks that were exercised by some test, to the total number of code blocks in the system under test Widely accepted, but still surprisingly easy to game if you want to do it (and have lazy or no code review). Code coverage only tells you that some test has called the code, not that it has actually checked the results it gets. So, if you write a unit test that "exercises" all the various branches of a method and simply ignores (or performs only minimal checks on) the return values, you get 100% code coverage and, as a bonus, a test that's unlikely to ever break! This might still be useful to make sure that the code doesn't throw any exceptions etc., but it's contrary to the spirit of how unit tests should be written...

lucumo 3y ago
Mutation testing can help with that. With mutation testing the program is changed automatically in small ways to see if that makes a test fail. https://en.wikipedia.org/wiki/Mutation_testing https://en.wikipedia.org/wiki/Mutation_testing That said, any metric is only useful for people actually using it to improve something, and completely useless when used in an adversarial way against someone willing to game the system. Coverage is pretty useful for a programmer to see where test gaps are, and easily gamed when it's felt to be too onerous.
kwhitefoot 3y ago
> a test that's unlikely to ever break! In a large code base I'm pretty sure you would end up throwing exceptions in lots of places.
abricq 3y ago
Totally agree that code coverage is not a sufficient metric for quality, but it's still an interesting one. At my company we have this language that is not mainstream (Oberon) and therefore there was no code-coverage reporting at all. I found this paper and realized it was super easy to write a minimal code-coverage analysis from our AST parser and code formatter (internal tools). Somehow it had never occured to me that it was this simple. The transition from "no code coverage at all" to "seing which module is covered and which is not" was really insightful !
- pjmlp 3y ago
  As someone that worked in projects with 100% guidelines for nice QA dashboards, eventually everyone games the system in large project, unit tests are written to cover all execution flows, not necessarly being actual tests. Thus every code refactoring becomes a pain, as the tests aren't really testing the expectations of a function call, rather ensuring every line of code is shown as being covered.
vrighter 3y ago
bool check_index_in_range(const std::vector<int>&v, size_t index) { // BUG ON NEXT LINE: SHOULD BE < NOT <= if(index <= v.size()) { // BRANCH A std::cout << "access was safe" << std::endl; return true; } else { // BRANCH B std::cout << "access was unsafe" << std::endl; return false; } } void main() { auto v = std::vector<int>(); v.push_back(1); v.push_back(2); v.push_back(3); // Assertion passes, prints that the access is safe. // Exercises branch A assert(check_index_in_range(v,1)); // Assertion passes, prints that the access is unsafe // Exercises branch B assert(!check_index_in_range(v,5)); // We now have 100% code coverage // but still failed to catch a trivial bug. // Even though the buggy line was actually tested twice. // Code coverage is not a good way to measure testing }
- abricq 3y ago
  Of course code coverage should not be seen as the only way to measure testing. But should it be systematically disregarded ? I rather see it as one of the tool for QAs to know where they stand at : 5% code coverage vs 85% is still an interesting information to have.
  - vrighter 3y ago
    100% coverage just gives you a false sense of security. I'd rather just avoid the metric completely
    - Veserv 3y ago
      Why does it give a false sense of security? It is not like >99% coverage is the finish line; it is the starting line, the bare minimum. That is like saying you get a false sense of security when a handgun does not pierce your newly developed tank armor.
      - pjmlp 3y ago
        Because being green on the QA dashboard doesn't mean the unit test makes any sense.

by rob74 3y ago

lucumo 3y ago

Mutation testing can help with that. With mutation testing the program is changed automatically in small ways to see if that makes a test fail. https://en.wikipedia.org/wiki/Mutation_testing https://en.wikipedia.org/wiki/Mutation_testing That said, any metric is only useful for people actually using it to improve something, and completely useless when used in an adversarial way against someone willing to game the system. Coverage is pretty useful for a programmer to see where test gaps are, and easily gamed when it's felt to be too onerous.

kwhitefoot 3y ago

> a test that's unlikely to ever break! In a large code base I'm pretty sure you would end up throwing exceptions in lots of places.

abricq 3y ago

Totally agree that code coverage is not a sufficient metric for quality, but it's still an interesting one. At my company we have this language that is not mainstream (Oberon) and therefore there was no code-coverage reporting at all. I found this paper and realized it was super easy to write a minimal code-coverage analysis from our AST parser and code formatter (internal tools). Somehow it had never occured to me that it was this simple. The transition from "no code coverage at all" to "seing which module is covered and which is not" was really insightful !

pjmlp 3y ago
As someone that worked in projects with 100% guidelines for nice QA dashboards, eventually everyone games the system in large project, unit tests are written to cover all execution flows, not necessarly being actual tests. Thus every code refactoring becomes a pain, as the tests aren't really testing the expectations of a function call, rather ensuring every line of code is shown as being covered.

vrighter 3y ago

bool check_index_in_range(const std::vector<int>&v, size_t index) { // BUG ON NEXT LINE: SHOULD BE < NOT <= if(index <= v.size()) { // BRANCH A std::cout << "access was safe" << std::endl; return true; } else { // BRANCH B std::cout << "access was unsafe" << std::endl; return false; } } void main() { auto v = std::vector<int>(); v.push_back(1); v.push_back(2); v.push_back(3); // Assertion passes, prints that the access is safe. // Exercises branch A assert(check_index_in_range(v,1)); // Assertion passes, prints that the access is unsafe // Exercises branch B assert(!check_index_in_range(v,5)); // We now have 100% code coverage // but still failed to catch a trivial bug. // Even though the buggy line was actually tested twice. // Code coverage is not a good way to measure testing }

abricq 3y ago
Of course code coverage should not be seen as the only way to measure testing. But should it be systematically disregarded ? I rather see it as one of the tool for QAs to know where they stand at : 5% code coverage vs 85% is still an interesting information to have.
- vrighter 3y ago
  100% coverage just gives you a false sense of security. I'd rather just avoid the metric completely
  - Veserv 3y ago
    Why does it give a false sense of security? It is not like >99% coverage is the finish line; it is the starting line, the bare minimum. That is like saying you get a false sense of security when a handgun does not pierce your newly developed tank armor.
    - pjmlp 3y ago
      Because being green on the QA dashboard doesn't mean the unit test makes any sense.