2 ms·

I can't count how many products I've seen with 100% code coverage that objectively, quantifiably doesn't work. That’s because code coverage doesn’t find the bu

I can't count how many products I've seen with 100% code coverage that objectively, quantifiably doesn't work. That’s because code coverage doesn’t find the bugs that result from code you didn’t write, but should have. Code coverage is but one measure, and to treat it as the measure is folly. (But, yes, I have heard a test manager at a large software company we’ve all heard of declare that test team was done because 100% coverage.)

thaumasiotes 3y ago
It's more because code coverage isn't measured correctly. What you want is branch coverage. If you have something like this: if condition1: do_something1() if condition2: do_something2() if condition3: do_something3() There are 8 possible paths for the code to follow here, but you can cover 100% of lines of code in one test. If your code coverage measure tells you that testing the case where conditions 1, 2, and 3 are all true achieves 100% coverage, your measure is worthless. Covering this code cannot be done in less than eight tests.
- alternatex 3y ago
  I think most code coverage analysis tools are smart enough to consider flow control statements. At least when it comes to C# ones.
  - fho 3y ago
    They do, but that's not the point GP made. An example how this could fail is if the first branch sets up something that branch two uses. Two tests are written that call branch 1&2 and not call them. 100% code coverage, even the non-happy path was tested. But in reality, if those branches have any interaction, you would need to write eight test cases for every combination of branches being run and not being run.
    - thaumasiotes 3y ago
      Well, again, eight is a lower bound. If the software uses any values beyond the bare minimum (here) of three separate booleans, you're likely to need more. Take this simplified code: if condition1: do_something() You need one test for code coverage and two for branch coverage, but if there's a bug in the code as written you'll find that you actually need more. Assume that we want to run do_something() when, and only when, condition2 is true. Now we have four cases: - condition1 && condition2: do_something() runs, which is correct. - condition1 && !condition2: do_something() runs, which is incorrect. - !condition1 && !condition2: do_something() does not run, which is correct. - !condition1 && condition2: do_something() does not run, which is incorrect. If you happened to write tests for the first and third of those options, your tests will make it look like your code works, and your (official) coverage metrics will look comprehensive. But because your coverage metrics (in reality) are actually terrible, your code is secretly not working. The big problem we've just run into is that the number of test cases we need depends on the number of bugs in our code. If the goal of writing tests is to prove that we don't have bugs, this is a terrible result; it means our goal is fundamentally impossible to achieve. We don't know whether we have bugs, so we don't know how many tests we need.
  - thaumasiotes 3y ago
    Why do you think that? It isn't true.
- mikestew 3y ago
  No, it's more because when the bug was filed, you said to yourself, "oh, was I supposed to update the $HOOZIWIZIT in that transaction, too? Oops." Code coverage won't save you from that. Matching test cases to requirements, or summat, can be helpful, but good golly it's a bunch more paperwork. Your point is a good one, but not where I was going.

thaumasiotes 3y ago

It's more because code coverage isn't measured correctly. What you want is branch coverage. If you have something like this: if condition1: do_something1() if condition2: do_something2() if condition3: do_something3() There are 8 possible paths for the code to follow here, but you can cover 100% of lines of code in one test. If your code coverage measure tells you that testing the case where conditions 1, 2, and 3 are all true achieves 100% coverage, your measure is worthless. Covering this code cannot be done in less than eight tests.

alternatex 3y ago
I think most code coverage analysis tools are smart enough to consider flow control statements. At least when it comes to C# ones.
- fho 3y ago
  They do, but that's not the point GP made. An example how this could fail is if the first branch sets up something that branch two uses. Two tests are written that call branch 1&2 and not call them. 100% code coverage, even the non-happy path was tested. But in reality, if those branches have any interaction, you would need to write eight test cases for every combination of branches being run and not being run.
  - thaumasiotes 3y ago
    Well, again, eight is a lower bound. If the software uses any values beyond the bare minimum (here) of three separate booleans, you're likely to need more. Take this simplified code: if condition1: do_something() You need one test for code coverage and two for branch coverage, but if there's a bug in the code as written you'll find that you actually need more. Assume that we want to run do_something() when, and only when, condition2 is true. Now we have four cases: - condition1 && condition2: do_something() runs, which is correct. - condition1 && !condition2: do_something() runs, which is incorrect. - !condition1 && !condition2: do_something() does not run, which is correct. - !condition1 && condition2: do_something() does not run, which is incorrect. If you happened to write tests for the first and third of those options, your tests will make it look like your code works, and your (official) coverage metrics will look comprehensive. But because your coverage metrics (in reality) are actually terrible, your code is secretly not working. The big problem we've just run into is that the number of test cases we need depends on the number of bugs in our code. If the goal of writing tests is to prove that we don't have bugs, this is a terrible result; it means our goal is fundamentally impossible to achieve. We don't know whether we have bugs, so we don't know how many tests we need.
- thaumasiotes 3y ago
  Why do you think that? It isn't true.
mikestew 3y ago
No, it's more because when the bug was filed, you said to yourself, "oh, was I supposed to update the $HOOZIWIZIT in that transaction, too? Oops." Code coverage won't save you from that. Matching test cases to requirements, or summat, can be helpful, but good golly it's a bunch more paperwork. Your point is a good one, but not where I was going.