4 ms·

There is only one good code coverage number and it's "100". If you aim for 100% code coverage then it really helps make your code better as you must test all th

by Tomek_ 16y ago

There is only one good code coverage number and it's "100". If you aim for 100% code coverage then it really helps make your code better as you must test all the parts, not only the easy ones - that forces you to write code that's easier to test, and "code easier to test" equals "better code" (more sane interfaces/API, decoupled classes, etc. etc.).

timtadh 16y ago
However, you shouldn't fall into the fallacy that if you have 100% "code coverage" that you code is really fully tested.
famousactress 16y ago
I disagree. Maybe flamebait, but I think 100% is usually either a silly goal, or a good start. 100% for a complex project is basically impossible to achieve (and assumes every area of the code is as important as the next), and for a lot of code 100% is nothing. Consider a function that divides to arguments and returns the result. I can pass in div(2,2). There, done. 100% coverage. [edit] I forgot to mention what value I do think it has. On teams that I'm on I always say I frankly don't care what the number is, as long as it's going up and not down.
- Tomek_ 16y ago
  Of course 100% code coverage won't make your code better in every possible aspect, nor it guarantee that tests are perfectly written, but at least a) the tough parts of the code won't be skipped during the test b) you will have cleaner, more usable (testable) API.
  - famousactress 16y ago
    I still think it's a bit of a wonked metric. The goal out to be to increase your coverage of possible test cases, not LOC. Since you can't cover every possible test case on any non-trivial project, they must have priority. Priority is probably something like how likely the case is to occur, balanced against the consequences of a failure. Test coverage is a simple indicator, and simple indicators are handy.. but chasing 100% too blindly will cause you to start seeing every line of code as equal, and they're not. [edit] - oh, I meant to mention that both of your points (a & b) make sense. I can see how chasing coverage could help make sure you test parts of your code you've been avoiding because they're hard to test. I just wonder if it's the best way to accomplish that goal.
kscaldef 16y ago
I've seen "100% coverage" specs that managed to run every line of code while only testing trivial, obviously correct, aspects and failing to test any of the actually important behavior. This sort of stuff gives TDD a bad name, but is, in my experience, sadly common. Also, at least in the Ruby community, people seem to only measure line coverage. Without branch and conditional coverage reports, it's possible to run 100% of the lines but still miss a lot of code paths.
alain94040 16y ago
I'll tell you what we do in hardware to get test coverage that is worth way more than 100%: we do fault injection. I have never heard of anyone trying to do the same in software, but it would be interesting. Here's how it works: for each statement in your code, break it, then run your entire test suite. If none of your tests have uncovered a problem, mark the code as uncovered. Repeat for each code statement. It's crazily expensive to compute, but it's so much more accurate than to say that a line of code was hit or not. It measures whether the code had a functional impact on the overall behavior.
- aarongough 16y ago
  That's generally referred to as 'mutation testing'. There are a number of systems that enable this in dynamic languages, the one I am most familiar with is Heckle: https://github.com/ryansobol/heckle https://github.com/ryansobol/heckle Mutation testing is great, it highlights all the areas where your tests aren't written well, and teaches you to write better tests. I'm not sure that using it all the time is the best use of resources in most cases though...

by Tomek_ 16y ago

timtadh 16y ago

However, you shouldn't fall into the fallacy that if you have 100% "code coverage" that you code is really fully tested.

famousactress 16y ago

I disagree. Maybe flamebait, but I think 100% is usually either a silly goal, or a good start. 100% for a complex project is basically impossible to achieve (and assumes every area of the code is as important as the next), and for a lot of code 100% is nothing. Consider a function that divides to arguments and returns the result. I can pass in div(2,2). There, done. 100% coverage. [edit] I forgot to mention what value I do think it has. On teams that I'm on I always say I frankly don't care what the number is, as long as it's going up and not down.

Tomek_ 16y ago
Of course 100% code coverage won't make your code better in every possible aspect, nor it guarantee that tests are perfectly written, but at least a) the tough parts of the code won't be skipped during the test b) you will have cleaner, more usable (testable) API.
- famousactress 16y ago
  I still think it's a bit of a wonked metric. The goal out to be to increase your coverage of possible test cases, not LOC. Since you can't cover every possible test case on any non-trivial project, they must have priority. Priority is probably something like how likely the case is to occur, balanced against the consequences of a failure. Test coverage is a simple indicator, and simple indicators are handy.. but chasing 100% too blindly will cause you to start seeing every line of code as equal, and they're not. [edit] - oh, I meant to mention that both of your points (a & b) make sense. I can see how chasing coverage could help make sure you test parts of your code you've been avoiding because they're hard to test. I just wonder if it's the best way to accomplish that goal.

kscaldef 16y ago

I've seen "100% coverage" specs that managed to run every line of code while only testing trivial, obviously correct, aspects and failing to test any of the actually important behavior. This sort of stuff gives TDD a bad name, but is, in my experience, sadly common. Also, at least in the Ruby community, people seem to only measure line coverage. Without branch and conditional coverage reports, it's possible to run 100% of the lines but still miss a lot of code paths.

alain94040 16y ago

I'll tell you what we do in hardware to get test coverage that is worth way more than 100%: we do fault injection. I have never heard of anyone trying to do the same in software, but it would be interesting. Here's how it works: for each statement in your code, break it, then run your entire test suite. If none of your tests have uncovered a problem, mark the code as uncovered. Repeat for each code statement. It's crazily expensive to compute, but it's so much more accurate than to say that a line of code was hit or not. It measures whether the code had a functional impact on the overall behavior.

aarongough 16y ago
That's generally referred to as 'mutation testing'. There are a number of systems that enable this in dynamic languages, the one I am most familiar with is Heckle: https://github.com/ryansobol/heckle https://github.com/ryansobol/heckle Mutation testing is great, it highlights all the areas where your tests aren't written well, and teaches you to write better tests. I'm not sure that using it all the time is the best use of resources in most cases though...