4 ms·

You know, I understand and agree that code coverage isn't the be-all and end-all of testing. But... we do all secretly agree that if it isn't pretty high, the

You know, I understand and agree that code coverage isn't the be-all and end-all of testing. But... we do all secretly agree that if it isn't pretty high, then the testing isn't doing much, right? Like, does anyone feel really good about saying there's good test coverage of a class with 50% code coverage? 60%? After I hit 80-90% coverage, I start wanting to know about other attributes of the tests. Are they really really really testing that all the outputs are exactly what we expect? Are they controlling dependencies in a useful, realistic way? Are they exploring a wide range of inputs? Before that, though, all of the above seem a bit premature. I don't deeply care if you've done a really thorough job of testing 20% of the code if the other 80% isn't even run.

derefr 12y ago
I would agree generally (because most code has exposure to the consumer even when it shouldn't), but, in the specific instance that 1. the code is effectively formed into a library/component/service that you only interact with through a well-defined public API, and 2. the tests test every exposed function of the API, then you don't need to test anything other than that API—100% code coverage (covering code-paths the API can't reach) is meaningless. Any code that the tests don't test in such a scenario is actually code that the library doesn't use; it's dead weight, and should be culled (and actually, given a whole-program optimization pass, it would be.)
sbov 12y ago
Like most things, the answer is... it depends. We have several with 0% and I'm just fine with that. They're exceedingly simple. And/or they're 15 years old and only change in very slight ways every few years. There's other classes that are 100% that I'm not fine with. Because while the code is short, the implications are complex. I feel like blindly talking about these metrics is like saying "hey, I bought an earring for $100, did I get ripped off?"
- aetherson 12y ago
  I think that you're broadening the question. Sure, sometimes you feel okay with code that's not at all tested. But surely you don't feel that code that is not at all tested is well tested. I'm talking about to what extent code coverage can and can not be used as a proxy for "well tested," not to what extent "well tested" can and can not be used as a proxy for "I feel good about my code."
Silhouette 12y ago
But... we do all secretly agree that if it isn't pretty high, then the testing isn't doing much, right? No, sorry, not even close. To me, measuring the value of a test suite by code coverage is like measuring the productivity of developers by lines of code. It's just not what actually matters. For example, it's well known that bugs in code tend to cluster. Given a large system where some parts are inherently complex and therefore error-prone but most of the code is routine boilerplate stuff, I would much prefer to see test effort concentrated on the challenging areas.
- collyw 12y ago
  If you write 100% code coverage, you will probably have a lot more lines of code.
mikekchar 12y ago
As others have said, code coverage is an extremely misleading statistic. Most tools measure code coverage by the number of lines of code that were executed in a test. I can easily write "tests" with no expectations that hit every line of code. It is merely showing that in the cases the programmer thought of, it doesn't crash. Consider this as well: many interpreted languages count the definition of a method/function as a "covered line of code". In other words, if a method/class is defined then you have at least one (maybe several) lines of code "covered". Add to that things like declaring variables and you can easily get to a point where (in good code with very short methods) you have 60% coverage if the method is simply defined and run. If you are writing really good OO code that has very few conditionals, then you may get very, very close to 100% test coverage simply by running each method. Also, tests are code too. You are just as likely to write a bug in your test as you are in your production code. In fact, I've noticed that in some systems, I actually end up writing more lines of test code than production code. I might actually be more likely to write a bug in the tests. Especially for bugs that are due to problems with requirements, you are very, very likely to put the same bug in the tests as you are the production code. This is why I don't like calling these things "tests". IMHO the very best "tests" document assumptions that the original programmer had. If you break the assumption, the the test should break. It may or may not be a bug, but it is a big hint that you should be thinking very carefully about what you are doing. Code coverage is a good statistic in the reverse. It doesn't tell you much if you have good code coverage. On the other hand, it can lead you to find places where your tests are unlikely to be sufficient. Depending on the language/environment you use, if you have less than 80% code coverage in an area, it might indicate that nothing of value is being tested. It is a good place to start looking to check. Note that setting a code coverage target will defeat this strategy, so I recommend never, ever doing it. As I said, getting nearly 100% code coverage without actually testing anything is relatively trivial if the code is in any way decent (and if it isn't, then you have bigger fish to fry). Setting a threshold above the place where poor testing is obvious just means that you no longer have a means of finding places that are poorly tested.
- aetherson 12y ago
  I think that you think you're arguing with me, but are actually agreeing with me. Like, I agree with every claim about fact that you're making. Of course you can have high code coverage without having good tests. If all you do is, like, touch lines of code, all you're really testing is that they don't raise an exception. That's why I said that once you hit 80% code coverage or so, I start to want to know not "what your code coverage is," but "how good are your tests?" But if you don't even run the majority of your code, then your tests are kind of by definition... not testing your code well.
coldtea 12y ago
>Like, does anyone feel really good about saying there's good test coverage of a class with 50% code coverage? 60%? Depends. 20% might be enough too. You can just test the 20% harder and more failure dependent paths in your app, not the 80% trivial BS code. Remember the 80/20 rule?
- aetherson 12y ago
  I do remember it! I feel like having bugs in my code that is only run 20% of the time is not an acceptable amount of bugs. Like, seriously guys, I'm sure we can all come up with some kind of degenerate case where 80% of the code is in exception-handling or some other really, really, really rarely called piece of code, and feel okay with not testing that. But that's hardly the ordinary scenario. And, for example, in the messaging code I've recently been writing, it certainly is the case that some of the code is exercised a lot more than others. For example, reads are done an order of magnitude more than writes, and writes are done an order of magnitude more than deletes. But that doesn't make me feel okay about bugs in my delete code. It's a whole feature. It needs to work.
  - coldtea 12y ago
    >I do remember it! I feel like having bugs in my code that is only run 20% of the time is not an acceptable amount of bugs. Then maybe you don't understand the notion of "opportunity cost". Perhaps you feel "0 is the only acceptable amount of bugs", but a) nobody cares for an imaginary bug-free product unless it ships. And shiping, for programs intended for the market, means cutting corners and, yes, shipping bugs too. b) programs that have 0 bugs are so few as to be statistical noise, including among TDD/full coverage programs. TDD will basically help you cover that the behavior you expect is happening and help you track breakage when you need to refactor. It wont eliminated all bugs. >But that doesn't make me feel okay about bugs in my delete code. It's a whole feature. It needs to work. Perhaps you conflated untested with "not working". People created succesful software for ages, software that changed the world, with no or less than 50% "test coverage".
cowardlydragon 12y ago
I thought programs spent 80% of their time in 20% of the code? TDD "code coverage" always seemed like a misguided breadth-first approach that needed to be depth-first primarily and breadth-first as needed. But I think the latter approach then leads to integration testing as the primary approach, so TDD zealots will hate that.
- cam- 12y ago
  Unit testing is a leading indicator. The higher the coverage the better the quality of the product is a good broad rule. Engineers hate unit testing to 100% until they do, and then they don't go back, it becomes the norm for their development process. The confidence and assurance you get from that level of coverage is amazing. It takes the stress out of software.

derefr 12y ago

I would agree generally (because most code has exposure to the consumer even when it shouldn't), but, in the specific instance that 1. the code is effectively formed into a library/component/service that you only interact with through a well-defined public API, and 2. the tests test every exposed function of the API, then you don't need to test anything other than that API—100% code coverage (covering code-paths the API can't reach) is meaningless. Any code that the tests don't test in such a scenario is actually code that the library doesn't use; it's dead weight, and should be culled (and actually, given a whole-program optimization pass, it would be.)

sbov 12y ago

Like most things, the answer is... it depends. We have several with 0% and I'm just fine with that. They're exceedingly simple. And/or they're 15 years old and only change in very slight ways every few years. There's other classes that are 100% that I'm not fine with. Because while the code is short, the implications are complex. I feel like blindly talking about these metrics is like saying "hey, I bought an earring for $100, did I get ripped off?"

aetherson 12y ago
I think that you're broadening the question. Sure, sometimes you feel okay with code that's not at all tested. But surely you don't feel that code that is not at all tested is well tested. I'm talking about to what extent code coverage can and can not be used as a proxy for "well tested," not to what extent "well tested" can and can not be used as a proxy for "I feel good about my code."

Silhouette 12y ago

But... we do all secretly agree that if it isn't pretty high, then the testing isn't doing much, right? No, sorry, not even close. To me, measuring the value of a test suite by code coverage is like measuring the productivity of developers by lines of code. It's just not what actually matters. For example, it's well known that bugs in code tend to cluster. Given a large system where some parts are inherently complex and therefore error-prone but most of the code is routine boilerplate stuff, I would much prefer to see test effort concentrated on the challenging areas.

collyw 12y ago
If you write 100% code coverage, you will probably have a lot more lines of code.

mikekchar 12y ago

As others have said, code coverage is an extremely misleading statistic. Most tools measure code coverage by the number of lines of code that were executed in a test. I can easily write "tests" with no expectations that hit every line of code. It is merely showing that in the cases the programmer thought of, it doesn't crash. Consider this as well: many interpreted languages count the definition of a method/function as a "covered line of code". In other words, if a method/class is defined then you have at least one (maybe several) lines of code "covered". Add to that things like declaring variables and you can easily get to a point where (in good code with very short methods) you have 60% coverage if the method is simply defined and run. If you are writing really good OO code that has very few conditionals, then you may get very, very close to 100% test coverage simply by running each method. Also, tests are code too. You are just as likely to write a bug in your test as you are in your production code. In fact, I've noticed that in some systems, I actually end up writing more lines of test code than production code. I might actually be more likely to write a bug in the tests. Especially for bugs that are due to problems with requirements, you are very, very likely to put the same bug in the tests as you are the production code. This is why I don't like calling these things "tests". IMHO the very best "tests" document assumptions that the original programmer had. If you break the assumption, the the test should break. It may or may not be a bug, but it is a big hint that you should be thinking very carefully about what you are doing. Code coverage is a good statistic in the reverse. It doesn't tell you much if you have good code coverage. On the other hand, it can lead you to find places where your tests are unlikely to be sufficient. Depending on the language/environment you use, if you have less than 80% code coverage in an area, it might indicate that nothing of value is being tested. It is a good place to start looking to check. Note that setting a code coverage target will defeat this strategy, so I recommend never, ever doing it. As I said, getting nearly 100% code coverage without actually testing anything is relatively trivial if the code is in any way decent (and if it isn't, then you have bigger fish to fry). Setting a threshold above the place where poor testing is obvious just means that you no longer have a means of finding places that are poorly tested.

aetherson 12y ago
I think that you think you're arguing with me, but are actually agreeing with me. Like, I agree with every claim about fact that you're making. Of course you can have high code coverage without having good tests. If all you do is, like, touch lines of code, all you're really testing is that they don't raise an exception. That's why I said that once you hit 80% code coverage or so, I start to want to know not "what your code coverage is," but "how good are your tests?" But if you don't even run the majority of your code, then your tests are kind of by definition... not testing your code well.

coldtea 12y ago

>Like, does anyone feel really good about saying there's good test coverage of a class with 50% code coverage? 60%? Depends. 20% might be enough too. You can just test the 20% harder and more failure dependent paths in your app, not the 80% trivial BS code. Remember the 80/20 rule?

aetherson 12y ago
I do remember it! I feel like having bugs in my code that is only run 20% of the time is not an acceptable amount of bugs. Like, seriously guys, I'm sure we can all come up with some kind of degenerate case where 80% of the code is in exception-handling or some other really, really, really rarely called piece of code, and feel okay with not testing that. But that's hardly the ordinary scenario. And, for example, in the messaging code I've recently been writing, it certainly is the case that some of the code is exercised a lot more than others. For example, reads are done an order of magnitude more than writes, and writes are done an order of magnitude more than deletes. But that doesn't make me feel okay about bugs in my delete code. It's a whole feature. It needs to work.
- coldtea 12y ago
  >I do remember it! I feel like having bugs in my code that is only run 20% of the time is not an acceptable amount of bugs. Then maybe you don't understand the notion of "opportunity cost". Perhaps you feel "0 is the only acceptable amount of bugs", but a) nobody cares for an imaginary bug-free product unless it ships. And shiping, for programs intended for the market, means cutting corners and, yes, shipping bugs too. b) programs that have 0 bugs are so few as to be statistical noise, including among TDD/full coverage programs. TDD will basically help you cover that the behavior you expect is happening and help you track breakage when you need to refactor. It wont eliminated all bugs. >But that doesn't make me feel okay about bugs in my delete code. It's a whole feature. It needs to work. Perhaps you conflated untested with "not working". People created succesful software for ages, software that changed the world, with no or less than 50% "test coverage".

cowardlydragon 12y ago

I thought programs spent 80% of their time in 20% of the code? TDD "code coverage" always seemed like a misguided breadth-first approach that needed to be depth-first primarily and breadth-first as needed. But I think the latter approach then leads to integration testing as the primary approach, so TDD zealots will hate that.

cam- 12y ago
Unit testing is a leading indicator. The higher the coverage the better the quality of the product is a good broad rule. Engineers hate unit testing to 100% until they do, and then they don't go back, it becomes the norm for their development process. The confidence and assurance you get from that level of coverage is amazing. It takes the stress out of software.