5 ms·

How test coverage is meaningful (or isn't)

27 points by llambda 15y ago

reiz 15y ago
I think testing and code coverage is important, but over evaluated. 1 year ago I worked on a Java Project with over 2500 test cases and over 82% code coverage. The portal was still pretty buggy. 82% code coverage means nothing. To get such a high code coverage we tested really everything. We even wrote unit tests for DTOs, Entities and so on. To have a bunch of tests for simple getters and setters is pretty use less in my opinion :-) I agree with the part, that to write good tests is not easy. That is not a job for students or for bad payed employes with less experience/skills. In my current project I have much less tests. I am just testing the important parts. But that I am testing very carefully. And I have now much less bugs then in my old project.
- jeffskj 15y ago
  The only problem with not writing tests for getters/setters is that is makes it harder to identify actual code that is untested. If code coverage tools were smarter about not counting getters/setters towards the coverage number it wouldn't be a problem but if they aren't tested your coverage report might say you have 25% coverage in a package and you don't know that's because it has a bunch of DTOs in it or if its because there's bunch of actual untested logic in that package. The way we solved that problem on my team was to write a little utility that used reflection to automatically test the basic getter/setter pairs which made writing the tests for DTOs basically one line of code.
  - astral303 15y ago
    Many coverage tools have ways to ignore getters/setters and things like "if (debug logging enabled) log" types of statements.
    - jeffskj 15y ago
      I'm not aware of any open source ones for Java that can do that, Atlassian Clover (not open source) might do it but I'm not sure on that. If you know of one I'd love to take a look at it.
      - reiz 15y ago
        I think the same. And I think getters and setters are useless! Just wasted lines of code. One more reason to switch to a dynamic language like Ruby or Python. The applications I saw in the last 14 months in the Ruby community are much more object oriented than the stuff I saw the last 10 years in the Java World. And because they are more object oriented their are much more easy to test. It is so easy to write test cases for a Ruby on Rails App, but it is a nightmare to do the same for JSF & EJB.
- zanny 15y ago
  I agree with this, that raw test coverage is a useless metric. The real value in tests is when you can specify interfaces that will rarely change, create a test suite for that interface that will rarely need to change, and use it to find any problems in the production code and any refactorings of that code done in the future, and that the test suite will verify correctness of the code no matter what changes. That is the true, sole, value of testing in my book. If having good RADs and UML acts as validation of models and endeavors, good testing verifies the code written to that spec.
pacaro 15y ago
Code coverage can provide two things 1) It can be a signal, a low number raises a red flag, but a high number doesn't raise a green flag. 2) It can be a great analysis tool. When you have partial coverage, digging through the coverage reports and seeing where you don't have coverage can tell you a lot about what isn't being tested, what the test writers assumptions were, etc.
stcredzero 15y ago
The reason, of course, why people focus on coverage numbers is because they want to know if they are testing enough. Bug rates and what they are doing should be the final arbiters. How often do Unit Tests find regressions? How often do bugs escape into the wild? As with many things, what is the return on investment?
yummyfajitas 15y ago
High test coverage is useful in one important regard. If I've got a module and 25 untested __unicode__(self) methods, I might not notice the untested big_complicated_method(self, args, *kwargs) in the coverage report. That's less of a problem if I have 100% test coverage (down to 95% after writing big_complicated_method).
- pbiggar 15y ago
  I think the important thing here is the change: 100 -> 95. A 5% decrease should be noticed, in some fashion. You don't need to be at 100% coverage to do that.
  - yummyfajitas 15y ago
    The change is important, but so is the absolute number of untested lines. A) Untested lines of code: 10-25 B) Untested lines of code: 2-4, 10-25, 54-68, 59, 63, 68, 71-77, 85 I know that in case A, I'm far more likely to look at 10-25 and come up with a good test case. In case B, I'm not sure I want to skim 2-4 (skip, it's ok), 68 (just a __unicode__), 59 (oh, that should never be executed), ah, 10-25 is what I should look at. It's sort of the broken windows theory applied to testing. That said, I don't have 100% test coverage in most of my projects.
    - pbiggar 15y ago
      A tool which told you there was an absolute change of 5%, but didnt tell you where that change was, would not be all that useful. I'm sympathetic to the "all-or-none" broken windows philosophy that many developers use, but I disagree with it. If you need to get to 100% of some metric in order to get some visibility on its changes, you're just not going to get that metric. At Circle, we make hosted CI, and I frequently talk to customers who use jenkins or something, and whose tests are broken. Inevitably, they broke the tests one day, their priorities dictated that they couldn't fix it right away, so they left their test suite red. Because it was red, they didn't notice the suite getting more and more broken over time. If their test suite hadn't been binary, they would have learned immediately when they broke more tests, and they would have been in a much better place. TL;DR I blame the tools.

27 points by llambda 15y ago

reiz 15y ago

I think testing and code coverage is important, but over evaluated. 1 year ago I worked on a Java Project with over 2500 test cases and over 82% code coverage. The portal was still pretty buggy. 82% code coverage means nothing. To get such a high code coverage we tested really everything. We even wrote unit tests for DTOs, Entities and so on. To have a bunch of tests for simple getters and setters is pretty use less in my opinion :-) I agree with the part, that to write good tests is not easy. That is not a job for students or for bad payed employes with less experience/skills. In my current project I have much less tests. I am just testing the important parts. But that I am testing very carefully. And I have now much less bugs then in my old project.

jeffskj 15y ago
The only problem with not writing tests for getters/setters is that is makes it harder to identify actual code that is untested. If code coverage tools were smarter about not counting getters/setters towards the coverage number it wouldn't be a problem but if they aren't tested your coverage report might say you have 25% coverage in a package and you don't know that's because it has a bunch of DTOs in it or if its because there's bunch of actual untested logic in that package. The way we solved that problem on my team was to write a little utility that used reflection to automatically test the basic getter/setter pairs which made writing the tests for DTOs basically one line of code.
- astral303 15y ago
  Many coverage tools have ways to ignore getters/setters and things like "if (debug logging enabled) log" types of statements.
  - jeffskj 15y ago
    I'm not aware of any open source ones for Java that can do that, Atlassian Clover (not open source) might do it but I'm not sure on that. If you know of one I'd love to take a look at it.
    - reiz 15y ago
      I think the same. And I think getters and setters are useless! Just wasted lines of code. One more reason to switch to a dynamic language like Ruby or Python. The applications I saw in the last 14 months in the Ruby community are much more object oriented than the stuff I saw the last 10 years in the Java World. And because they are more object oriented their are much more easy to test. It is so easy to write test cases for a Ruby on Rails App, but it is a nightmare to do the same for JSF & EJB.
zanny 15y ago
I agree with this, that raw test coverage is a useless metric. The real value in tests is when you can specify interfaces that will rarely change, create a test suite for that interface that will rarely need to change, and use it to find any problems in the production code and any refactorings of that code done in the future, and that the test suite will verify correctness of the code no matter what changes. That is the true, sole, value of testing in my book. If having good RADs and UML acts as validation of models and endeavors, good testing verifies the code written to that spec.

pacaro 15y ago

Code coverage can provide two things 1) It can be a signal, a low number raises a red flag, but a high number doesn't raise a green flag. 2) It can be a great analysis tool. When you have partial coverage, digging through the coverage reports and seeing where you don't have coverage can tell you a lot about what isn't being tested, what the test writers assumptions were, etc.

stcredzero 15y ago

The reason, of course, why people focus on coverage numbers is because they want to know if they are testing enough. Bug rates and what they are doing should be the final arbiters. How often do Unit Tests find regressions? How often do bugs escape into the wild? As with many things, what is the return on investment?

yummyfajitas 15y ago

High test coverage is useful in one important regard. If I've got a module and 25 untested __unicode__(self) methods, I might not notice the untested big_complicated_method(self, args, *kwargs) in the coverage report. That's less of a problem if I have 100% test coverage (down to 95% after writing big_complicated_method).

pbiggar 15y ago
I think the important thing here is the change: 100 -> 95. A 5% decrease should be noticed, in some fashion. You don't need to be at 100% coverage to do that.
- yummyfajitas 15y ago
  The change is important, but so is the absolute number of untested lines. A) Untested lines of code: 10-25 B) Untested lines of code: 2-4, 10-25, 54-68, 59, 63, 68, 71-77, 85 I know that in case A, I'm far more likely to look at 10-25 and come up with a good test case. In case B, I'm not sure I want to skim 2-4 (skip, it's ok), 68 (just a __unicode__), 59 (oh, that should never be executed), ah, 10-25 is what I should look at. It's sort of the broken windows theory applied to testing. That said, I don't have 100% test coverage in most of my projects.
  - pbiggar 15y ago
    A tool which told you there was an absolute change of 5%, but didnt tell you where that change was, would not be all that useful. I'm sympathetic to the "all-or-none" broken windows philosophy that many developers use, but I disagree with it. If you need to get to 100% of some metric in order to get some visibility on its changes, you're just not going to get that metric. At Circle, we make hosted CI, and I frequently talk to customers who use jenkins or something, and whose tests are broken. Inevitably, they broke the tests one day, their priorities dictated that they couldn't fix it right away, so they left their test suite red. Because it was red, they didn't notice the suite getting more and more broken over time. If their test suite hadn't been binary, they would have learned immediately when they broke more tests, and they would have been in a much better place. TL;DR I blame the tools.