3 ms·
> The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of s
by defined 9y ago
> The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of security.
This is true.
But isn't that true of any test? A suite of naive or badly written tests can also give us a false sense of security, so why write any at all (I don't mean that literally)?
I think that a greater level of confidence in our code can be achieved by a combination of
- Judicious choices of unit and integration tests
- Static and dynamic analysis (if the language supports it), and
- Property-based testing (the canonical example of which is QuickCheck[1]). Property-based testing is a great way to help us hit those edge cases.
As for 100% code coverage, I think it is worth striving for, not just for the sake of having all lines of code tested. It can expose design and testability flaws, for example.
If 100% coverage cannot be achieved, we need to ask ourselves:
- Did we really need that piece of code we couldn't test? Is it actually called anywhere, or is it one of those YAGNI things?
- Did we write code that is not very testable? Are there functions or methods, for example, that are so dependent on external state that they can't be mocked or tested some other way? Should we refactor it?
- If it's a trivial line of code like a getter or setter that cannot possibly be wrong, then it should be fairly trivial to generate automatically a test case for it. More severe defects have probably been caused by a single line of untested code[2][3] than we may suspect.
- If it's a getter or setter, and it's not used (and therefore code covered) in other test cases, maybe it's superfluous and should be removed.
References
[1]: https://en.m.wikipedia.org/wiki/QuickCheck https://en.m.wikipedia.org/wiki/QuickCheck
[2]: https://www.imperialviolet.org/2014/02/22/applebug.html https://www.imperialviolet.org/2014/02/22/applebug.html
[3]: http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse.html http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collaps...
- saltedmd5 9y ago> If it's a trivial line of code like a getter or setter that cannot possibly be wrong, then it should be fairly trivial to generate automatically a test case for it. More importantly, if it's a getter or setter, the actual behaviour to be tested is probably broader than "should set/get x." Test scenarios, not methods.