4 ms·

The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of sec

The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of security. So if I want to contribute to your project all I have to do is write some pointless tests that are sure to execute every single getter and setter method.(yes I have seen tests that exist solely to execute getters and setters). I don't have to actually test any known edge cases.

enraged_camel 9y ago
Exactly. It's like testing a newly built bridge by crossing it a hundred times with a bike. Sure, you "covered" it, but you sure as hell did not test it.
- hashmp 9y ago
  Brilliant analogy.
- ioquatix 9y ago
  If you write unit tests like this, it's your own fault when the bridge collapses from driving a car over it.
ben_jones 9y ago
A good example would be in dynamically typed languages where you can hit a code path but forget to validate a certain input type which could cause a bug. Or one I've been personally hating lately, a nil pointer dereference in Golang from a well tested library that didn't do proper error propagation.
- tyingq 9y ago
  Yep. Intel's AMT bug was exactly that...unexpected input, but in a static typed language. And would not have been discovered with 100% code coverage.
ademarre 9y ago
> a false sense of security Yes. Code coverage should not be the primary measure of quality of your tests. That thinking leads to tests designed to cover lines, not use cases.
defined 9y ago
> The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of security. This is true. But isn't that true of any test? A suite of naive or badly written tests can also give us a false sense of security, so why write any at all (I don't mean that literally)? I think that a greater level of confidence in our code can be achieved by a combination of - Judicious choices of unit and integration tests - Static and dynamic analysis (if the language supports it), and - Property-based testing (the canonical example of which is QuickCheck[1]). Property-based testing is a great way to help us hit those edge cases. As for 100% code coverage, I think it is worth striving for, not just for the sake of having all lines of code tested. It can expose design and testability flaws, for example. If 100% coverage cannot be achieved, we need to ask ourselves: - Did we really need that piece of code we couldn't test? Is it actually called anywhere, or is it one of those YAGNI things? - Did we write code that is not very testable? Are there functions or methods, for example, that are so dependent on external state that they can't be mocked or tested some other way? Should we refactor it? - If it's a trivial line of code like a getter or setter that cannot possibly be wrong, then it should be fairly trivial to generate automatically a test case for it. More severe defects have probably been caused by a single line of untested code[2][3] than we may suspect. - If it's a getter or setter, and it's not used (and therefore code covered) in other test cases, maybe it's superfluous and should be removed. References [1]: https://en.m.wikipedia.org/wiki/QuickCheck https://en.m.wikipedia.org/wiki/QuickCheck [2]: https://www.imperialviolet.org/2014/02/22/applebug.html https://www.imperialviolet.org/2014/02/22/applebug.html [3]: http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse.html http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collaps...
- saltedmd5 9y ago
  > If it's a trivial line of code like a getter or setter that cannot possibly be wrong, then it should be fairly trivial to generate automatically a test case for it. More importantly, if it's a getter or setter, the actual behaviour to be tested is probably broader than "should set/get x." Test scenarios, not methods.
frik 9y ago
Exactly. It's all about logic. They other types of bugs are caught by the IDE, the compiler or the first run anyway. Let someone else look over your code, they might spot logic bugs that you oversee. If you write tests yourself, you will often oversee such logical bugs. I have done (semi-)automatic reasoning and code verification with 100% code coverage of OS drivers. All the mathematical reasoning doesn't spot logical errors. You need more than one person to look over it.
FroshKiller 9y ago
A "pointless" test of a getter or setter could save you a lot of trouble later after someone introduces a side effect.
saltedmd5 9y ago
This is completely true, but it's an argument against using coverage as a metric for test quality, not against test coverage per se.
ioquatix 9y ago
Nope, if you submit a PR with crappy unit tests, I'll throw it politely back in your face and tell you to rewrite them :D

enraged_camel 9y ago

Exactly. It's like testing a newly built bridge by crossing it a hundred times with a bike. Sure, you "covered" it, but you sure as hell did not test it.

hashmp 9y ago
Brilliant analogy.
ioquatix 9y ago
If you write unit tests like this, it's your own fault when the bridge collapses from driving a car over it.

ben_jones 9y ago

A good example would be in dynamically typed languages where you can hit a code path but forget to validate a certain input type which could cause a bug. Or one I've been personally hating lately, a nil pointer dereference in Golang from a well tested library that didn't do proper error propagation.

tyingq 9y ago
Yep. Intel's AMT bug was exactly that...unexpected input, but in a static typed language. And would not have been discovered with 100% code coverage.

ademarre 9y ago

> a false sense of security Yes. Code coverage should not be the primary measure of quality of your tests. That thinking leads to tests designed to cover lines, not use cases.

defined 9y ago

> The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of security. This is true. But isn't that true of any test? A suite of naive or badly written tests can also give us a false sense of security, so why write any at all (I don't mean that literally)? I think that a greater level of confidence in our code can be achieved by a combination of - Judicious choices of unit and integration tests - Static and dynamic analysis (if the language supports it), and - Property-based testing (the canonical example of which is QuickCheck[1]). Property-based testing is a great way to help us hit those edge cases. As for 100% code coverage, I think it is worth striving for, not just for the sake of having all lines of code tested. It can expose design and testability flaws, for example. If 100% coverage cannot be achieved, we need to ask ourselves: - Did we really need that piece of code we couldn't test? Is it actually called anywhere, or is it one of those YAGNI things? - Did we write code that is not very testable? Are there functions or methods, for example, that are so dependent on external state that they can't be mocked or tested some other way? Should we refactor it? - If it's a trivial line of code like a getter or setter that cannot possibly be wrong, then it should be fairly trivial to generate automatically a test case for it. More severe defects have probably been caused by a single line of untested code[2][3] than we may suspect. - If it's a getter or setter, and it's not used (and therefore code covered) in other test cases, maybe it's superfluous and should be removed. References [1]: https://en.m.wikipedia.org/wiki/QuickCheck https://en.m.wikipedia.org/wiki/QuickCheck [2]: https://www.imperialviolet.org/2014/02/22/applebug.html https://www.imperialviolet.org/2014/02/22/applebug.html [3]: http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse.html http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collaps...

saltedmd5 9y ago
> If it's a trivial line of code like a getter or setter that cannot possibly be wrong, then it should be fairly trivial to generate automatically a test case for it. More importantly, if it's a getter or setter, the actual behaviour to be tested is probably broader than "should set/get x." Test scenarios, not methods.

frik 9y ago

Exactly. It's all about logic. They other types of bugs are caught by the IDE, the compiler or the first run anyway. Let someone else look over your code, they might spot logic bugs that you oversee. If you write tests yourself, you will often oversee such logical bugs. I have done (semi-)automatic reasoning and code verification with 100% code coverage of OS drivers. All the mathematical reasoning doesn't spot logical errors. You need more than one person to look over it.

FroshKiller 9y ago

A "pointless" test of a getter or setter could save you a lot of trouble later after someone introduces a side effect.

saltedmd5 9y ago

This is completely true, but it's an argument against using coverage as a metric for test quality, not against test coverage per se.

ioquatix 9y ago

Nope, if you submit a PR with crappy unit tests, I'll throw it politely back in your face and tell you to rewrite them :D