3 ms·

> Anyone who thought critically about it for ten minutes knows it’s nonsense Can you explain a bit more? I have only done coverage on my home projects. I get 1

> Anyone who thought critically about it for ten minutes knows it’s nonsense Can you explain a bit more? I have only done coverage on my home projects. I get 100% every time because otherwise why bother. I started experimenting with 100% branch coverage. I understand sometimes there's a few lines you can't test like if there's a fork() but in my cases I was lucky enough to not need it and didn't need to exclude anything (I was thinking about excluding an impossible to execute default case since I did val&7 and tested all 8 cases but I ended up using if statements instead for that one off case) It seems to me as long as your testing your own code its great. It gets annoying when testing an API that you can't ask to fail. I never mock anything. It seems like coverage is a great solution In one project I used fuzzing data and another I specified valid date and say anything invalid is not a bug and may report incorrect results (I think I detected most/all bad data and rejected it which people hated)

shoo 5y ago
> Can you explain a bit more? Not the parent commenter, but i'll attempt to chime in: in addition to the default "dopey" practice of measuring coverage in terms of statements of code, in addition to data coverage as mentioned by the parent, you could also consider coverage in terms of possible execution paths through the code or coverage in terms of requirements. For interesting computational code (i dunno, consider a mixed integer program solver) it might be possible to get 100% statement or 100% branch coverage at the same time as getting approximately 0% data coverage, 0% execution path coverage, and 0% coverage of functional requirements. Another perspective is how much value the test gives relative to the effort and cost to set it up and maintain it. You can have 100% unit test coverage (statement, branch, data, whatever) but maybe your application doesn't even do anything when it boots, because you forgot to call any of the units. So if effort and time is limited, you might get more return on investment from writing some good integration tests or hooking up a fuzzer than focusing on hitting the somewhat arbitrary but easy to measure 100% statement coverage number. Also, like everything in life, focusing on a single objective like "statement coverage" and trying to maximise it can lead to strange outcomes. There was a fascinating blog post / war story somewhere describing a software system architecture where the organisation mandated non-negotiable minimum 80% statement coverage for unit test suites. This system had some pretty low level code and a bunch of that code wasn't possible to test in unit tests. So how do you hit the target? Well, if you add a few useless layers to your application architecture that are easy to unit test, then you test them and leave the hard code that interfaces with the real world / hardware untested, and you hit your mandated target, but the overall level of software quality is worse.
- satisfice 5y ago
  well said
satisfice 5y ago
Do you know what states are? Of course you do. So then you know that executing a line of code in one state could cause behavior that won’t occur in another state. Code coverage doesn’t say anything about state coverage. You’ve heard of table lookups? Of course you have. So then you know that different results will come from a lookup that is based on a different query or hash key. The lookup is one line of code. But maybe you visit the code hundreds of times before you hit upon the result that gives you bad data which tanks the rest of your algorithm. Come on, man. You know what I mean. Another example: I once wrote a table comparison routine in ObjectPal. I tested it and it seemed to work, but the first time I compared a table with embedded bitmaps I got “Object of type bitmap cannot be compared with <> operator” See what I mean?

shoo 5y ago

> Can you explain a bit more? Not the parent commenter, but i'll attempt to chime in: in addition to the default "dopey" practice of measuring coverage in terms of statements of code, in addition to data coverage as mentioned by the parent, you could also consider coverage in terms of possible execution paths through the code or coverage in terms of requirements. For interesting computational code (i dunno, consider a mixed integer program solver) it might be possible to get 100% statement or 100% branch coverage at the same time as getting approximately 0% data coverage, 0% execution path coverage, and 0% coverage of functional requirements. Another perspective is how much value the test gives relative to the effort and cost to set it up and maintain it. You can have 100% unit test coverage (statement, branch, data, whatever) but maybe your application doesn't even do anything when it boots, because you forgot to call any of the units. So if effort and time is limited, you might get more return on investment from writing some good integration tests or hooking up a fuzzer than focusing on hitting the somewhat arbitrary but easy to measure 100% statement coverage number. Also, like everything in life, focusing on a single objective like "statement coverage" and trying to maximise it can lead to strange outcomes. There was a fascinating blog post / war story somewhere describing a software system architecture where the organisation mandated non-negotiable minimum 80% statement coverage for unit test suites. This system had some pretty low level code and a bunch of that code wasn't possible to test in unit tests. So how do you hit the target? Well, if you add a few useless layers to your application architecture that are easy to unit test, then you test them and leave the hard code that interfaces with the real world / hardware untested, and you hit your mandated target, but the overall level of software quality is worse.

satisfice 5y ago
well said

satisfice 5y ago

Do you know what states are? Of course you do. So then you know that executing a line of code in one state could cause behavior that won’t occur in another state. Code coverage doesn’t say anything about state coverage. You’ve heard of table lookups? Of course you have. So then you know that different results will come from a lookup that is based on a different query or hash key. The lookup is one line of code. But maybe you visit the code hundreds of times before you hit upon the result that gives you bad data which tanks the rest of your algorithm. Come on, man. You know what I mean. Another example: I once wrote a table comparison routine in ObjectPal. I tested it and it seemed to work, but the first time I compared a table with embedded bitmaps I got “Object of type bitmap cannot be compared with <> operator” See what I mean?