3 ms·
> One widely accepted measure is that of branch coverage, the ratio of basic code blocks that were exercised by some test, to the total number of code blocks in
by bluGill 3y ago
> One widely accepted measure is that of branch coverage, the ratio of basic code blocks that were exercised by some test, to the total number of code blocks in the system under test
I know that statement is true, but WHY is it true? I have never seen anyone look at a coverage report and take any useful action. Instead I see management looking at those reports for a number to rate people on, but no useful action is ever taken. When I've looked at the reports I keep finding trivial code that would be hard to test for little gain.
I eliminated the report and nobody misses it. Instead we report to management total number of tests run (test cases, and each assert). Which is easy to game, but since the number naturally increases fast isn't worth the bother.
- taeric 3y agoIt is no different than any other hard measurement. What is the best heart rate for you, as an example? In many, it is more a directional check. 100% is likely not to give you 100% confidence. But, 0% will make everyone less confident. And if you have two teams, one at 50% and one at 80%, it is probably safe to assume that the testing culture is better for the 80% team. And why branch coverage, as opposed to path coverage? Largely because you hit the easy to measure things before you worry about the hard to measure ones.
- stcredzero 3y agoI know that statement is true, but WHY is it true? I have never seen anyone look at a coverage report and take any useful action. We made code coverage a part of CI, and we've been using that to drive up code coverage. It's been working. Our tests have gone from ~25% to ~50% coverage. Breakages making it out to staging have gone down. Also, some incentives are well aligned with this. If people see that things aren't easy to test, then they get rearranged to make them easy to test. Not all incentives are aligned, but not everything is perfect. This is why reviews are useful. When I've looked at the reports I keep finding trivial code that would be hard to test for little gain. If your trivial code is also hard to test, then this indicates architectural code debt. Why isn't the trivial code trivial to test?
- bluGill 3y agoHow do you test the code that calls a OS function? (think of something like shutdown - you don't want your test to turn off your CI system) The obvious answer is write a wrapper and a mock implementation of the wrapper. Now you have the wrapper that is untested, so you write a wrapper for that so you - I actually had someone go down this 3 levels before giving up. I work in embedded where we have to control real hardware.
- nickpsecurity 3y agoThe other thing you can do is composition of individually-verified parts. In your example, you make tests that really exercise the shutdown function. Also, note any preconditions these functions need to meet so the application calling it can be consistent with specs. From there, just assume shutdown works if used within the tested spec. This method also encourages reusing working specs, code, and tests. We see that in DO-178C with the RTOS’s, etc.