5 ms·

Ever since I developed code coverage tools at Apple in 1989, and tested them for Borland in the early 90’s, I knew and have been telling people in MY conference

Ever since I developed code coverage tools at Apple in 1989, and tested them for Borland in the early 90’s, I knew and have been telling people in MY conference slides that code coverage is a nearly useless metric. Anyone who thought critically about it for ten minutes knows it’s nonsense. The one thing code coverage tells you that is of any significant value is what you haven’t tested. You still know very little about what you have tested. Meanwhile, this article continues the dopey practice of using the word “coverage” only with respect to code and not with respect to many other forms of coverage, such as data. Data coverage is not easy to measure, yet it is far more important. Generally speaking, academics know very little about testing, and academic papers about testing are nearly useless. If you want to learn about testing from an academic, read sociology, philosophy, and cognitive science.

amw-zero 5y ago
Yet it's extremely popular to have huge, inefficient test suites running on continuous integration servers hundreds of times a day. It boggles the mind.
- datavirtue 5y ago
  This is a good thing if you consider the price of not having them. I have worked in shops without tests. It's a great way to hand out free money to unsuspecting customers.
  - amw-zero 5y ago
    Yes I agree - I have been a testing fanatic for the better part of the last 10 years, after being absolutely paralyzed at a company without tests. But, after all this time, I believe their cost-to benefit-ratio is horrendous. It’s fairly common to hear of test suites with a 2:1 ratio of test to implementation lines. That would be fine if they didn’t immensely prevent refactoring and block merges / deployments. Contrast that with something like F*, where the specification and code are right alongside each other, and the implementation gets proven. They are reporting a 5:1 ratio of specification to implementation code lines, which is much better than only a few years ago. Soon I think we’ll be striking distance to largely get rid of huge test suites which are so commonplace today.
    - exdsq 5y ago
      The one thing is a spec doesn't mean the product is correct - you still need testing, just in a different way. It'll probably replace unit tests though.
      - logicchains 5y ago
        A spec can catch anything your tests can. And if your spec overlooks something, your tests will overlook it too.
        exdsq 5y ago
        I've seen projects using formal methods including mathematical specifications miss things caught in end-to-end tests written by domain experts rather than formal method engineers. Perhaps you're right in theory but I've yet to see it in practice.
        amw-zero 5y ago
        This just means that you thought of something in the test suite that wasn't in the spec. You can underspecify something.
      - amw-zero 5y ago
        Correct, there’s still nothing stopping us from writing the wrong spec or writing an insufficient spec, but there’s really no solution to that problem anyway. And the same problem exists with tests today - you can have an insufficient amount of test cases, or you can write the wrong cases because you misunderstand the requirements. The difference being, when you realize your tests are wrong, what do you do? Change and add more test cases. You may also need to remove some, because they don’t make sense anymore. Vs., updating logic in a specification, and that’s it.
    - satisfice 5y ago
      I’m not sure what your point is about ratios. I don’t think they mean anything, strictly speaking. Perhaps they are related to some special knowledge you have that is not conveyed by the numbers. Testing is a social act. Testing itself cannot be obsolete until complexity and misunderstanding is obsolete. This will happen never. Perhaps certain kinds of shallow automated checking will become obsolete over time.
dboreham 5y ago
Possibly the most valuable post I've read on these news pages.
datavirtue 5y ago
Damn, I love geeking out over testing.
ternaryoperator 5y ago
> academic papers about testing are nearly useless. You must not have read the post--it's principally summary of a paper by two academics at the Univ. of Waterloo.
- satisfice 5y ago
  I did read the post, and I am referring to the paper by the academics. What did you think I was talking about? Hello is this mike on?
ParkerPierre 5y ago
> Anyone who thought critically about it for ten minutes knows it’s nonsense Can you explain a bit more? I have only done coverage on my home projects. I get 100% every time because otherwise why bother. I started experimenting with 100% branch coverage. I understand sometimes there's a few lines you can't test like if there's a fork() but in my cases I was lucky enough to not need it and didn't need to exclude anything (I was thinking about excluding an impossible to execute default case since I did val&7 and tested all 8 cases but I ended up using if statements instead for that one off case) It seems to me as long as your testing your own code its great. It gets annoying when testing an API that you can't ask to fail. I never mock anything. It seems like coverage is a great solution In one project I used fuzzing data and another I specified valid date and say anything invalid is not a bug and may report incorrect results (I think I detected most/all bad data and rejected it which people hated)
- shoo 5y ago
  > Can you explain a bit more? Not the parent commenter, but i'll attempt to chime in: in addition to the default "dopey" practice of measuring coverage in terms of statements of code, in addition to data coverage as mentioned by the parent, you could also consider coverage in terms of possible execution paths through the code or coverage in terms of requirements. For interesting computational code (i dunno, consider a mixed integer program solver) it might be possible to get 100% statement or 100% branch coverage at the same time as getting approximately 0% data coverage, 0% execution path coverage, and 0% coverage of functional requirements. Another perspective is how much value the test gives relative to the effort and cost to set it up and maintain it. You can have 100% unit test coverage (statement, branch, data, whatever) but maybe your application doesn't even do anything when it boots, because you forgot to call any of the units. So if effort and time is limited, you might get more return on investment from writing some good integration tests or hooking up a fuzzer than focusing on hitting the somewhat arbitrary but easy to measure 100% statement coverage number. Also, like everything in life, focusing on a single objective like "statement coverage" and trying to maximise it can lead to strange outcomes. There was a fascinating blog post / war story somewhere describing a software system architecture where the organisation mandated non-negotiable minimum 80% statement coverage for unit test suites. This system had some pretty low level code and a bunch of that code wasn't possible to test in unit tests. So how do you hit the target? Well, if you add a few useless layers to your application architecture that are easy to unit test, then you test them and leave the hard code that interfaces with the real world / hardware untested, and you hit your mandated target, but the overall level of software quality is worse.

amw-zero 5y ago

Yet it's extremely popular to have huge, inefficient test suites running on continuous integration servers hundreds of times a day. It boggles the mind.

datavirtue 5y ago
This is a good thing if you consider the price of not having them. I have worked in shops without tests. It's a great way to hand out free money to unsuspecting customers.
- amw-zero 5y ago
  Yes I agree - I have been a testing fanatic for the better part of the last 10 years, after being absolutely paralyzed at a company without tests. But, after all this time, I believe their cost-to benefit-ratio is horrendous. It’s fairly common to hear of test suites with a 2:1 ratio of test to implementation lines. That would be fine if they didn’t immensely prevent refactoring and block merges / deployments. Contrast that with something like F*, where the specification and code are right alongside each other, and the implementation gets proven. They are reporting a 5:1 ratio of specification to implementation code lines, which is much better than only a few years ago. Soon I think we’ll be striking distance to largely get rid of huge test suites which are so commonplace today.
  - exdsq 5y ago
    The one thing is a spec doesn't mean the product is correct - you still need testing, just in a different way. It'll probably replace unit tests though.
    - logicchains 5y ago
      A spec can catch anything your tests can. And if your spec overlooks something, your tests will overlook it too.
      - exdsq 5y ago
        I've seen projects using formal methods including mathematical specifications miss things caught in end-to-end tests written by domain experts rather than formal method engineers. Perhaps you're right in theory but I've yet to see it in practice.
        amw-zero 5y ago
        This just means that you thought of something in the test suite that wasn't in the spec. You can underspecify something.
    - amw-zero 5y ago
      Correct, there’s still nothing stopping us from writing the wrong spec or writing an insufficient spec, but there’s really no solution to that problem anyway. And the same problem exists with tests today - you can have an insufficient amount of test cases, or you can write the wrong cases because you misunderstand the requirements. The difference being, when you realize your tests are wrong, what do you do? Change and add more test cases. You may also need to remove some, because they don’t make sense anymore. Vs., updating logic in a specification, and that’s it.
  - satisfice 5y ago
    I’m not sure what your point is about ratios. I don’t think they mean anything, strictly speaking. Perhaps they are related to some special knowledge you have that is not conveyed by the numbers. Testing is a social act. Testing itself cannot be obsolete until complexity and misunderstanding is obsolete. This will happen never. Perhaps certain kinds of shallow automated checking will become obsolete over time.

dboreham 5y ago

Possibly the most valuable post I've read on these news pages.

datavirtue 5y ago

Damn, I love geeking out over testing.

ternaryoperator 5y ago

> academic papers about testing are nearly useless. You must not have read the post--it's principally summary of a paper by two academics at the Univ. of Waterloo.

satisfice 5y ago
I did read the post, and I am referring to the paper by the academics. What did you think I was talking about? Hello is this mike on?

ParkerPierre 5y ago

> Anyone who thought critically about it for ten minutes knows it’s nonsense Can you explain a bit more? I have only done coverage on my home projects. I get 100% every time because otherwise why bother. I started experimenting with 100% branch coverage. I understand sometimes there's a few lines you can't test like if there's a fork() but in my cases I was lucky enough to not need it and didn't need to exclude anything (I was thinking about excluding an impossible to execute default case since I did val&7 and tested all 8 cases but I ended up using if statements instead for that one off case) It seems to me as long as your testing your own code its great. It gets annoying when testing an API that you can't ask to fail. I never mock anything. It seems like coverage is a great solution In one project I used fuzzing data and another I specified valid date and say anything invalid is not a bug and may report incorrect results (I think I detected most/all bad data and rejected it which people hated)

shoo 5y ago
> Can you explain a bit more? Not the parent commenter, but i'll attempt to chime in: in addition to the default "dopey" practice of measuring coverage in terms of statements of code, in addition to data coverage as mentioned by the parent, you could also consider coverage in terms of possible execution paths through the code or coverage in terms of requirements. For interesting computational code (i dunno, consider a mixed integer program solver) it might be possible to get 100% statement or 100% branch coverage at the same time as getting approximately 0% data coverage, 0% execution path coverage, and 0% coverage of functional requirements. Another perspective is how much value the test gives relative to the effort and cost to set it up and maintain it. You can have 100% unit test coverage (statement, branch, data, whatever) but maybe your application doesn't even do anything when it boots, because you forgot to call any of the units. So if effort and time is limited, you might get more return on investment from writing some good integration tests or hooking up a fuzzer than focusing on hitting the somewhat arbitrary but easy to measure 100% statement coverage number. Also, like everything in life, focusing on a single objective like "statement coverage" and trying to maximise it can lead to strange outcomes. There was a fascinating blog post / war story somewhere describing a software system architecture where the organisation mandated non-negotiable minimum 80% statement coverage for unit test suites. This system had some pretty low level code and a bunch of that code wasn't possible to test in unit tests. So how do you hit the target? Well, if you add a few useless layers to your application architecture that are easy to unit test, then you test them and leave the hard code that interfaces with the real world / hardware untested, and you hit your mandated target, but the overall level of software quality is worse.