4 ms·

In other words, more tests do find more bugs, but it's the number of tests and not their code coverage that has most of the predictive value. It's a surprising

In other words, more tests do find more bugs, but it's the number of tests and not their code coverage that has most of the predictive value. It's a surprising result, so if you'll excuse me, I have a couple of lecture slides on software testing I need to revise Is it just me or was this _not_ surprising at all? I mean I suppose I should have expected what he said, given it sometimes seems hard to convince other people about this but to me it's a well known fact. There are so many ways this can go wrong. I mean it's so easy to give the one counter example needed to break the myth of 100% test coverage being good for much: Well you executed the branch/line at least once with one potential input. Was it an edge case input or a happy path input? More tests than is needed for "100% coverage" means that you actually executed some lines multiple times, hopefully with not just 10 happy path scenarios but with 1 happy path and 9 edge cases. Now remove the happy path scenarios for trivial code and also the edge case scenarios for trivial code and your coverage might only be 80% but you have the same actual test suite effectiveness. When you keep adding tests, add more to the 9 edge cases, staying with the same coverage but make the suite more robust. Of course 20% is better than 0% and 50% is better than 20%. Somewhere between 50 and 100 is an optimal point. For lack of a good estimate, let's go with the old 80/20 rule. Something around 80% test coverage is what you can use to make a tool fail the build. If you pass the 80% mark you _might_ have a good test suite but it doesn't mean you stop there. It's just a reminder that you might have missed adding tests when that thing triggers a red build but don't use it to mean that you actually have a good test suite.

void_mint 5y ago
> Is it just me or was this _not_ surprising at all? It wasn't surprising to anyone that has reflected about the value of tests. Mindless testing/TDD isn't usually reflective, though.
- fiddlerwoaroof 5y ago
  It’s hard to do test-first TDD “mindlessly” because writing a test usually forces you to think in terms of the specification of the behavior you’re about to implement.
  - void_mint 5y ago
    > It’s hard to do test-first TDD “mindlessly” because writing a test usually forces you to think in terms of the specification of the behavior you’re about to implement. Tests are code. Code can be sloppy, fallible, useless. Writing a consumer before you write a provider doesn't make either more robust, just the point at which they meet more clear. You can certainly write a test that uses a function, but the test doesn't actually test anything at all, just that the function exists. aka "mindless TDD".
    - tharkun__ 5y ago
      Very well said. I have nothing against TDD if that's something that helps a particular person write good code and good tests. I find though that people that write good tests are just people that write good tests. Most people write tests that assert on implementation details instead of inputs and output and that's easily doable via TDD as well.
      - fiddlerwoaroof 5y ago
        It’s hard to assert based on implementation details if you don’t have an implementation yet: the advantage of test-first code is that you usually have to write the tests in terms of the interfaces of the input and output types rather than inspecting the code under test.
        tharkun__ 5y ago
        That's because you know how to write good tests but TDD didn't force you to do that. Simply put, TDD just means you write the assertion first, see it red, then change the code to make it green. You can do that by specifying a failing test via input/output testing. You can also very easily write a failing test that tests that the YXZ() method is gonna be called on some internal thing that you're replacing with a mock. Add that method and voila, green test, TDD happy. Bad test though.
      - void_mint 5y ago
        Yes, I agree with this. Process exists to increase the bottom, but if you're trying to rely on TDD to get valuable tests, you're probably just thinking of the problem incorrectly (I think). In my experience, the absolute best way to get valuable tests is to have someone whose job it is to own them, their correctness, and add to them when new issues arise. An SDET, or a rotating team member, or whatever. The average dev isn't going to write a test that's super valuable if they also wrote the code.
spmurrayzzz 5y ago
Like most others here in the comments, I find the results not particularly controversial. Pretty easy to imagine scenarios where lines of code are run, but the tests themselves are suboptimal or incomplete. As an aside— I've often mused at to whether theres a more useful multivariate (but still coarse) measure that could combine coverage with cyclomatic complexity and number of tests/assertions. Seems like it would be incrementally better than just coverage alone.
thaumasiotes 5y ago
> Well you executed the branch/line at least once with one potential input. Was it an edge case input or a happy path input? How does that matter? If something about the input causes a difference in the execution of the code, then 100% coverage means you necessarily tests both kinds of input. You can't reach the edge case branch with the happy path input. Now, if your code is just pumping data from one point to another point, it's possible that the destination will vary its behavior based on the input in a way that your coverage-based testing can't see. But if you're doing something with the input yourself, then 100% coverage means you tested every possible kind of input. (Actually, a "branching" problem can still arise if you have something like if( condition1 ) { do_stuff(); } if( condition2 ) { do_other_stuff(); } if( condition3 ) { do_other_other_stuff(); } Because with this code, it's possible to get test coverage of every line of code (there are 6 of them, not counting the '}'s) while not covering every branch (there are 8 of them, depending on the combined truth values of condition1, condition2, and condition3).)
- spankalee 5y ago
  That's because you have such simple conditions. How do you compute them? if (x < y) { ... } really changes things. Did you test for x === y? for NaN? for very close doubles that should have passed/failed for business reasons? For types other than numbers if your language allows? Expressions can have any number of edge cases that code coverage can't account for.
  - Supermancho 5y ago
    Depends on the language/type safety offered. If your language has a NaN (javascript) and you can assert the expected behavior of x < NaN (false) or if you don't even want to encounter that, then you're able to reason back to the generation of the inputs. Wherever x and y came from, if there was a possibility of an NaN being a value, you want the generating function to exception or cast via some sort of validation before handing them off to other unsuspecting functions.
  - thaumasiotes 5y ago
    Well, imagine two different analyses of the code: 1. Does each branch do the right thing? 2. Does the code go down the correct branch in the first place? I've focused more on the first question, and you're focusing more on the second. But it's fair to say that each question is a reasonable focus of testing, which weakens my comment above.
datavirtue 5y ago
100% or you suck. You need 100% and then some for FULL coverage. I work with people who turn in 80% PRs...these people are bug factories.
wonnage 5y ago
If (as the study says) # of tests is the true correlate then 1. We need some way to compare # of tests across differently-sized codebases, i.e a percentage metric like coverage 2. The fact that correlation disappeared when controlling for absolute # of tests seems to suggest it's still encouraging people to write more tests. Once your org gets to a certain size you need these blunt instruments like minimum coverage requirements because you can't just rely on individual excellence anymore.

void_mint 5y ago

> Is it just me or was this _not_ surprising at all? It wasn't surprising to anyone that has reflected about the value of tests. Mindless testing/TDD isn't usually reflective, though.

fiddlerwoaroof 5y ago
It’s hard to do test-first TDD “mindlessly” because writing a test usually forces you to think in terms of the specification of the behavior you’re about to implement.
- void_mint 5y ago
  > It’s hard to do test-first TDD “mindlessly” because writing a test usually forces you to think in terms of the specification of the behavior you’re about to implement. Tests are code. Code can be sloppy, fallible, useless. Writing a consumer before you write a provider doesn't make either more robust, just the point at which they meet more clear. You can certainly write a test that uses a function, but the test doesn't actually test anything at all, just that the function exists. aka "mindless TDD".
  - tharkun__ 5y ago
    Very well said. I have nothing against TDD if that's something that helps a particular person write good code and good tests. I find though that people that write good tests are just people that write good tests. Most people write tests that assert on implementation details instead of inputs and output and that's easily doable via TDD as well.
    - fiddlerwoaroof 5y ago
      It’s hard to assert based on implementation details if you don’t have an implementation yet: the advantage of test-first code is that you usually have to write the tests in terms of the interfaces of the input and output types rather than inspecting the code under test.
      - tharkun__ 5y ago
        That's because you know how to write good tests but TDD didn't force you to do that. Simply put, TDD just means you write the assertion first, see it red, then change the code to make it green. You can do that by specifying a failing test via input/output testing. You can also very easily write a failing test that tests that the YXZ() method is gonna be called on some internal thing that you're replacing with a mock. Add that method and voila, green test, TDD happy. Bad test though.
    - void_mint 5y ago
      Yes, I agree with this. Process exists to increase the bottom, but if you're trying to rely on TDD to get valuable tests, you're probably just thinking of the problem incorrectly (I think). In my experience, the absolute best way to get valuable tests is to have someone whose job it is to own them, their correctness, and add to them when new issues arise. An SDET, or a rotating team member, or whatever. The average dev isn't going to write a test that's super valuable if they also wrote the code.

spmurrayzzz 5y ago

Like most others here in the comments, I find the results not particularly controversial. Pretty easy to imagine scenarios where lines of code are run, but the tests themselves are suboptimal or incomplete. As an aside— I've often mused at to whether theres a more useful multivariate (but still coarse) measure that could combine coverage with cyclomatic complexity and number of tests/assertions. Seems like it would be incrementally better than just coverage alone.

thaumasiotes 5y ago

> Well you executed the branch/line at least once with one potential input. Was it an edge case input or a happy path input? How does that matter? If something about the input causes a difference in the execution of the code, then 100% coverage means you necessarily tests both kinds of input. You can't reach the edge case branch with the happy path input. Now, if your code is just pumping data from one point to another point, it's possible that the destination will vary its behavior based on the input in a way that your coverage-based testing can't see. But if you're doing something with the input yourself, then 100% coverage means you tested every possible kind of input. (Actually, a "branching" problem can still arise if you have something like if( condition1 ) { do_stuff(); } if( condition2 ) { do_other_stuff(); } if( condition3 ) { do_other_other_stuff(); } Because with this code, it's possible to get test coverage of every line of code (there are 6 of them, not counting the '}'s) while not covering every branch (there are 8 of them, depending on the combined truth values of condition1, condition2, and condition3).)

spankalee 5y ago
That's because you have such simple conditions. How do you compute them? if (x < y) { ... } really changes things. Did you test for x === y? for NaN? for very close doubles that should have passed/failed for business reasons? For types other than numbers if your language allows? Expressions can have any number of edge cases that code coverage can't account for.
- Supermancho 5y ago
  Depends on the language/type safety offered. If your language has a NaN (javascript) and you can assert the expected behavior of x < NaN (false) or if you don't even want to encounter that, then you're able to reason back to the generation of the inputs. Wherever x and y came from, if there was a possibility of an NaN being a value, you want the generating function to exception or cast via some sort of validation before handing them off to other unsuspecting functions.
- thaumasiotes 5y ago
  Well, imagine two different analyses of the code: 1. Does each branch do the right thing? 2. Does the code go down the correct branch in the first place? I've focused more on the first question, and you're focusing more on the second. But it's fair to say that each question is a reasonable focus of testing, which weakens my comment above.

datavirtue 5y ago

100% or you suck. You need 100% and then some for FULL coverage. I work with people who turn in 80% PRs...these people are bug factories.

wonnage 5y ago

If (as the study says) # of tests is the true correlate then 1. We need some way to compare # of tests across differently-sized codebases, i.e a percentage metric like coverage 2. The fact that correlation disappeared when controlling for absolute # of tests seems to suggest it's still encouraging people to write more tests. Once your org gets to a certain size you need these blunt instruments like minimum coverage requirements because you can't just rely on individual excellence anymore.