32 ms·

The benefit of 100% test coverage is that there are no more discussions on what to test and what not. When in doubt, test it. In larger groups of developers the

The benefit of 100% test coverage is that there are no more discussions on what to test and what not. When in doubt, test it. In larger groups of developers there are otherwise ongoing discussions if A needs to be tested or not. I have seen culture wars around this, from people who don't want to test and are in the eyes of others always testing not enough and vice versa. Especially with a diverse development force with different ages, seniority and cultural background. It's often easier to just aim for 100% test coverage instead (with excluding some categories of files). EDIT: I would not and did not start with 100% unit testing. But if there are ongoing culture wars and discussions didn't lead to a workable compromise, 100% test coverage worked for me and after some days test coverage was a non issue.

shawnz 6y ago
Even with 100% coverage, that doesn't mean you've found every bug that can possibly be found with unit tests. Your tests could always cover more inputs, more situations, etc.
- KingOfCoders 6y ago
  No, you can't find bugs that you can't think of. Property based testing and random values in unit tests find lots of bugs you didn't think of though.
  - brlewis 6y ago
    > you can't find bugs that you can't think of Rather, you won't find bugs that you choose not to think of because you've let the "100%" number lull you into complacency, even though you know it's 100% of lines/branches, not 100% of inputs.
    - KingOfCoders 6y ago
      Yes. My problem in 40y of programming is still making bugs and those I make come from not thinking about edge cases or from wrong assumptions and not from being lulled into writing tests to meet a 100% number. But personalities differ and if being lulled into security by writing towards a 100% number is a problem for you I would be careful, I totally agree here.
      - monksy 6y ago
        I agree that fuzz testing+linting can help you. However from my persective lower level tests help you to build trust about the software you're releasing.
rimliu 6y ago
This is dangerously close to not understaning what you are doing. _Thinking_ what to test has a benefit of making you think, not just cargo-culting.
- KingOfCoders 6y ago
  Obviously you need to think about how to test. Usually this leads to finding omissions in your code adding edge cases.
tonyedgecombe 6y ago
Rule by diktat is a recipe for demotivated staff.
- KingOfCoders 6y ago
  Ongoing culture wars are a recipe for demotivated staff. Perhaps I am wrong, and I would not start with a 'diktat' for obvious reasons. As a manager, you didn't have discussions about the level of necessary code coverage? Would be interested on how you managed unit testing without 'dictat'. How it would fit into integration testing and explorative testing. What level did developers in your deparment usually find "adequat" ? If you considered it too low, how did you raise test coverage as a manager without defining a coverage level?
redleader9345 6y ago
"larger group of developers" is the phrase that caught my attention. Humans don't scale well. This is where microservices do become attractive. This service is owned by a small team, and that team makes these types of judgements. It may be very different from how other service is owned and maintained, and that's ok.
- KingOfCoders 6y ago
  From my limited experience you get cross team discussions about unit testing, especially if one microservice has too many bugs in the eyes of other teams giving development a bad reputation or making working with a microservice hard. Especially if it breaks with releases and other teams get paged.
  - DodgyEggplant 6y ago
    Yes. you move the people don't scale well issues to multiple teams don't scale well
sanderjd 6y ago
I like thinking about the trade off between a simpler rule that is mostly right vs. decisions require judgement and consensus. I think the simpler rule is usually the better side of the trade-off. But in this case I think the cure might be worse than the disease. Tests for plumbing code often end up being brittle tests of methods getting called on mocks in the right order. People will notice that these require a lot of toil to keep them running as code changes while providing very little benefit in avoiding mistakes. People will rankle at being told that they must write these tests, which they can see are a waste of time. I've done it both ways. I'm much happier with my work when I'm not trying to write tests that are tedious and don't seem to provide any value, in order to hit an arbitrary coverage metric. I suspect my teammates feel the same way, so on teams where I have input into the decision on this, I do not advocate for 100% coverage. It does make it harder to have the discussion of which tests should and shouldn't be written, but I think it's worth that cost.
- KingOfCoders 6y ago
  Yes I have seen bad unit tests very often. Writing good testing code is harder than writing business code. Especially junior developers struggle with this, most often because many companies write not enough tests to learn writing good tests. And if you're in an environment, where this is a non-issue I think thats great. Don't fix something that doesn't need to be fixed.
ben509 6y ago
Yup. The debate over unit testing, being political, is a far bigger impediment to progress than the actual tests, which are a technical hurdle. It's essentially the same reason for linters and auto-formatters. It has a side benefit that it forces devs to write testable code, which inclines them to reasonbly factored code.
tams 6y ago
100% test coverage is pretty easy to game as long as the metric is known. Just exercise all paths and write as few assertions as you can. I'd prefer <100% coverage plus discussions about what to test (and how) much more than working with a test suite built on the wrong incentives.
- KingOfCoders 6y ago
  If you are someone who games metrics for his benefit or hire people who are gaming metrics for their benefit, I assume yes, this metric is very easy to game as are most metrics. Metric systems are not cheater proof.
WrtCdEvrydy 6y ago
> just aim for 100% test coverage instead (with excluding some categories of files). That's where the 'gaming' comes in. The tests start just going through lines without hitting a single expect statement. The ignore files start becoming battlegrounds in the PRs because people just exclude half the damn project. We just have a simple rule... if you wrote code, you have to write coverage for it. If it breaks and your test doesn't catch the breakage, the bug fix goes back to you. Some people will ask "but what about what I'm working on now", you'll have to communicate that you feel your previous work was far more important.
- deleuze 6y ago
  > the bug fix goes back to you. Some people will ask "but what about what I'm working on now", you'll have to communicate that you feel your previous work was far more important. this feels punitive, especially in the eyes of management. unless you're in a safety critical area where fully testing every code path is a hard requirement, people will eventually write bugs. i'd rather work somewhere that recognizes defects occur and has a fast iterative process to push out new changes rather than one based on shame for having written a bug.
  - WrtCdEvrydy 6y ago
    > has a fast iterative process to push out new changes rather than one based on shame for having written a bug. That is the fastest most iterative process we have found so far... as the expert on the original code, you are able to deliver the best outcome. You're not being shamed for writing a bug, you're being shamed for not testing your code.
misja111 6y ago
This argument, that it avoids discussions, does make sense but only if you're in a team where discussions tend to be a waste of time. When my colleagues are knowledgable and open minded I would embrace every opportunity to have a good discussion.
chucky 6y ago
> It's often easier to just aim for 100% test coverage instead (with excluding some categories of files). Congratulations, now you have a war over which categories of files are excluded from the "100% test coverage" rule. ;)
- KingOfCoders 6y ago
  Thank you!

shawnz 6y ago

Even with 100% coverage, that doesn't mean you've found every bug that can possibly be found with unit tests. Your tests could always cover more inputs, more situations, etc.

KingOfCoders 6y ago
No, you can't find bugs that you can't think of. Property based testing and random values in unit tests find lots of bugs you didn't think of though.
- brlewis 6y ago
  > you can't find bugs that you can't think of Rather, you won't find bugs that you choose not to think of because you've let the "100%" number lull you into complacency, even though you know it's 100% of lines/branches, not 100% of inputs.
  - KingOfCoders 6y ago
    Yes. My problem in 40y of programming is still making bugs and those I make come from not thinking about edge cases or from wrong assumptions and not from being lulled into writing tests to meet a 100% number. But personalities differ and if being lulled into security by writing towards a 100% number is a problem for you I would be careful, I totally agree here.
    - monksy 6y ago
      I agree that fuzz testing+linting can help you. However from my persective lower level tests help you to build trust about the software you're releasing.

rimliu 6y ago

This is dangerously close to not understaning what you are doing. _Thinking_ what to test has a benefit of making you think, not just cargo-culting.

KingOfCoders 6y ago
Obviously you need to think about how to test. Usually this leads to finding omissions in your code adding edge cases.

tonyedgecombe 6y ago

Rule by diktat is a recipe for demotivated staff.

KingOfCoders 6y ago
Ongoing culture wars are a recipe for demotivated staff. Perhaps I am wrong, and I would not start with a 'diktat' for obvious reasons. As a manager, you didn't have discussions about the level of necessary code coverage? Would be interested on how you managed unit testing without 'dictat'. How it would fit into integration testing and explorative testing. What level did developers in your deparment usually find "adequat" ? If you considered it too low, how did you raise test coverage as a manager without defining a coverage level?

redleader9345 6y ago

"larger group of developers" is the phrase that caught my attention. Humans don't scale well. This is where microservices do become attractive. This service is owned by a small team, and that team makes these types of judgements. It may be very different from how other service is owned and maintained, and that's ok.

KingOfCoders 6y ago
From my limited experience you get cross team discussions about unit testing, especially if one microservice has too many bugs in the eyes of other teams giving development a bad reputation or making working with a microservice hard. Especially if it breaks with releases and other teams get paged.
- DodgyEggplant 6y ago
  Yes. you move the people don't scale well issues to multiple teams don't scale well

sanderjd 6y ago

I like thinking about the trade off between a simpler rule that is mostly right vs. decisions require judgement and consensus. I think the simpler rule is usually the better side of the trade-off. But in this case I think the cure might be worse than the disease. Tests for plumbing code often end up being brittle tests of methods getting called on mocks in the right order. People will notice that these require a lot of toil to keep them running as code changes while providing very little benefit in avoiding mistakes. People will rankle at being told that they must write these tests, which they can see are a waste of time. I've done it both ways. I'm much happier with my work when I'm not trying to write tests that are tedious and don't seem to provide any value, in order to hit an arbitrary coverage metric. I suspect my teammates feel the same way, so on teams where I have input into the decision on this, I do not advocate for 100% coverage. It does make it harder to have the discussion of which tests should and shouldn't be written, but I think it's worth that cost.

KingOfCoders 6y ago
Yes I have seen bad unit tests very often. Writing good testing code is harder than writing business code. Especially junior developers struggle with this, most often because many companies write not enough tests to learn writing good tests. And if you're in an environment, where this is a non-issue I think thats great. Don't fix something that doesn't need to be fixed.

ben509 6y ago

Yup. The debate over unit testing, being political, is a far bigger impediment to progress than the actual tests, which are a technical hurdle. It's essentially the same reason for linters and auto-formatters. It has a side benefit that it forces devs to write testable code, which inclines them to reasonbly factored code.

tams 6y ago

100% test coverage is pretty easy to game as long as the metric is known. Just exercise all paths and write as few assertions as you can. I'd prefer <100% coverage plus discussions about what to test (and how) much more than working with a test suite built on the wrong incentives.

KingOfCoders 6y ago
If you are someone who games metrics for his benefit or hire people who are gaming metrics for their benefit, I assume yes, this metric is very easy to game as are most metrics. Metric systems are not cheater proof.

WrtCdEvrydy 6y ago

> just aim for 100% test coverage instead (with excluding some categories of files). That's where the 'gaming' comes in. The tests start just going through lines without hitting a single expect statement. The ignore files start becoming battlegrounds in the PRs because people just exclude half the damn project. We just have a simple rule... if you wrote code, you have to write coverage for it. If it breaks and your test doesn't catch the breakage, the bug fix goes back to you. Some people will ask "but what about what I'm working on now", you'll have to communicate that you feel your previous work was far more important.

deleuze 6y ago
> the bug fix goes back to you. Some people will ask "but what about what I'm working on now", you'll have to communicate that you feel your previous work was far more important. this feels punitive, especially in the eyes of management. unless you're in a safety critical area where fully testing every code path is a hard requirement, people will eventually write bugs. i'd rather work somewhere that recognizes defects occur and has a fast iterative process to push out new changes rather than one based on shame for having written a bug.
- WrtCdEvrydy 6y ago
  > has a fast iterative process to push out new changes rather than one based on shame for having written a bug. That is the fastest most iterative process we have found so far... as the expert on the original code, you are able to deliver the best outcome. You're not being shamed for writing a bug, you're being shamed for not testing your code.

misja111 6y ago

This argument, that it avoids discussions, does make sense but only if you're in a team where discussions tend to be a waste of time. When my colleagues are knowledgable and open minded I would embrace every opportunity to have a good discussion.

chucky 6y ago

> It's often easier to just aim for 100% test coverage instead (with excluding some categories of files). Congratulations, now you have a war over which categories of files are excluded from the "100% test coverage" rule. ;)

KingOfCoders 6y ago
Thank you!