6 ms·

> I’ve heard managers and teams mandating 100% code coverage for applications. That’s a really bad idea I hope many managers and programmers out there don't ta

> I’ve heard managers and teams mandating 100% code coverage for applications. That’s a really bad idea I hope many managers and programmers out there don't take this the wrong way. I've been an engineer on a project that was attempting to get 100% code coverage on a piece of software I was writing. I heard constant remarks during this period that were similar to "You don't need 100% code coverage, it doesn't do anything!" These engineers who I was working with had only read articles like this and didn't stop to think about what the article was trying to say. From my experience there is no safe rule of thumb for for how many tests should be implemented for a project. There should be just enough to feel safe (as hathawsh has said). If you're recommending to engineers on your team to stop implementing tests when they say "I'm at 60% coverage and it'll take ~2 days to get to 100%" I'd really hope you take the time to understand why they want 100% coverage. The software I was working on when my coworkers started telling me to stop writing tests was code designed to trigger alarms when patient's vital signs met certain criteria set by doctors. I am very thankful that I did hit 100% coverage because between 60% and 100% there were many small edge cases that, had they caused a death, I wouldn't have been able to sleep well. Had they said I was explicitly disallowed from working on the tests I would have come in on a weekend (or taking a PTO) and implemented them then. It's our ethical responsibility to know when and where paranoia is worth the marginal time penalty.

bulkan 8y ago
> ...trigger alarms when patient's vital signs met certain criteria set by doctor Most of us aren't working on life or death code like this. My React app doesn't need 100% code coverage but you put it well when you said "There should be just enough to feel safe"
- gravypod 8y ago
  What I really want to drive home is that you should listen to the engineers who are working on the project. Testing is one of those things that is entirely subject to project needs. Like you say, a React app doesn't need 100% coverage, neither does code running a CI, something far out of the serving path, etc. But we shouldn't have a knee-jerk reaction to an engineer saying we should have 100% coverage.
jorblumesea 8y ago
Seems like a very specific concern in a very specific line of business. The vast majority of engineers write CRUD apps for business clients that mostly care about whether it completely explodes or not. It also depends on the service. Some services are critical infrastructure and should have their edge cases tested. Some services provide non essential functions and can get away with less. I agree that there is no magic number, but 100% coverage is clearly overkill for the average engineering team. In fact, broadly demanding any code coverage percentage is probably over simplifying the issue. It just shouldn't be 0 :)
Cpoll 8y ago
> between 60% and 100% there were many small edge cases that, had they caused a death, I wouldn't have been able to sleep well. But you're not saying the right words here. 100% doesn't mean you've covered every edge case (although I suppose 60% necessarily means you _haven't_). I can hit 100% without actually asserting anything. I think it's harmful to talk about 100% without also considering mutation testing, boundary analysis, manual mutation testing...
- gravypod 8y ago
  If the code was open source I would show it to you. There are definitely cases that weren't covered during the initial creation of the service that came up in QA/prod. The entire process was a war story in and of itself. The tests aren't perfect but I have done everything in my power to make sure they cover all of the use cases that appear, samples of bad data from production, samples of bad data I forced to happen in other services, asserting failure cases, and (most importantly) are vigilantly kept up to date. I also, on my own initiative, found people internally to help run a small QA study where they interacted with this system on a daily basis for about 2 months. The senior engineers were not supportive of other forms of testing in any way (even the user testing). They flat out refused my proposal to attempt to create an integration test suite to cover it's communication to other services as well as other services consuming it's data (this I started over the weekend and was told to proceed no longer). tl;dr: You're absolutely correct but I still think my testing procedures were better than nothing.
  - kwhitefoot 8y ago
    I feel for you. I argued for unit testing for ten years before they were instituted and even then it was half hearted.
  - kristiandupont 8y ago
    >I still think my testing procedures were better than nothing. I am sure they were, but you also said: >take the time to understand why they want 100% coverage which to me indicates that they are under the false impression that this gives some sort of completeness, i.e. "make sure this plastic fence covers the entire stretch of mountain road". We have a coverage threshold of 90% and it is helpful because it works as a reminder if you try to check in code that isn't covered by tests. I think that pushing that to 100% would add little value but a lot of overhead. Never tried it though so I fully admit that it's just my hunch.
userbinator 8y ago
It's our ethical responsibility to know when and where paranoia is worth the marginal time penalty. Exactly. You are in the tiny minority writing literally life-and-death code, and for which a lot of other common advice given regarding general software development likely does not apply either (like "move fast and break things".) Also, for your type of application I would probably want 100% state space coverage too.
rorykoehler 8y ago
I'm a huge fan of tests that ensure the app doesn't break in a significant way in production. How many tests that is depends on the situation. For your case I would agree that 100% test coverage is correct.
konschubert 8y ago
Not to take away from your point, just to remind people: 100% coverage still does not mean 100% of all cases tested - it just means that every line has been run with SOME data and every conditional statement was run once in both directions.
adrianN 8y ago
Really reliable software does full MC/DC testing, but even that doesn't catch all mistakes. SQLite for example does this but there are still bugs discovered occasionally. It also leads to insane amounts of test code compared to the actual application code. That is much too expensive to maintain for most projects.
danpalmer 8y ago
I had a recent project where: - 0-80% was adding useful tests. - 80-98% was mostly useless. - 98-100% found some really interesting bugs and forced refactoring that made testing easier, and was definitely worth it.
chriswarbo 8y ago
My problem with "100% coverage" is Goodhart's law ( https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law ). Keep adding tests as long as they tell us something new, give us more confidence, document some regression, etc. Keep adding tests if they're exposing edge cases (e.g. MAX_INT), even if those code paths are already covered. Keep adding tests if they're asserting useful things about the results, even if those code paths are already covered. Stop adding tests when you're only trying to make the coverage number increase.
jsight 8y ago
I'd never thought about this before, but you can definitely flip this around to a pathological case of doing the opposite of striving for 100% coverage. The problem with 100% coverage that I think of is that the last 30% is often boilerplate code (generated getters/setters in Java and that kind of thing). But what if an engineer started with the boilerplate and only then progressed to tests that are actually important? You might get to 30% without testing anything useful at all. Then you might test half of the important stuff and hit an arbitrary metric of 70%. If someone were ruled by the sonarqube score, they might even do this. Yeah, thinking about the why matters more than the %.
Too 8y ago
So many problems with "100% coverage". Even if you measure it as branch coverage, it only indicates how much of the code that is already written has been covered. It doesn't tell you how much code that is missing, eg guards against malicious input, null-checks or other out of range arguments. This reason alone should be obvious enough for anyone to understand that the metric is completely bogus. As others mentioned it also doesn't say anything about state space coverage or input argument space coverage or all combinatorial paths through your code. Only that some line/branch has been hit at least once.
AtlasBarfed 8y ago
100% code coverage is a breadth-first metric, and generally sacrifices in depth-first testing (especially of the "core loop" as in 80% of the time is spent in 20% of the code is sacrificed). The decidability/halting problem hints that perfect testing is an impossibility for systems of any complexity, as in not just too big of a big-O... it's flat out not possible on a turing machine of any power. That doesn't mean give up all testing, but there is a LOT of dogma in testing philosophy.
natashas 8y ago
I agree with others - mission critical systems (nuclear plants, aviation, medical software etc) have quite different standard for quality.
mmcnl 8y ago
Excellent comment, thanks for sharing. One ought to really think about why tests are needed and what value they add, and not just simply follow some random (thoughtful) article on the internet.

bulkan 8y ago

> ...trigger alarms when patient's vital signs met certain criteria set by doctor Most of us aren't working on life or death code like this. My React app doesn't need 100% code coverage but you put it well when you said "There should be just enough to feel safe"

gravypod 8y ago
What I really want to drive home is that you should listen to the engineers who are working on the project. Testing is one of those things that is entirely subject to project needs. Like you say, a React app doesn't need 100% coverage, neither does code running a CI, something far out of the serving path, etc. But we shouldn't have a knee-jerk reaction to an engineer saying we should have 100% coverage.

jorblumesea 8y ago

Seems like a very specific concern in a very specific line of business. The vast majority of engineers write CRUD apps for business clients that mostly care about whether it completely explodes or not. It also depends on the service. Some services are critical infrastructure and should have their edge cases tested. Some services provide non essential functions and can get away with less. I agree that there is no magic number, but 100% coverage is clearly overkill for the average engineering team. In fact, broadly demanding any code coverage percentage is probably over simplifying the issue. It just shouldn't be 0 :)

Cpoll 8y ago

> between 60% and 100% there were many small edge cases that, had they caused a death, I wouldn't have been able to sleep well. But you're not saying the right words here. 100% doesn't mean you've covered every edge case (although I suppose 60% necessarily means you _haven't_). I can hit 100% without actually asserting anything. I think it's harmful to talk about 100% without also considering mutation testing, boundary analysis, manual mutation testing...

gravypod 8y ago
If the code was open source I would show it to you. There are definitely cases that weren't covered during the initial creation of the service that came up in QA/prod. The entire process was a war story in and of itself. The tests aren't perfect but I have done everything in my power to make sure they cover all of the use cases that appear, samples of bad data from production, samples of bad data I forced to happen in other services, asserting failure cases, and (most importantly) are vigilantly kept up to date. I also, on my own initiative, found people internally to help run a small QA study where they interacted with this system on a daily basis for about 2 months. The senior engineers were not supportive of other forms of testing in any way (even the user testing). They flat out refused my proposal to attempt to create an integration test suite to cover it's communication to other services as well as other services consuming it's data (this I started over the weekend and was told to proceed no longer). tl;dr: You're absolutely correct but I still think my testing procedures were better than nothing.
- kwhitefoot 8y ago
  I feel for you. I argued for unit testing for ten years before they were instituted and even then it was half hearted.
- kristiandupont 8y ago
  >I still think my testing procedures were better than nothing. I am sure they were, but you also said: >take the time to understand why they want 100% coverage which to me indicates that they are under the false impression that this gives some sort of completeness, i.e. "make sure this plastic fence covers the entire stretch of mountain road". We have a coverage threshold of 90% and it is helpful because it works as a reminder if you try to check in code that isn't covered by tests. I think that pushing that to 100% would add little value but a lot of overhead. Never tried it though so I fully admit that it's just my hunch.

userbinator 8y ago

It's our ethical responsibility to know when and where paranoia is worth the marginal time penalty. Exactly. You are in the tiny minority writing literally life-and-death code, and for which a lot of other common advice given regarding general software development likely does not apply either (like "move fast and break things".) Also, for your type of application I would probably want 100% state space coverage too.

rorykoehler 8y ago

I'm a huge fan of tests that ensure the app doesn't break in a significant way in production. How many tests that is depends on the situation. For your case I would agree that 100% test coverage is correct.

konschubert 8y ago

Not to take away from your point, just to remind people: 100% coverage still does not mean 100% of all cases tested - it just means that every line has been run with SOME data and every conditional statement was run once in both directions.

adrianN 8y ago

Really reliable software does full MC/DC testing, but even that doesn't catch all mistakes. SQLite for example does this but there are still bugs discovered occasionally. It also leads to insane amounts of test code compared to the actual application code. That is much too expensive to maintain for most projects.

danpalmer 8y ago

I had a recent project where: - 0-80% was adding useful tests. - 80-98% was mostly useless. - 98-100% found some really interesting bugs and forced refactoring that made testing easier, and was definitely worth it.

chriswarbo 8y ago

My problem with "100% coverage" is Goodhart's law ( https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law ). Keep adding tests as long as they tell us something new, give us more confidence, document some regression, etc. Keep adding tests if they're exposing edge cases (e.g. MAX_INT), even if those code paths are already covered. Keep adding tests if they're asserting useful things about the results, even if those code paths are already covered. Stop adding tests when you're only trying to make the coverage number increase.

jsight 8y ago

I'd never thought about this before, but you can definitely flip this around to a pathological case of doing the opposite of striving for 100% coverage. The problem with 100% coverage that I think of is that the last 30% is often boilerplate code (generated getters/setters in Java and that kind of thing). But what if an engineer started with the boilerplate and only then progressed to tests that are actually important? You might get to 30% without testing anything useful at all. Then you might test half of the important stuff and hit an arbitrary metric of 70%. If someone were ruled by the sonarqube score, they might even do this. Yeah, thinking about the why matters more than the %.

Too 8y ago

So many problems with "100% coverage". Even if you measure it as branch coverage, it only indicates how much of the code that is already written has been covered. It doesn't tell you how much code that is missing, eg guards against malicious input, null-checks or other out of range arguments. This reason alone should be obvious enough for anyone to understand that the metric is completely bogus. As others mentioned it also doesn't say anything about state space coverage or input argument space coverage or all combinatorial paths through your code. Only that some line/branch has been hit at least once.

AtlasBarfed 8y ago

100% code coverage is a breadth-first metric, and generally sacrifices in depth-first testing (especially of the "core loop" as in 80% of the time is spent in 20% of the code is sacrificed). The decidability/halting problem hints that perfect testing is an impossibility for systems of any complexity, as in not just too big of a big-O... it's flat out not possible on a turing machine of any power. That doesn't mean give up all testing, but there is a LOT of dogma in testing philosophy.

natashas 8y ago

I agree with others - mission critical systems (nuclear plants, aviation, medical software etc) have quite different standard for quality.

mmcnl 8y ago

Excellent comment, thanks for sharing. One ought to really think about why tests are needed and what value they add, and not just simply follow some random (thoughtful) article on the internet.