10 ms·
The Myth of Code Coverage
- kojiromike 6y agoFirst he says "ideal project", then "1/3 of the code every software project is irrelevant, buggy, overly complicated, or simply sucks". Doesn't seem ideal to me.
- hyperman1 6y agoI was also going to say around 60-70 pct, but for a different reason: what's left in my code is mostly checking of assertions, debug logging and handling rare errors, i.e code that's not supposed to run. This is code that will dump some state to the logs, then crash without causing corruption. It tends to be 'tested' accidentaly when I am making big changes, and I add more of it if I get a crash that's undebuggable from dumps and logs.
- ardy42 6y ago> I was also going to say around 60-70 pct, but for a different reason: what's left in my code is mostly checking of assertions, debug logging and handling rare errors, i.e code that's not supposed to run. I grant you assertion-checking, but code for handling rare errors is not the code you should skip writing unit tests for. If it runs infrequently then you're far less likely to stumble on a regression during other testing, so it's even more important to unit test.
- hyperman1 6y agoDepends on rare errors. I meanthings like 'My database crashed halfway a transaction, my file system drops from under my application, my back end service gave up. You can't do much here. Dump some info, abrt the half- done work, maybe try again somewhere in the future. And yes, that last part migh deserve a test.
- morelisp 6y agoIf you can't do much, it shouldn't be much code, so coverage should stay high. I used to be of the "70% is good enough" school, but I found that as I wrote better programs, I also achieved better test coverage. Not because I was writing more tests, but because I was choosing core designs with fewer edge cases, simplifying my error handling with fewer reachable paths, and admitting I might as well just crash in more cases. So now my line/branch coverage is more like 95-100%, but my code is easier to read and I write _fewer_, mostly functional/behavioral, tests. Most lines I don't reach are fatal errors, and most fatal errors are the only statement in their branch. Code you can't reach from test cases isn't a sign you should write more tests, it's a sign you should remove that code from the program. Today I view 90-95% really as a baseline, and focus more on path coverage (most standard tools are quite bad for this still) and edge cases in data (e.g. denormed floats or different kinds of invalid data I want to make sure stay invalid) that don't affect coverage one way or another.
- hyperman1 6y agoIt adds up surprisingly fast. I live mostly in the java world, and libraries throwing 3 or so required to catch exceptions are common. I'd agree that 90pct is doable for your own algorithmic code. Glue code for libraries, commonly not very well thought out, requires a very different style. Defensive up to the paranoid, and catching every gremlin at the source. It's not uncommon to have more than half of the code dedocated to exception handling. Dumping said libraries is rarely an option, unfortunately. I'm not necesarily disagreeing with you, but maybe we live in different code worlds.
- morelisp 6y agoWe definitely do, and I see how that might apply - I do write some Java but avoid as much as possible the disaster of checked exceptions. Mostly I am talking about Python, Go, TypeScript, and other more esoteric stuff. I don't agree that this defensive style is necessary though - I do (like everyone these days) write a ton of glue code, and that's as I described. Java is nearly as bad as JavaScript for language-cart-pulling-program-horse designs, it's just a different set of mistakes. It could be so much better if developers just did less.
- donatj 6y agoI generally agree with the crux of this and would like to take it a step further. It's far more important to test edge cases and the lesser used endpoints. We have major parts of our product where the coverage isn't great, but get tested _every time_ you use the product. The coverage on our login process is mediocre, but every time a Developer or QA person uses the product, they log in. We'd hear about it. Our landing page post login is largely the same story. Our rarely used tools several clicks deep into the site _all_ have much better coverage. We don't have continuous deployment and it's not a goal.
- rzimmerman 6y agoJust because code is touched during the test process doesn’t necessarily mean it has been tested. Coverage is more useful for finding chunks of code that aren’t exercised at all by tests. Branches that never get hit that probably deserve extra scrutiny. Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code. Likewise, there’s stuff that’s just not worth it to wrap in a test. The effort involved in mocking/wrapping/abstracting sometimes isn’t worth the effort for some code, like simple calls to common libraries, reading a file, CLI help strings, etc. This stuff generally is better served by end-to-end or integration tests. Or just gets exercised naturally as part of development. Or can crash without much consequence. My point is coverage is a tool that tells you where to look, not the one and only metric that tells you how reliable your code is.
- g14i 6y agoTotally agree. In one of our projects (around 200K Python LoC) we had 90% CodCov, but we frequently found some bugs. Recently we started to heavily use mutation testing and fuzzing to find edge cases on parts of the code that were already "covered" according to the CodCov report. It was definitely worth it. I highly recommend investing in mutation and fuzzy testing.
- whymarrh 6y agoI was just about to bring up mutation testing. I've had some pretty great success with PIT [1] when writing Java. Code coverage and mutation test coverage pair wonderfully together. [1]:https://pitest.org
- mathgladiator 6y agoI'm a strident adherent to having 100% code coverage, but this mostly works because my history in infrastructure and a desire to sleep well. The key idea that I have found that if you can't synthetically get your process into a specific state, then life is going to be hard. The problem with a lot of people is that they view testing as a burden rather than a criticism of the code they are testing. If you can't test your code easily, then your code is awful. End of story.
- postalrat 6y agoYou must be trolling.
- breck 6y ago> " If you can't test your code easily, then your code is awful. End of story." I don't think this is trolling. Maybe I would have phrased it in a nicer way, but I agree with their math. Now I don't think you need to get to 100%, but if you can't easily see how you could get to 100%, that's a problem.
- postalrat 6y agoHow easy is it test a function add(a, b) that just adds a and b and returns the result? Or how bout testing: function divide(a, b, fn) { let [_a, _b] = fn(a, b); return _a / _b; }
- kevwil 6y ago"Myth" is a strong word, given the author supports 2/3 coverage goals. I call clickbait.
- dahfizz 6y agoThere is a big difference here between strongly typed, compiled languages and weakly typed interpreted languages. High code coverage is much more important on e.g. Javascript or Python because running the code is the only way you know that it "compiles" and you didn't do something dumb like mistake the type of arguments, access fields that don't exist, etc. In a compiled codebase, I think functional / end-to-end tests are more important. I don't care as much if every single function is exercised by unit tests if I know that the software works as intended for all supported use cases.
- dudul 6y agoThat's interesting because I think the exact opposite. For dynamically typed languages I think it's more valuable to have end-to-end tests to make sure that the whole pipeline works correctly. I want to validate that all the call sites for function F are passing an int, as intended, as opposed to a string. For statically typed languages, I already know that all call sites for F are passing an int. So I want to unit test all small parts of the pipeline, in isolation, to make sure I easily spot which part doesn't follow its contract. If a function F uses a function G for something, I'm gonna unit test F and make sure it does behave correctly for all possible returns from G, and that's it.
- morelisp 6y agoTyped language also make it possible to exhaustively test inputs. For functions that only have a few dozen or hundred possible inputs we can exhaustively test every single case (even millions/billions as part of a slow suite). For looser languages, maybe the function takes a fuzzy notion of "a number" - OK, do I need to try the string "12"? What about "012"? What about "12.0" or " 12" or "\n12"?
- bluGill 6y agoDynamically types languages are just plane unsuitable for the type of large project where you need to break your end to tests down into unit tests. If you have a trivial program (under 100,000 lines of code or so), your end to end tests won't take that long to run if you take care, you will never be able to maintain that much in a dynamic language just because change becomes so hard without static types.
- deleted 6y ago[deleted]
- leifg 6y agoAs with so many other code quality metrics and tools, automated testing (and by extension code coverage) is not necessarily about the number but about discovery. If you are finding out that certain parts of your software are hard to test or will never be reached by any of your tests, that should tell you something about your architecture.
- DavidWoof 6y agoCode coverage is an excellent tool but a terrible metric. It's basically the lines-of-code of test driven development. It's useful as a general guide for developers, but its use as a management metric is extremely destructive.
- theodpHN 6y agoEven better are claims of data coverage. USER: And every possible combination of data has been tested and validated to be correct like your management assured us? PROGRAMMER: No, that's simply not possible. For example, if there are 10,000 possible values for variable one, 1,000,000 values for variable two, and 5,000 values for variable three, that'd be 50 trillion possible combinations. And you have thousands of variables. USER: OK, but if that's true, why did your management assure us you'd verify every possible combination? PROGRAMMER: Because that's what you wanted to hear?
- alexchamberlain 6y agoInteresting article, but the answer I'd be looking for is #itdepends. What's the impact of a bug in this code base? How likely is there to be a bug? How quickly can you rectify a bug if it happens? Most of my Makefiles and setup.py files have 0% test coverage. Libraries that are automatically deployed to a few projects, where I can't easily ship a fix and I don't really fully understand the impact - I'd aim for 100%, but not arbitrarily; I want to make sure we've actually tested the edge cases.
- morelisp 6y agoHrm, when I run `./setup.py test` I run well over 90%, usually 100%, of the lines of code in my `setup.py`. Is this not coverage?
- alexchamberlain 6y ago:) python3.9 -m pytest for me. My point was more that not every line of code should be treated the same way.
- lmilcin 6y agoBecause code coverage should measure coverage of REQUIREMENTS, not lines of code. Right coverage metric would take as input list of all requirements broken into smallest conditions and then check if they all are verified by a unit test. In particular, this would cause almost all code to be covered because if a line of code was not covered by test it would mean it is not necessary to meet any of the requirements. Interestingly enough, unit tests would also be then complete documentation of every single requirement and that is how I understand "test driven development". Sadly, I have yet to see it implemented anywhere.
- CivBase 6y ago> 1/3 of the code every software project is irrelevant, buggy, overly complicated, or simply sucks. It has a reason to be where it is, but chances are, one year down the road, it will become a liability. Being dogmatic about tests and covering every line will only make it more difficult to get rid of it. If you have actually tested every line of code with your test suite (not the same as "covering" every line), then your code better not be irrelevant, buggy, or simply suck. That is the sort of thing you are supposed to uncover and fix as part of the whitebox test development process. If your covered code is that bad, you either haven't actually tested the code or you have completely missed the point of software verification. Code coverage can be a very misleading metric. I have to clarify the difference between "tested" and "covered" code to coworkers and management all the time. In the same way that "standard lines of code" is an insufficient measurement of complexity or development effort, "coverage" is an insufficient measurement of test quality. However, it can still help with ballpark estimates. Uncovered code is actually a much more useful metric, IMO. You can cover code without testing it, but there is no way you could have tested uncovered code. I think the point the author should be making here is that not every app needs to be 100% tested.
- Izkata 6y ago> Uncovered code is actually a much more useful metric, IMO. You can cover code without testing it, but there is no way you could have tested uncovered code. Hurrah, finally I see someone with the same viewpoint I've pushed for some time, though for a different reason. For me, 80% covered isn't the same thing as 20% uncovered. The difference is in the code that has been examined and decided tests for one reason or another aren't worth writing. Could be untestable (elsewhere in these comments is a malloc example), too difficult to reasonably test (API calls abstracted behind a facade; mock the facade in other tests but the facade itself is kept tiny and only tested manually), utilities like python's "__repr__", etc, etc. Having a way to mark such code as "does not need tests" is the key to making this viewpoint useable though, and I'd also really like it if coverage tools focused on counting uncovered down to 0% rather than covered up to 100%.
- mumblemumble 6y agoI would love to see a different metric be introduced: Code elusion. All it is is the inverse of code coverage. It would do a better job with the mental game. You may not be able to conclude that code is adequately tested just by knowing that the tests exercise it. But you can be pretty confident that code is not being tested if it eludes all of the tests.
- breck 6y ago> Being dogmatic about tests and covering every line will only make it more difficult to get rid of it. I find the opposite to be the case. I'm very comfortable tossing away a bunch of code with great test coverage. You can always cherry-pick it back later and know that it works. (I'm not 100% purist, but certainly I'd say 80-90+% and not 66%). The important thing is you should be so fast at writing testable code that getting to 80% is a like going for a light jog. If it feels like a slog, you've identified weaknesses in your craftmanship that you should practice.
- dnautics 6y ago> You can always cherry-pick it back later and know that it works. Exactly, "this is what git is for". I think being uncomfortable tossing tested code is more of a reflection of the temperament of the person saying it than any truism about coders in general. If anything a little bit of a roadblock with test cases can be a good thing. "By deleting this test I am really certain I want to remove this feature". There's one place in my code base where I have extra tests in there, that I have to replace every time (it's snapshot-testing code-generating code), but that one little roadblock make me feel good because I know that the code that is being emitted is human-readable and easy for an end-user to debug.
- breck 6y ago> I have to replace every time (it's snapshot-testing code-generating code), Do you store your snapshots separately? One note for devs reading this: I think snapshot tests are very cool, but I recommend trying them as a very separate thing than unit tests (if you are not already doing that). I removed all snapshot tests from our unit testing pipeline and it's much better for it. Was very surprised that Jest doesn't have warnings that snapshot testing is a very different thing, and should not be intermixed with unit tests. Snapshot testing is sort of like code coverage, a great tool for telling you what to look for, and you should make it so that doing snapshot testing is easy, but the snapshots themselves should be stored outside of the main repo/main testing repo, and in their own thing. Not sure if you are doing that or not, but for anyone else looking at snapshot testing, this is a common mistake I see.
- pmb 6y ago100% is the way to be. Once you get there, then you can start having useful conversations about how to make the coverage more meaningful. But first, you have to make sure that every line is run at least once without crashing (except for the lines that are supposed to crash things - those you need to verify DO crash things). Interestingly, once something is designed for testability it is more likely to not have bugs. But saying that therefore you don't need the tests is silly, because without the tests you wouldn't have designed for testability.
- hinkley 6y agoDisagree, empirically. Once you bury the needle, nobody wants to talk about whether the tests are garbage, because doing so means that you have to admit to misrepresenting things to management. So you’re either calling people out or complicit in the lie. People don’t want to look at these actions honestly, so they deflect to protect their egos. None of these dynamics are an issue when you’re at 80% code coverage. It’s easier to fix how we do things while still slowly raising the stats. You’re trying to run when you haven’t learned to walk (or with a few teams I’ve seen, crawl).
- deleted 6y ago[deleted]
- trinovantes 6y agoBack in my undergrad software testing course, we were taught test coverage in the context of a control flow graph. I think most coverage measurements (at least in the GitHub projects where I've seen a coverage badge) only refer to statement coverage where it checks if every node of the CFG is reached. It's one of the weakest forms of testing and will rarely reveal the more obscure bugs that edge/(simple, prime, complete) path coverage testing can reveal. I personally think the industry should stop using (node) coverage as some kind of golden metric.
- npollock 6y agoWhat types of solutions exist for automating the creation of front-end tests? Either full automation, or partial with some user input/direction. I found https://kwola.io/ https://kwola.io/ but wondering if there are other alternatives
- hinkley 6y agoIf you could automatically test a thing, what’s the line between generating tests and doing static analysis?
- maerF0x0 6y ago> Being dogmatic about tests and covering every line will only make it more difficult to get rid of it. This is only true if one is lackadaisical about how they architect their code and (unit) tests. You should be able to completely smash a function and it's tests without breaking any other tests ... If not then you're testing a different unit within those broken unittests.
- hinkley 6y agoSomeone on my team likes to write their own mocks instead of using the mocking framework. A lot of people don’t really understand the point of testing, but this behavior is pretty far down on the spectrum. The mock is hard to write by hand, so they get sunk cost fallacy and share it between tests. Now the tests are coupled to each other, which makes it hard to change features. But wait, there’s more. They made space for some delegation/composition that never arrived or disappeared, and so they’re sharing the same mock across two different test suites, in two different directories. And made very coarse grained commits, so even if I wanted to know how tf they got here, which I really don’t, I’d have trouble doing it. Writing complex scaffolding for your tests is supposed to hurt. Pain is your body telling you something is wrong. If you can’t set the preconditions in a few lines then you don’t understand the problem, or you don’t understand your own code. Both are bad for your coworkers. Big decisions are the culmination of a bunch of little decisions. They are not the little decisions themselves. Don’t write your code like they are, and all of this shit gets ten times easier.
- UK-Al05 6y agoThe mock should display the same behaviour as the real thing. So I don't think sharing one is bad. The ideal fake is one that shares the same behaviour as the real thing. And you have tests that run across both to validate this. Quite frankly mocks become the bane of large code bases. I don't know how many times I've seen tests that are basically useless because the mock behaves nothing like how it behaves in the real world. Like throwing an exception for not found rather than null. Keeping those contracts in sync are much easier with fakes.
- capoDanger 6y agoNothing engineers like more than evidence that they don't need to write tests!
- eikenberry 6y agoI always say between 60-80%, 60% for greenfield or new testing as that is generally right for covering the golden path(s) and the major error cases. It should slowly grow over time as bugs get tests written for them, but if it gets over 80% you probably want to refactor some. >80% test coverage digs way to far into the code, testing the implementation instead of the interfaces/apis. It is a definite code smell that probably means the tests were written poorly and need some major reworking.
- capableweb 6y ago> that is generally right for covering the golden path(s) and the major error cases The thing is, your metric of 50-80% has nothing to do with if you're actually covering the golden path(s) and major error cases. The error cases you didn't think of, wouldn't count in this metric of course, but they might still be missed. Code coverage as a metric says as much as number of lines as a metric. Meaning nothing, it's just a number. Aiming at any sort of code coverage metric misses to think about what to actually cover. 0 code with 0 tests has 100% code coverage, doesn't mean it's actually a good program.
- mdpopescu 6y agoHmm... in my experience, stable code is more likely to be correct than code that changes often. In fact, I would consider this obvious, given that there's a non-zero chance to introduce a bug with every change.
- nullc 6y agoI think code coverage (particularly condition coverage) is mostly useful for one purpose: You need to get it close to 100% before you can use mutation testing. Without mutation testing you really shouldn't be particularly confident that your tests are testing much of anything useful.
- MauranKilom 6y agoAll these discussions in the comments about coverage percentages would be so much more insightful if everyone mentioned the kind of code they work on. Client-side JS? Real-time video processing firmware? A language support library? Automotive middleware? Java for an ATM? An ML library wrapper? Personal website backend in Rust? Android game engine? Cosmic particle simulation? I don't even care about the implied "what could go wrong" but about the size of the input spaces and the complexity of the algorithms behind them. I certainly don't subscribe to the idea that there is a one-size-fits-all testing strategy for all the above, and I see little gain in arguing for or against a certain approach with someone who has entirely different constraints and code architectures to work with. Require 100% code coverage on an async task library? Sure, go for it. 100% coverage on a (deep-learning-free) algorithm identifying constellations in pictures of the night sky? Do you really want to have to violate every single assumption one by one in your unit tests? How do you even get that input data (and who will create more nonsense data when you improve the algorithm)? Or in other words: For any reply in this thread you can probably come up with a scenario where it makes sense to operate as described in it. There is some nice perspective but overall I find it a bit unactionable...
- whateveracct 6y agoCode coverage as a single percent is low value and mostly management fodder. Code coverage as a percent per module has more value. Code coverage as highlighted source calling out uncovered code, missed branches, etc is useful during development to help you remember to test stuff.
- deleted 6y ago[deleted]
- golergka 6y agoConversations like this a meaningless without first determining what kind of software are we talking about. If it's a first version made quickly to test an idea, the ideal would be 0%. Rapidly developing product, 40%, mature product, 60%, database (like Postgres or SQLite) - 100%, and any software that deals with money, health and airspace should have each live of code covered at least three times by completely different tests, both unit and integration. There's not one size that fits all in software development. And if you don't specify what software are you talking about, everybody will explicitly talk about their own domain and experience.
- smithza 6y agoLoosely coupled code could well have unit testing and mocking covering >90% with great results. We should not conflate unit and integrations tests here. I have also found regression testing to be a great way to declare and interpret intent when trying to understand someone else's code. I think the classic book "Working Effectively with Legacy Code" declared "legacy code" to simply be code that has no tests.
- 29athrowaway 6y agoCode coverage works best when you have a strongly typed language. Otherwise it is a misleading metric.
- GuB-42 6y ago> The only sure-fire way to improve code coverage (and by that keep software relevant) is to identify and remove the unnecessary code. I remember what I think is a demoscene group taking that to the extreme. They ran coverage analysis and removed all uncovered code, thus achieving 100% coverage. The point was to fit all their code in an very small executable (typically 4k or 64k). It was absolutely terrible for robustness: all edge cases are ignored. But since it was a demoscene production, it didn't matter, but if it was a server, that would have made it the most exploitable code ever.
- leonfs 6y agoIn our teams, is less about bugs and more about the team culture. A test mindset that ranges from the business ideas, to the new features developed and the code that we write. We enforce 100% test coverage. Not 90, not 95, not 99, but 100. You need to test your code if you want to reach production. And production is the only environment we have. Having just prod generates less friction and allows us to make more changes, more features. But in order to live by this premise, the team needs to be really good at testing.
- jerrod 6y agoHey Jerrod here, CEO of Codecov. We live in coverage world every day and I agree that there is work to be done. Fundamentally, no one ever said that coverage had to just be a binary of 0 or 1 as a "hit" line, though it is definitely the branding that code coverage has today. As some others on this thread mentioned, how is something being tested should matter: - Unit vs. integration vs. end-to-end test vs. other testing - Flakiness of test (non-determinism) - Accuracy of test (E.g., mutation testion / fuzzing) Also, What is being tested matters: - How important is this line? How is often is it actually being called in production? - Are their errors / exceptions on this line that we can see? I do not believe that 100% coverage is the goal, nor should it be for all teams. Do you have feedback here? I'd love to talk more about this.