11 ms·

Coverage is not strongly correlated with test suite effectiveness

• domk 5y ago
  In practise, doesn't increasing the coverage highly correlate with increasing the test suite size, therefore proving the effectiveness? Conversely, I struggle to think how coverage could be increased significantly without increasing the test suite size in reality.
  • gravypod 5y ago
    It depends. When you tell developers "you must have 100% code coverage" they usually write tests that don't actually validate any functionality and instead get into every if block. Tests are useful when they test edge cases and assert behavior. I've told this story many times before but at a previous job a senior engineer told me "100% code coverage is useless and you shouldn't go for it" but since he was being dogmatic and not actually thinking about what he was saying he was arguing against something very sensible. I was testing an expert system where everything was large if/else trees encoded in types + configuration. I wanted to make sure I tested all edge cases and activated all of the blocks when they made sense. I had to fight for that extra coverage and it was, in the end, a massive help.
    • tharkun__ 5y ago
      100% code coverage is still not a good use of everyone's time in most projects and languages. There's a bunch of trivial code that even feels wrong to test. Spend more time cooking up edge case tests that execute some of the branches way more than once. Think of your code like a heat map. Higher heat on lines that get exercised more often by your tests. It's fine that _some_ code has no color at all, while you some other paths to be bright red in the end, instead of always just going for a uniform orange for everyhing.
      • gravypod 5y ago
        > 100% code coverage is still not a good use of everyone's time in most projects and languages The key here is most. Think about your use case before applying anything you read online. > It's fine that _some_ code has no color at all, while you some other paths to be bright red in the end, instead of always just going for a uniform orange for everything. This was the logic tree of a device used for health care work. Cost of failure was high. I had very good coverage of all edge cases that other systems could produce and tested a lot of extremes + a DSL for describing the input state. The important thing here: an expert system is not most programs and the cost of a failure should really drive what your testing methodology is.
        • tharkun__ 5y ago
          I did write most on purpose, instead of all. Looks like your case is one where 100% coverage is a good thing and the cost paid for it is absolutely acceptable - nay needed to be paid. And you didn't just go for 100% and stop there because you hit some metric but you actually did the thing that's more important too ("data coverage"). Kudos!
  • Swizec 5y ago
    If it takes you 3 days to write a test that covers a once-in-a-million condition and your service gets 2 requests per day, it will take you about 500,000 days to hit that condition once. You've increased test coverage, but was it effective? Eh probably could do something more useful with your time. (yes this is a contrived example, adjust numbers for your situation)
    • colechristensen 5y ago
      Depends on what the consequence of that failure might be. It could be anything from completely unnoticed to ending somebody’s life, rarity is only one dimension of risk.
  • lhorie 5y ago
    test("when a metric becomes a target, it stops being a good metric", () => { runApp(); // look ma, lots of "coverage"! assert(true, 'No errors!'); }); // unfortunately paraphrased from real code
    • xiaq 5y ago
      This test will inflate the test coverage, but it is a valid smoke test (assuming that any unhandled exception will cause the test to fail).
      • lhorie 5y ago
        You shouldn't write tests like this; there's a high likelihood that the test will be flaky or not representative enough of production, and if the test fails, you often get completely non-actionable error messages. If you just want to know that your app is broken, it's far better to monitor your live app (or staging environment or deployment pipeline or whatever) since that monitoring infrastructure can then be leveraged to collect other runtime health data in a more granular fashion.
  • Couto 5y ago
    Coverage only indicates which parts of the codebase were touched by the test suite. A big test suite size doesn't mean high coverage. Coverage can be increased without increasing the test suite by reducing the code base size (within pratical limits obviously) Personally, I only find coverage as a good indicator of which code still needs to be tested, like forgetting some edge cases or conditional branches.
  • colechristensen 5y ago
    Quality of trests is important, if you’re chasing 100% you’re liable to be writing inconsequential or incorrect tests just to achieve a number.
  • goodpoint 5y ago
    The correlation obviously exists for low levels of coverage. Above 80% or 90% it becomes a poor measure.
• laanako08 5y ago
  This affirms my intuition that the power of test suites arises from their coverage of the data-cases, and call-sequence, and not simply from "visiting" more lines of code. This also is likely the underlying reason for the extreme effectiveness of fuzz-testing and property-based testing.
  • amw-zero 5y ago
    They go hand in hand. It’s obvious that missing branch coverage means your data cases are not exercising all of the edge cases in the code.
    • falcolas 5y ago
      Not all code is reachable. Especially things like top-level try/except clauses.
      • amw-zero 5y ago
        They're in your source code - they are reachable.
      • Supermancho 5y ago
        From the entry point, stub a function to throw an exception. You'll reach it. Unit testing isn't about finding every path. Nor is it about writing unit tests for every possible combination of values in a program. A java function that uses an int does not need to test -2147483648 to 2147483647 as inputs. That's not helpful.
        • amw-zero 5y ago
          That’s an interesting example, that’s exactly where you’re likely to find bugs - at boundary values. You should test those more than you should test the number 5. This is exactly why unit testing gives a false sense of security. You’ve done a lot, you’ve written all kinds of tests - but at the end of the day, you don’t get a proportionate amount of confidence about the code because there are infinite more cases that you haven’t thought about.
• caltelt 5y ago
  I usually just use Unit Test coverage as extra insurance that I didn't accidentally forget a path when writing up tests for new classes, or when trying to assess others tests to see if/what they miss. Outside of that, I basically ignore it. That (per)mutation testing, sounds like pitest, which I've had fun with using to gauge the effectiveness of tests I've written in the past.
• falcolas 5y ago
  This makes perfect sense: a simple function with two code paths that splits on the comparison of two signed integers immediately requires a minimum of three test cases for correctness, yet it only takes two to achieve 100% code coverage. Checking for correctness for corner case values - maxint, minint, zero - adds a minimum of another 9 cases. And it will take many, many more test cases if you're working with a weakly typed language and you're potentially comparing an integer with a floating point value. Or strings. And so forth.
  • throwawaylinux 5y ago
    Not to disagree with the idea that you generally need more test cases than control flow paths to really test correctness well. Just a question about your example -- let's say your requirement is a function that does this: void fn(int a, int b) { if (a == b) printf("equal"); else printf("not); } What are the 3 test cases you would write? What are the 9? fn(1, 1) -> "equal" fn(1, 0) -> "not" What more useful tests are there here? Other than just picking arbitrary numbers, maxint/minint/1/0/-1/etc are not significant edge cases here.
    • fiddlerwoaroof 5y ago
      It’s not a problem here, but if the type doesn’t guarantee reference equality for equal value, you’d also need a test to differentiate between a == b And (in Java) a.equals(b) In order to catch misguided refactorings. (I’ve introduced bugs this way before I learned that == can give a false positive for strings if the strings are interned)
      • merb 5y ago
        in java the method that takes two `int`s would still use ==. java has value types which don't need `equals`.
        • fiddlerwoaroof 5y ago
          Yeah, that’s why I said “it’s not a problem here”.
• tharkun__ 5y ago
  In other words, more tests do find more bugs, but it's the number of tests and not their code coverage that has most of the predictive value. It's a surprising result, so if you'll excuse me, I have a couple of lecture slides on software testing I need to revise Is it just me or was this _not_ surprising at all? I mean I suppose I should have expected what he said, given it sometimes seems hard to convince other people about this but to me it's a well known fact. There are so many ways this can go wrong. I mean it's so easy to give the one counter example needed to break the myth of 100% test coverage being good for much: Well you executed the branch/line at least once with one potential input. Was it an edge case input or a happy path input? More tests than is needed for "100% coverage" means that you actually executed some lines multiple times, hopefully with not just 10 happy path scenarios but with 1 happy path and 9 edge cases. Now remove the happy path scenarios for trivial code and also the edge case scenarios for trivial code and your coverage might only be 80% but you have the same actual test suite effectiveness. When you keep adding tests, add more to the 9 edge cases, staying with the same coverage but make the suite more robust. Of course 20% is better than 0% and 50% is better than 20%. Somewhere between 50 and 100 is an optimal point. For lack of a good estimate, let's go with the old 80/20 rule. Something around 80% test coverage is what you can use to make a tool fail the build. If you pass the 80% mark you _might_ have a good test suite but it doesn't mean you stop there. It's just a reminder that you might have missed adding tests when that thing triggers a red build but don't use it to mean that you actually have a good test suite.
  • void_mint 5y ago
    > Is it just me or was this _not_ surprising at all? It wasn't surprising to anyone that has reflected about the value of tests. Mindless testing/TDD isn't usually reflective, though.
    • fiddlerwoaroof 5y ago
      It’s hard to do test-first TDD “mindlessly” because writing a test usually forces you to think in terms of the specification of the behavior you’re about to implement.
• windows2020 5y ago
  function (int i) { return 1 / i; } 100% coverage means nothing here when i = 0.
  • windows2020 5y ago
    Furthermore, a line of code is arbitrary, and so is this metric.
  • amw-zero 5y ago
    Well, coverage can still tell you that this line is not covered at all. The absence of coverage is a signal, but the presence of it is not sufficient to prove that the code works for all inputs. We’ve also known this since forever, this has always been Djikstra’s message to software developers.
  • imcotton 5y ago
    100% coverage is baseline, not the ceiling.
  • xxs 5y ago
    kind of a bad function, though - can return -1, 0 and 1 only - plus the integer division by zero, unless 1 is a float type then 0 is positive infinite which is good.
• hondo77 5y ago
  "(F)ive large open source Java projects" is hardly a big and diverse enough sample to be making such a sweeping conclusion.
• jacobsenscott 5y ago
  It is easy to write a test that executes code without actually testing anything. I use coverage to find code with no tests all at, and write tests for that code. But once it is "covered" the coverage report is useless. In interpreted languages (ruby/python/etc) coverage at least tells you if there's a syntax error before running it in production, which is useful. Test first also improves the quality of the tests just because it forces you to write a failing test first, and then write the code to make it pass, and you do this over and over as you build up the production code. In this way you are actually proving that the code you are adding is making tests pass. That doesn't mean your tests are great - you are probably still mostly testing the happy path. But then you have a solid foundation to layer mutation/fuzz testing on. I can't tell you how often I see people write tests after writing the production code - and I can go and delete most of the production code and the tests still pass. Writing quality tests for existing code is much harder than writing tests for code that doesn't exist - it is counterintuitive if you haven't done it both ways.
  • taeric 5y ago
    I was going to be worried if mutation testing wasn't mentioned here. Is a great way to test the effectiveness of your tests at catching the common mistakes people make in code. That is, a mutation suit doesn't test your code, per se. It tests your test suit.
    • sverhagen 5y ago
      I've worked a lot on Java, and while there is great mutation testing tools (well, PIT specifically), I find them hard to "scale" practically, i.e. to run them in an automated fashion "every time". You either get long running builds with them in it, or you have to deal with the logistics of moving caches around (so that the mutation testing can be incremental). And as just another tool that a developer MAY use if they so choose to, it's... well... nice... but no longer front of mind, for most people on the team. I'm very interested if you're experiences have been different?
      • taeric 5y ago
• kazinator 5y ago
  Well, the number of case combinations that can occur can be exponential in the code size, whereas coverage is linear. Say we have a sequence of five one-branch if statements: if (cond_i) { do_stuff_i } for i from 0 to 4. We can get complete coverage by executing that sequence just once with all conditions being true. But there are 32 distinct pathways through it, which could have all sorts of bugs. Coverage also doesn't test for correctness at all; suppose that the one case that hits 100% coverage by executing every do_stuff_i produces the wrong result for that case. 100% coverage was still achieved. A test suite consisting of nothing but a coverage test is ineffective in doing anything other than showing that the software doesn't bomb or infinite loop.
• amatecha 5y ago
  (2014) Also, the actual paper is paywalled :(
• icedchai 5y ago
  I was on a team that needed to write tests to deploy their app on an "enterprise CRM" that you probably all have heard of. You couldn't upload the app for distribution unless you had X% test coverage. Of course, nobody wants to write all these tests. We had a few real tests. But most of our tests would call methods and check for != null. This worked great, got us up past 70% coverage with little effort. Supposedly these apps went through a "review" of some sort before they were released to the public. Oddly, our test methodology was never an issue.
• KZeillmann 5y ago
  A team getting to 100% test coverage and enforcing it feels like an application of Goodhart's Law. When a measure becomes a target, it ceases to be a good measure. At my last job, we required 100% code coverage for most code, and the other parts of the code (in ideality) were marked with ignores that were well-thought-through. In practice, I ended up writing a bunch of test cases only to hit the if blocks. It didn't mean that the tests I wrote were any good. I'm still a fan of 100% test coverage, but with some leeway for what needs to be ignored, and what needs to be tested later (non-critical paths, if you have a deadline, perhaps)
  • dnautics 5y ago
    It gets crazy with demands for 100% test coverage, in languages where you're encouraged to write compile-time logic, macros, setup-dependent minor details, time bombs, asserts which throw compiler errors when you misconfigure, etc.
  • SilverRed 5y ago
    I have never used the % stat as anything more than curiosity but I have certainly looked at the highlighted reports generated to see large blocks of untested code which is genuinely useful info to see.
• muglug 5y ago
  Hillel Wayne discussed the paucity of decent research in software engineering land: https://buttondown.email/hillelwayne/archive/i-ing-hate-science/ https://buttondown.email/hillelwayne/archive/i-ing-hate-scie... (discussion: https://news.ycombinator.com/item?id=27892615 https://news.ycombinator.com/item?id=27892615)
• lwansbrough 5y ago
  Now, I've never worked at a big company with lots of developers, but testing seems dramatically overvalued. My company currently employs zero testing (meaning zero automated tests). Anything that could negatively impact the company if it blew up is examined pretty closely and then set loose. Occasionally things break. Our company serves tens of millions of users monthly. We have bugs, we fix them as needed. We're not writing banking software so the risk profile is pretty small, but I think more companies fall into this category than test engineers would have you believe. I'd give myself 2 more years in the industry if I had to do TDD, or anything else that strives for even moderate coverage.
  • jeffbee 5y ago
    I am a huge fan of unit testing and the best thing about TDD is getting your function signatures and other APIs sorted out before writing your implementation. Having to write the calling function first helps ensure that the called function has a signature that will be useful to other callers, instead of something awkward that seemed OK when you started writing the implementation but later turned out to be imperfect. Also, TDD wards off the problem of people who write untestable code. A lot of functions simply cannot be tested. TDD eliminates that problem.
    • lwansbrough 5y ago
      The notion that you can get software right the first time seems... a bit naive to me, regardless of the methodology, respectfully. We solved the internal API problems in our app simply by either versioning internal services/APIs (ie. MyServiceClassV2), adding a new method signature and deprecating the old one, or updating the existing method signature, which is pretty safe in strictly typed languages. (And if you write code that uses reflection, you get to be on call when it breaks.)
      • jeffbee 5y ago
        I find your comments quite interesting. How do you validate MyServiceClassV2 if you didn't bother writing any tests for MyServiceClass? One of the benefits of testing is it enables refactoring. Without tests, code bases just become increasingly haunted graveyards where nobody is willing to change anything.
• TheDudeMan 5y ago
  I want my 90 seconds back.
• amw-zero 5y ago
  You don’t even need to do a study to arrive at this conclusion. Has no one read Djikstra’s writings? https://www.cs.utexas.edu/users/EWD/ewd02xx/EWD249.PDF https://www.cs.utexas.edu/users/EWD/ewd02xx/EWD249.PDF “Program testing can be used to show the presence of bugs, but never to show their absence!” Everybody in our industry wants to bash math and say that anyone can write programs, but programs are logical systems and can only be fully understood with math and logic. We have the tools for understanding and reasoning about infinitely large structures, programmers just refuse to use them, and even deride them. Well, I am not in that camp, and I work on math and logic skills to help me develop working software. It isn’t perfect, but it’s the only path that makes sense to me. > But complexity sells better and the market pulls in the opposite direction. I still remember finding a book on how to use "Wordperfect 5.0" of more than 850 pages, in fact a dozen pages more than my 1951 edition of Georg Joos, "Theoretical Physics"! It is time to unmask the computing community as a Secret Society for the Creation and Preservation of Artificial Complexity. And then we have the software engineers, who only mention formal methods in order to throw suspicion on them. In short, we should not expect too much support from the computing community at large. And from the mathematical community I have learned not to expect too much support either, as informality is the hallmark of the Mathematical Guild, whose members —like poor programmers— derive their intellectual excitement from not quite knowing what they are doing and prefer to be thrilled by the marvel of the human mind (in particular their own ones). For them, the Dream of Leibniz is a Nightmare. In summary, we are on our own. But that does not matter. In the next fifty years, Mathematics will emerge as The Art and Science of Effective Formal Reasoning, and we shall derive our intellectual excitement from learning How to Let the Symbols Do the Work. Calculemus! https://www.cs.utexas.edu/users/EWD/transcriptions/EWD12xx/EWD1243a.html https://www.cs.utexas.edu/users/EWD/transcriptions/EWD12xx/E...
  • brabel 5y ago
    > We have the tools for understanding and reasoning about infinitely large structures, programmers just refuse to use them, and even deride them. Could you point out which tools you're talking about?
    • amw-zero 5y ago
• sverhagen 5y ago
  I've been asked many times what is the right code coverage percentage to aim for. I've also asked this many times in interviews, to tease out a discussion. My answer is: you need full coverage. I then continue to explain that I didn't say 100%. Although I believe that the average Java microservice (which is the general sphere that I move around) can easily achieve >98% coverage. Easily. But "full coverage" means that all reasonable scenarios should be covered. No code coverage tool is going to measure that for you (I think). Reasonable scenarios are... everything? Time is finite, so most teams will never be exhaustive, but one can try. The "80% coverage crowd" is throwing in the towel before even starting.
  • saalweachter 5y ago
    I've occasionally been lulled into a false sense of security by the difference between "every line is run" and "the function is fully tested". On the other hand, even if I'm half-assing the unit tests, and implementing the bare minimum necessary to reach whatever arbitrary coverage% is being sought ... I still find bugs, bugs that almost certainly would have a real impact in production, even if I already ran an end-to-end test and was pretty sure it was working fine[*]. Not every time I write tests, but often enough to feel like it pays for all of the time "wasted". So even if coverage metrics maybe arbitrary, I don't really begrudge having to meet them. And honestly, once you're forced to start writing the tests just for coverage, it isn't that much extra work to make them a little more comprehensive than the floor set by coverage. [*] Honestly I'm shocked at how well buggy software works, for the most part.
    • AlexCoventry 5y ago
      That's my experience, too. Coverage isn't a good proxy for test quality, for sure, but it's a good proxy for having looked carefully at every line. I think coverage testing is useful as long as you're prepared to throw those tests away when you refactor, or at least ignore them during refactoring.
  • jamesfinlayson 5y ago
    Agreed - if you can test it then test it. 100% coverage may not be easily achievable with some languages/frameworks, but as other have pointed out, it's a good idea to work towards it for dynamic languages so you can reduce the possibility of runtime errors.
• arcticbull 5y ago
  As the suite approaches 100% coverage I'm not surprised coverage doesn't correlate with effectiveness. However on the other end, as the suite approaches 0%, I think the correlation is much stronger. I view my job as knowing when to stop.
• cratermoon 5y ago
  Goodhart-Strathern's Law: When a measure becomes a target, it ceases to be a good measure
• hsn915 5y ago
  Aiming for 100% test coverage actually produces negative value. You don't need a "study paper" to know this. Just work with a team that aims for 100% coverage for a few months and you will see it for yourself. The negative value comes from: 1. The time wasted on writing all these tests that are mostly ceremonious in nature. But, more importantly: 2. It makes refactoring a big pain in the ass. Why? Because 100% test coverage forces you to test the implementation details of everything. So, every time you do a non-trivial refactoring, you will get tens or hundreds of failing tests. This is mostly noise. If there's any signal in any of these failed tests (signal = information that you screwed something up during the refactoring) it will be very difficult to catch. What do you do when you refactor something and break a 100 tests? You delete all the failing tests and start again: look at the coverage report and start writing tests for the portions of the code that are not covered.
  • lmilcin 5y ago
    It is very unpopular view. I have never been a fan of unit testing and instead I prefer end to end functional tests -- where you write tests that verify your application still behaves exactly as expected but does not care how it is implemented. This usually requires much less code, does not deter refactoring and also focuses on the one thing that is really important for the client. Since the outside interface of the application is less likely to change compared to the implementation, the tests tend to be more stable and require less maintenance over life of the application. Unit tests were supposed to help refactoring (by making it easier to ensure the code still works after change). The sad reality is that I can never trust unit tests. I still need to research the code around the function I am modifying to be sure I am not breaking anything. Also my personal style is to do shit ton of refactoring. This means I usually start with something that only vaguely resembles the end result. I spend a lot of time moving stuff around until I get rid of everything that I don't like. This absolutely precludes writing unit tests up front. But also, after I have spend so much time polishing the code, writing unit tests for it is absolutely the last thing I want to do. There is absolutely no possibility that people will put effort into writing tests well if they did good job writing the actual implementation. Yet another reason why unit tests are broken is that there is no mechanism, no feedback loop to ensure quality of tests. When you write application code, if it is broken you will have feedback in the form of defects and outages or various other problems. One reason that documentation is usually broken is because there typically does not exist mechanism that would ensure that documents stay in sync with design and implementation. The same problem with unit tests. The only strong signal you get is when the test does not work. But there is no good signal to fix real important problems like missing tests for important part of the contract.
• lmilcin 5y ago
  That is pretty obvious. The main reason is that what we would really want is measure coverage of the specification of the unit. Now, that would be difficult, so we measure something else with obvious poor result.
• satisfice 5y ago
  Ever since I developed code coverage tools at Apple in 1989, and tested them for Borland in the early 90’s, I knew and have been telling people in MY conference slides that code coverage is a nearly useless metric. Anyone who thought critically about it for ten minutes knows it’s nonsense. The one thing code coverage tells you that is of any significant value is what you haven’t tested. You still know very little about what you have tested. Meanwhile, this article continues the dopey practice of using the word “coverage” only with respect to code and not with respect to many other forms of coverage, such as data. Data coverage is not easy to measure, yet it is far more important. Generally speaking, academics know very little about testing, and academic papers about testing are nearly useless. If you want to learn about testing from an academic, read sociology, philosophy, and cognitive science.
  • amw-zero 5y ago
    Yet it's extremely popular to have huge, inefficient test suites running on continuous integration servers hundreds of times a day. It boggles the mind.
    • datavirtue 5y ago
      This is a good thing if you consider the price of not having them. I have worked in shops without tests. It's a great way to hand out free money to unsuspecting customers.
      • amw-zero 5y ago
        Yes I agree - I have been a testing fanatic for the better part of the last 10 years, after being absolutely paralyzed at a company without tests. But, after all this time, I believe their cost-to benefit-ratio is horrendous. It’s fairly common to hear of test suites with a 2:1 ratio of test to implementation lines. That would be fine if they didn’t immensely prevent refactoring and block merges / deployments. Contrast that with something like F*, where the specification and code are right alongside each other, and the implementation gets proven. They are reporting a 5:1 ratio of specification to implementation code lines, which is much better than only a few years ago. Soon I think we’ll be striking distance to largely get rid of huge test suites which are so commonplace today.
• gilbetron 5y ago
  Say we have an application covered by 1000 tests. We fast forward a few years and (amuse this crazy notion) the application is decommissioned. We look and see that 500 tests never failed; those 500 always passed. Did we waste dev time by writing those tests? It's an interesting question to think about.
  • exdsq 5y ago
    If I have a phone I need for work and insure it because if it breaks I could lose my client, but never actually need to claim on it, did I waste my money?
    • gilbetron 5y ago
      No, but I don't see the connection to my question. Maybe a slightly closer example is, if I have a phone, and 2 backup phones, and I never use the 2 backup phones, were they a waste?
  • gilbetron 5y ago
    It's also interesting to see the response from people when I even pose this question, or any other question that might even remotely cast doubt on the CI/TDD dogma. Another question is: how do we rectify the complexity added to the total body of our source code when we add in really complex tests? Is there a point at which we are adding more complexity and increasing the potential bug surface area?
• arwhatever 5y ago
  High coverage does not necessarily imply test suite effectiveness, nor does it even guarantee that your test suite provides positive value. But 0% coverage does tell you a lot about your test suite effectiveness. Also, beware the person too eager to proclaim that high test coverage is not necessarily good! Last time someone told me this was discussing process maturity during a job interview. I let the team slide with this answer. Found out their coverage was effectively 0 and their process maturity was "break everything until right before each quarterly release" and I was out of there before my first week on the job.
  • malthuswaswrong 5y ago
    That's a shame. You could have gained new experiences and taught a team something new. Writing a program is easy and done a million times a day. Fixing a department is hard and people rarely get an opportunity to do it.
    • arwhatever 5y ago
      The Internet is a deluge of information and training material on this subject, so the teams that want to teach themselves this skill have already done so, leaving the teams that don’t want to do so. I didn’t go into detail in my initial description, but this team misrepresented their process maturity to the extent that I had no qualms about bouncing out just as quickly as I got it to take a candid look around. I’ve beaten my head against the wall for years on teams that didn’t give shit, so made testing and process maturity my primary focus during my job hunt, and made that clear to this team during the interview.
• jake_morrison 5y ago
  Beyond a certain point, test coverage has diminishing returns. You end up testing failure cases which are better addressed by good error handling. For example, inability to open a file due to file permissions can be covered in a more general way. Erlang is one of the most effective platforms for making reliable software. It has the concept of "supervisors" which will monitor and restart code if there is a problem. This handles both known problems which are appropriately managed by the standard strategy as well as unexpected problems. You can also use other static code analysis tools that ensure that you have dealt with every possible return code from a function. Other things become more useful than testing, e.g. implementing observability to tell you what is happening at runtime. Instead of more unit tests, I would prefer to have more validation checks when deploying code and in production. A validation check for Blue/Green deployment prevents bad code from going live. A production check ensures that nothing bad is happening that affects users, e.g. if we would normally get 10 signups per hour, and we are now getting zero, then it's time to page someone. I will take that over an extra 10% code coverage any day. Put the effort that you were going to use on code coverage into something that gets higher value.
• datavirtue 5y ago
  Hmmm....how to count tests? By the number of methods or the number of assertions?
  • exdsq 5y ago
    Ideally, according to best practices, these should be roughly the same!
• dehrmann 5y ago
  I find it somewhat rare that I update code and replace an implementation in a way that tests immediately pass. That said, I think the value of unit tests isn't so much in the coverage metric as much as 1) showing someone did the legwork to test the code 2) document some amount of caveats and expected behavior, and 3) provide the next person to edit the implementation a small test "framework" (mocks, dummy data) to build on.
  • datavirtue 5y ago
    This.
  • makeitdouble 5y ago
    For some codebases I think tests breaking at every change is the expected behavior. In a way it gives an approximation of the dependency graph of the code worked on, and acts as a reminder of the other functionalities that one might not have intended to touch.
  • SilverRed 5y ago
    The tests provide a useful confirmation step. You make a change and the tests go "Hey did you know that this affected the output in these 3 places?" and most of the time you did know that and update the tests, but often you see a change you didn't actually expect and can now look deeper in to it. I have also found immense value in tests that just check that literally nothing has changed. Like one that checked that the columns of a db table have not changed and when they have, the test reminds you that you need to decide if the new column is blacklisted from a certain feature and then add the column name to the test so it passes again.
• drittich 5y ago
  I think it's easy to conclude code coverage is not a proxy for useful tests. But the lack of any significant degree of code coverage does tell you something. And encountering zero code coverage is certainly a problem. So, there is a continuum here, and a balance to be found. I have come to realize there are many reasons we test, and they are often in competition with each other for getting developer's time to work on them. (Yes, like the Selfish Gene, having us write more code is simply the way tests get us to write more tests, which is really the end goal of it all.) Lately I have been toying with the idea of creating different suites of tests for each of these testing concerns, so I could track coverage across them independently. I'm thinking of a) basic proof-of-execution tests, which some people think don't tell you much, but in certain codebases, simply validating that you can build a string of SQL from a varied bunch of parameters and execute it without getting a parsing error provides a significant amount of value, regardless of the results returned. And, you can write that test knowing nothing of the business logic. Another level of tests might be concerned with b) the structure of the method: testing all branches or combinations of parameters. Another suite c) would focus on functional tests. And then you have d) mutation and e) characterization tests. For different parts of the codebase, you might choose to favour increasing coverage for one of these suites over the others. You also might wish to add a suite of security or performance tests. You don't want to get too granular, but the reason code coverage as a metric is not super helpful is we don't necessarily know what those tests are doing, so the more we can constrain their focus, the more meaningful the code coverage results become. And as a bonus, some of these types of tests are very quick to write (a, d, and e), so you can get quickly get to some useful degree of coverage even on a code base you don't understand, or that has no coverage at all. The value of this is not to be discounted, as writing tests can be very time-consuming, and seem like an impossible task when faced with a large, legacy codebase.
• drodgers 5y ago
  I've always wondered about building a test-sensitivity metric to complement coverage (and thus make coverage more useful): - Pick a random line of code (skip comments and whitespace if you can) - Delete/modify it in some way - Run the test suite and see if it passes Do this thousands of times on a codebase, then report on the test suite failure rate accross all of the random changes, broken down by file etc. You could track it all in a long running database and focus each test on lines which have recently changed to make it less expensive. I've never actually done this, but I'd guess that you could get a good sense of which files have good tests, and (combined with traditional coverage data) which files have bad tests vs just no tests.
  • ericpauley 5y ago
    This is known as mutation testing[1]. [1] https://en.m.wikipedia.org/wiki/Mutation_testing https://en.m.wikipedia.org/wiki/Mutation_testing
  • anonymoushn 5y ago
    Some especially corectness-concerned or compatibility-concerned people use mutation testing, which changes each line of code and fails the test run if no tests fail as a result of a change.
• eyelidlessness 5y ago
  Coverage is: - a negative signal (“low” coverage = no coverage) - not a positive signal (“high” coverage only means code paths were reached) - doesn’t mean your tests are thorough - doesn’t mean your tests are correct - most importantly for my current work, doesn’t mean you even have an accurate number (istanbul reports 100% coverage if it doesn’t instrument anything; nested languages aren’t instrumented at all, good luck finding out coverage of an XSL transform in JS) - (still relevant to my work) a good way of conveying safety of small, methodical refactors that can be eyeballed along with incremental changes to see they’re logically equivalent
• cowanon22 5y ago
  This is yet another paper where the title exaggerates the importance of the conclusion. I found a copy of the original paper - there are several risks to the model that the authors mentioned: - They only mutated conditionals to test coverage, not any of the other possible errors the test suite may be looking for. - Equivalent mutations may be miscounted, particularly when developers test a lot of off by one errors. - There data may not meet the assumptions of the Kendell r correlation used. - Most of their data had low levels of coverage - previous research shows high coverage is needed before it is related to effectiveness. - They did not account for object oriented code boilerplate code (getter/setters) which do not need to be tested, causing their counts to be potentially be off. This is major, as they were using only Java projects. - They had very narrow inclusion criteria, so the results may not be generalizable to all codebases. The projects had to have over 1000 test written; the average LOC was generally in the 100k range. This honestly sounds like some grad students final project, not advice for real world digestion.
• ummonk 5y ago
  "Next, they selected random subsets of the project's original test suite of varying sizes, ran each against the buggy mutants, and counted how often the subsetted test suite caught the bug." I'm sorry what? Of course if you only run a random subset of tests, it won't matter whether the test suite achieves 100% line coverage. What a garbage study.
  • TFortunato 5y ago
    Nobody is arguing that usong a subset of tests won't catch all bugs. They are stating that having X% code coverage does not imply that you will be able to catch X% of bugs (at least, of the trivial type they introduced here) so you shouldn't use code coverage as a proxy for how good your test suite is. This may be obvious to you, but I've definitely seen teams fall into the trap of using code coverage as a primary measure of how good testing is.
• namelosw 5y ago
  Testing is an art that takes quite a while to master. People are able to write tests but many don't know "how" to write tests. Testing is extremely valuable but at the same time very easy to get burned. When testing done wrong (like abusing tests for coverage): 1. It makes the code too inflexible to refactor, leaving the system too rigid to grow. 2. It makes people frustrated on how to write tests in this project. 3. It makes people spend more time on writing tests than implementation. So some quick tips: 1. Only test the behavior from the user/consumer's perspective, do not test implementations. (Therefore the internal can be refactored without deleting tests.) 2. Be simple, clear, and consistent about how to write test on the project. If it's layered architecture, be clear about which layers need to be tested. And be minimal, two layers of testing is usually more than enough. Make sure the testing rules is enforced in the code base in the beginning, and then it will be downhills since people are mostly copying. 3. People always talk about the Testing Pyramid: the idea is coarse-grained tests are valuable but too costly while the fine-grained tests are cheap, so it's all trade-off and the optimal is to make them like a pyramid. But in reality, certain layers of tests could be drastically optimized. The idea is maximize the granularity on behaviors covered, while minimize the infrastructure involved (e.g. carefully designed monolith + hexagonal architecture), or make the infrastructure fast and parallelizable (e.g. SQL sandboxing). The end result is a certain layer could be a better trade-off than others, so concentrate tests there instead of writing tests repeatedly.
  • tpxl 5y ago
    > 1. Only test the behavior from the user/consumer's perspective, do not test implementations. (Therefore the internal can be refactored without deleting tests.) Sometimes you still need to test implementations. We had database writers/readers that automatically retried on failure/disconnect/failover/whatever and you can't test those by testing the API.
    • namelosw 5y ago
      Thanks for pointing it out. I was not clear that I was referring to the tests related to business requirements (which is the majority of the tests). Infrastructures could be treated like third-party modules as they could be tested like any open source project separately.
• hiram112 5y ago
  Testing is becoming less and less helpful in the "Enterprise" world. I don't even think it's possible in many cases. The typical CRUD apps out here in the contracting world - many in serious disrepair after a few different offshore or junior teams have worked on them - have been designed using way too many microservices and databases, then duct taped together with message queues and "Cloud" infrastructure like SNS or Lambda. Unit tests are pointless since 90% of your code does not fit the typical use case they'll teach you about in Bootcamp or in some TDD book - it's not like you're implementing some linked list implementation where you can write a nice, self-contained test suite that goes through a few valid inputs, edge cases, and proves things work. Instead, you're just assembling dozens of frameworks and managed services and 3rd party libraries, and wrapping it all up with a UI on the front. Integration tests used to be helpful. But nowadays, you'd need to recreate some huge Docker environment for each test, complete with several pre-loaded databases, half a dozen supporting microservices, and ensure you can connect to a complete test environment in the cloud for your managed infrastructure. Or you can do what's becoming increasingly common in the Java / Spring world where you have this elaborate system of mocks. Eventually, your mock classes will become more complex and elaborate than the original class itself. You'll spend 50% of your time trying to keep this integration environment working, with all the race conditions it entails as databases, queues, services sometimes start "out of order", and it won't even provide any benefit, because your mocks don't behave at all like the non-mocks do outside of tests. And don't get me started on front-end testing these days. What I see most amateur developers doing is just instantiating their little React components in some pseudo-browser environment and then testing that, yes, some internal class used by the component library is, in fact, also applied on their test component. Which means they're not testing anything relevant at all. And it also means all the tests break when we move to the next version of the component library, and all those internal classes are renamed or deprecated.
• whateveracct 5y ago
  Use the highlighted source code coverage report. It's especially good if the language calls out uncovered branches.
• wonnage 5y ago
  My personal experience working on frontend is that it's often fastest to get everything looking right in the browser, and then go back and write tests. There is a ton of manual testing involved (i.e, getting UI into all its possible states) that just goes out the window when writing unit tests because you can't easily test for stuff like the correct whitespace between elements. But then react-storybook and Happo came along (I'm sure the concepts have existed long before). I can actually commit the states I was trying to test to the repo (as stories), and CI can render them and take screenshot diffs with Happo; these are now the unit tests. So now the work I'm doing during development doubles as the tests as well, and there's no second pass required. My first exposure to this was really eye opening. All my career I've been getting preached to that I should personally write more tests, and never stopped to think how I could make "developing" and "testing" the same thing. I'm imagining a magical tool someday that records all the things I do during manual testing and turns them into unit tests.
  • porker 5y ago
    > But then react-storybook and Happo came along (I'm sure the concepts have existed long before). I can actually commit the states I was trying to test to the repo (as stories), and CI can render them and take screenshot diffs with Happo; these are now the unit tests. So now the work I'm doing during development doubles as the tests as well, and there's no second pass required. Ohhh. Thanks, I didn't know this was possible (the committing of states, screenshots I've done before). I'm still in the hell of manual-testing the front end on multiple browsers and resolutions to check that changes don't break anything.
• deleted 5y ago
  [deleted]
• tylerc230 5y ago
  I've found that architectures like redux which force the separation of logic and side effects (eg file writes or networking) allow me to write robust tests against the logic code without mocks or stubs. My approach is to write enough code that I have a good idea of what the interface to SUT should look like. Then I write the failing test and make it pass then write another test... The tests I write generally only change when the interface changes or when the requirements change. It doesn't make the code too rigid b/c I'm only writing tests against the interface of this big blob of code. I'm free to refactor away and my test will still run as long as I don't change the interface.