11 ms·

I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all

• mcv 9y ago
  If you absolutely want to cover all cases, you need to do mutation testing. A mutation testing system analyses your code, changes an operator (> becomes < or >= for example), and then runs your tests. If one test fails, your tests covered that statement. It seems to me you need to have serious OCD to go for 100% mutation coverage, but that is what you really need to do if perfect coverage is your aim. The other option is to accept that perfect coverage is not feasible, or possibly even a trap, and that you just need to cover the interesting bits.
  • jacques_chester 9y ago
    Strictly, mutation testing can't assure all cases either -- combinatorial state explosions are fun. But it does improve assurance a lot.
  • MereInterest 9y ago
    How would automated mutation testing handle the case of accidentally causing infinite loops, or invoking undefined behavior?
    • rjdavis3 9y ago
      I use pitest to run mutation coverage on most of my Java code bases. pitest implements a timeout to check for infinite loops introduced by changing the code. http://pitest.org/faq/ http://pitest.org/faq/
      • computerex 9y ago
        Pitest has solved the halting problem?
        • mcv 9y ago
          Timeouts are not a perfect solution to the halting problem, but usually good enough.
        • solatic 9y ago
          In the real world, you can't wait five minutes for the server to return a response. If the server returns the correct response after ten minutes, it's still wrong, unless the programmer has explicitly acknowledged that the procedure in question is a long-running procedure and has lengthened (but not eliminated) the time-out accordingly. The halting problem is an irrelevant, Ivory-tower distraction in production code.
  • mannykannot 9y ago
    There can be circumstances in which the substituted operator is just as valid as the operator being substituted for, resulting in false positives.
    • mcv 9y ago
      How would that be a false positive? If the substituted operator does not cause a test to fail, then your tests don't cover it. If the function of the program is not changed by changing the operator, then the operator does nothing and should be removed. That's the idea at least. All focus on high coverage, whether line coverage or mutation testing, creates an incentive to remove redundant robustness checks, and maybe that's not such a good idea after all. But that's a problem with all unit testing, and not just with mutation testing.
      • mannykannot 9y ago
        I was thinking that, for example, that a >= test is as valid as == in some cases. If == is actually used and is correct, substituting >= would not cause any valid test to fail. As I wrote this, however, I realized that if you substitute the inverse operator (!= for ==, < for >= etc.) and it is not caught, you can infer it is not covered. (edit: or that, in the context of the program as a whole, the result of the operation is irrelevant. I imagine this might legitimately happen, e.g. in the expansion of a macro or the instantiation of a template.)
        • mcv 9y ago
          Substituting == for >= should cause a test to fail. Either == is correct, or >=. They can't both be correct. Should the two things always be equal? Then they shouldn't be unequal. Is one allowed to be bigger than the other? Then that should be allowed, and not rejected. If this change doesn't cause a test to fail, you're not testing edge cases for this comparison. (Of course that's assuming you want perfect coverage.)
          • mannykannot 9y ago
• theparanoid 9y ago
  There's use in looking at the testing for industrial strength code e.g. compilers, DirectX, OpenGL conformance (god help you). With that background, I was confused by TDD to put it politely.
• pacaro 9y ago
  This was one of the fun things about working on Midori, the whole system was available to us, components all communicated through well defined interfaces, you really could (if necessary) mock the interfaces above and below and be sure that your logic was being exercised. The browser team, in particular, pushed hard to be close to 100% coverage. The overall number for the project was (IIRC) in the 85% range. When I'm writing tests, I'm not so concerned about what the tool tells me about which lines are covered, I like to work through the mental exercise of knowing which basis paths are covered. If a function has too many for me to reason about, that is a problem in itself. As an aside, while I'm rambling, all the examples in the article appeared to represent unnecessary abstraction, which is the opposite problem. If you have many methods in a class with only one basis path, what purpose does the class serve. These testing concerns may be the code smell that points to deeper problems
  • cbanek 9y ago
    > As an aside, while I'm rambling, all the examples in the article appeared to represent unnecessary abstraction, which is the opposite problem. One other thing about code coverage, unit testing, and other testing fads is I think they actively affect the architecture, and usually in the overthinking it way. Instead of having one tight bit of procedural code (which may have some state, or some dependency calls), people split it up into multiple classes, and then test each bit. This allows them to use the class architecture to mock things, but really has just multiplied out the amount of code. And in the end, you're running tests over the golden path probably even less. It's even possible to have 100% of the code covered, and not run the golden path, because you're always mocking out at least one bit.
    • sbov 9y ago
      It's strange to me that we continue to make languages that force us to make certain architectural choices to help facilitate testing.
      • cbanek 9y ago
        Indeed. It's almost as if testing should be built into the language itself. (I'm always a big fan of including self-test code in projects)
• fowl2 9y ago
  Which why I really love things like MSR's PEX[1]. "Pex uses a constraint solver to produce new test inputs which exercise different program behavior. " So this is not lines, but I guess.. branches and numerical limits. Either way it's cool and I wish it was integrated somewhere. They've got a online demo thing http://www.pexforfun.com/ http://www.pexforfun.com/ [1]https://www.microsoft.com/en-us/research/publication/pex-white-box-test-generation-for-net/ https://www.microsoft.com/en-us/research/publication/pex-whi...
  • tjalfi 9y ago
    IntelliTest[0] is the productized version of PEX. The first release was in Visual Studio 2015 Enterprise edition. [0] https://blogs.msdn.microsoft.com/visualstudio/2015/09/30/intellitest-for-net-test-more-with-less-effort/ https://blogs.msdn.microsoft.com/visualstudio/2015/09/30/int...
    • nuclx 9y ago
      Still a pity it's only included in Enterprise edition. We need a strong foundation of accessible tools for automated test input generation improving on the current state of the art fuzzers. See e.g. danluu's vision of "Combining AFL and QuickCheck for directed fuzzing" [https://danluu.com/testing/ https://danluu.com/testing/].
• ReligiousFlames 9y ago
  This is what I love about the design and operational philosophy of the Erlang/OTP evironment: the trickier class of harder-to-test/reproduce transient/timing pre-defensive-coding issues can sometimes be resolved automagically under supervision trees restarting said Erlang process. Log the stacktrace as a failure to maybe be fixed and move on. That's in addition to a robust soft-realtime, SMP VM which allows remote, live debugging.
  • d6e 9y ago
    As a newbie to Erlang, how does Erlang handle this case better for a service than a Python process which logs and restarts on failure? Also, this runtime testing strategy doesn't handle cases where you have other systems depending on that Erlang code path being successful, does it?.
• lobster_johnson 9y ago
  Code coverage should perhaps not be counted in lines, but in the number of type permutations covered. Then your challenge is determining all the types that _can_ be passed as inputs.
  • mannykannot 9y ago
    By types, I assume you mean categorically different types of data, rather than the types of the type system that your language recognizes?
• kasparsklavins 9y ago
  Tests are not meant for ensuring it works with all inputs. You cant simply just throw values at it hoping its all okay. To prove it works with all possible inputs, there are other tools at your disposal.
  • UweSchmidt 9y ago
    What tools would that be? I'd like to hear about real world test scenarios where all possible inputs are tested.
    • AstralStorm 9y ago
      Many embedded safety and medical applications prove correctness for all inputs and their combinations. Somme also verify error behaviour. (Out of range.) Granted, this is a relatively small input space, typically a few sensors.
    • Kubuxu 9y ago
      One of them would be a fuzz testing. You start with a predefined 'corpus' of inputs which are then modified by the fuzzer to reach yet not covered code paths. In the process it discovers many bugs and crashes. The input fuzzing process is rarely purely random. There are advanced techniques that allow the fuzzer to link input data to conditions of not covered branches. It is quite useful mechanism for checking inputs, formats, behaviour patterns (if you have two solutions but one model, one simple that works 100% but is slowish and one more complex but very fast). See: https://github.com/dvyukov/go-fuzz#trophies https://github.com/dvyukov/go-fuzz#trophies and http://lcamtuf.coredump.cx/afl/ http://lcamtuf.coredump.cx/afl/
      • kasparsklavins 9y ago
        Fuzzing is essentially the same as just throwing values at it hoping its all okay. What I meant was what @AstralStorm said about mathematical and logic proofs that it works for all defined values.
        • 9y ago
• ubernostrum 9y ago
  I only have one piece of code for which I can say true 100% coverage exists: a library that works with HTML/CSS color values, and which ships a test that generates all 16,777,216 hexadecimal and integer rgb() color values, and runs some functions with each value. However, I don't run that as part of the normal test suite. It only gets run when I'm prepping a new release, as a final verification step; the normal runs-every-commit test suite just exercises with a selection of values likely to expose obvious problems.
  • test1235 9y ago
    How long does that take?
    • ubernostrum 9y ago
      Depends on the machine. See the comments in the file: https://github.com/ubernostrum/webcolors/blob/master/tests/full_colors.py https://github.com/ubernostrum/webcolors/blob/master/tests/f... The fun part is people who criticize the generation of the integer triplets; yes, it's three nested loops and that's bad, but the total number of iterations will always be 16,777,216 no matter what algorithm you decide to use to generate them. So it uses nested loops since that's the most readable way to do it.
  • dahauns 9y ago
    Hmm...true 100% input coverage? Including the time domain? (Order dependency, Idempotence, etc...) :)
  • voidpointer 9y ago
    Just out of curiosity... What is the point in testing all 2^24 possible color values?
    • ubernostrum 9y ago
      The point is being certain I haven't missed an edge case somewhere. The hard part is the percentage rgb() values, of which there are technically an uncountably infinite number (since any real number in the range 0-100 is a legal percentage value). For those I generate all 16,777,216 integer values, and verify that converting to percentage and back yields the original value.
• bad_user 9y ago
  I don't see how that's a "tragedy", as in "an event causing great suffering, destruction, and distress, such as a serious accident, crime, or natural catastrophe". You're also making it sound like somebody promised you that tests can prove the absence of bugs, when that was never the bargain and smart people have already told you so, probably before you were born even. > Testing shows the presence, not the absence of bugs Dijkstra (1969)
  • StavrosK 9y ago
    If we're being pedantic, a tragedy is _a drama or literary work in which the main character is brought to ruin or suffers extreme sorrow, especially as a consequence of a tragic flaw, moral weakness, or inability to cope with unfavorable circumstances_. Definitely not a tragedy.
    • thehardsphere 9y ago
      If we're being pedantic, there's a second common definition of the word tragedy that does not refer to a drama or literary work, but "an event causing great suffering, destruction, and distress, such as a serious accident, crime, or natural catastrophe." I think when you take all of the time wasted on useless tests written merely for the sake of having tests, that waste is tragic. You could be doing anything else with that time.
      • StavrosK 9y ago
        I agree, I was just kicking GP's pedantry up a notch.
      • sqeaky 9y ago
        I am curious if there is any good way to measure that amount of waste. I agree that waste from extra and useless tests exists, but I am not convinced that waste is entirely bad. It seems to me that waste is unavoidable with teams newer to automated testing and may simply be part of the cost of using automated tests. If that is the case then it seems better to compare the cost of bugs with no automated to cost of superfluous tests. In that comparison extra tests definitely seems like the lesser of two evils, even without hard numbers. I would prefer hard numbers because my intuition could be wrong.
• gontard 9y ago
  I agree that test coverage does not ensure that the code is really tested. But not covered code is not tested code. So i use coverage in this way, to spot not tested code.
  • collyw 9y ago
    You can test stuff manually.
    • sqeaky 9y ago
      You really shouldn't, or if you do it should be an extreme minority of tests and automated tests should do the heavy lifting. Automated tests can do so much more than manual tests that shops still living by manual tests either have a damn good reason or are just as wrong as people who argued against Revision Control Systems (or people who argued for gotos instead of functions). Automated tests will be executed identically each time, so no missing test cases because someone slacked off or made a typo. Automated test can serve as examples for how to use the code. Automated tests can aid in porting to new platforms, once it builds you can find all the bugs your care about swiftly. Automated test can be integrated with documentation, tools like doxyegn and mdbook make this easier. Automated tests enable Continuous Integration. Are you familiar with Travis CI or Jenkins? If not, imagine a computer that a team commits their code to, instead of directly to the mainline Revision Control (Git master, svn head, etc...). That computer builds the software, runs all the tests, perhaps on every supported platform or in an environment very close to production, then only merges commits that appear to fully work. This doesn't completely eliminate bugs and broken builds, but the change is so large that teams without it are at a clear competitive disadvantage. When integrated into process tests can be used to protect code from changes. If a test exercises an API and the team knows that is the purpose then when they change things in a way that break the test they shouldn't... This sounds vague or obvious, but consider this: At Facebook and Google they have a rule that if it is not tested new code doesn't have to care if it breaks. Both companies have team that make broad Sweeping changes. Facebook wrote a new std::string and Google use clangtools to make automated changes in thousands of places at once. Even if code breaks or APIs change as long as tests pass these people can be sure that they negatively impacted the product and are following their team's rules. Automated Tests can... This list could go on for a very long time.
• gontard 9y ago
  I think that mutation testing should replace code coverage. It really ensure that the test verify the code behavior and not only invoked it.
  • AstralStorm 9y ago
    The trouble with mutation testing is that most mutations will completely break an application. Why not prove the code is correct instead? Should be much cheaper than 100% coverage and more certain.
    • 0hjc 9y ago
      Why do you say most mutations will completely break an application? This is certainly not my experience. Mutation testing systems normal use fairly stable operators (e.g changing a > to a >=). In most locations in the code changes such as these will have only a subtle effect.
• thaumasiotes 9y ago
  > even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases. On the other hand, if you do cover all your inputs, you've covered all the cases regardless of what % code coverage or path coverage you have.