5 ms·

Isn't that the definition of branch testing, to test all possible branches within code and also testing the logic in all of those branches?

• gregmac 8y ago
  Consider a line like: value = 1 / input; You can get "100% coverage" if you test that with `input = 1`, but unless you check with `input = 0` you're missing a quite important logical check.
  • akvadrako 8y ago
    Not really - internally inputs 0 and 1 use different branches.
    • RussianCow 8y ago
      That's not a branch; otherwise you would have an infinite (or impossibly large) number of branches for just that one line of code. A branch is when you execute one set of code upon a given condition, and another if that condition is not met.
      • qznc 8y ago
        It depends on the language. In C it is not a branch because division by zero is undefined and not a path you consider. In Java you can argue that there are two branches. One branch that throws an exception and one that does not.
        • zaphar 8y ago
          No coverage reporting library will attempt to tell you that kind of coverage. You are essentially in violent agreement with the op but turning it into an argument by using different words for the same concept.
      • akvadrako 8y ago
        I didn't say every number is a different branch. But on many processors, divide by zero triggers an interrupt. That's semantically the same as a branch.
  • breckuh 8y ago
    Can't you just have a NonZeroInt type?
    • whyonearth 8y ago
      Only if your language supports dependent types.
      • krylon 8y ago
        In Ada, you can define integer types that only accept a given range of values.
      • tonyg 8y ago
        It's perfectly possible in languages with ordinary ADTs. data Nat = Z | S Nat data NonZeroNat = OnePlus Nat data NonZeroInt = Negative NonZeroNat | Positive NonZeroNat
        • jwatte 8y ago
          C isn't one of those languages, though.
          • tonyg 8y ago
            The analogous would be the Go-style represent-a-sum-badly-as-a-product, struct nonzero_t { int is_negative; unsigned int one_less_than_the_absolute_value; }; which, under interpretation, ranges from -(2^32) to -1 and +1 to +(2^32).
• gcc_programmer 8y ago
  You test all possible executions of your program when you test with all possible data inputs, which is infinite.
  • paulie_a 8y ago
    That is why I like using random data generators for tests. You can input some static data and then the rest is random. Every once in a while a bug pops out when you see a test fail that was previously passing.
    • regecks 8y ago
      SQLite is heavily subjected to fuzzing already ... maybe this vulnerability was discovered that way.
    • jcden 8y ago
      None of the above methods are used for testing. You use boundary testing, branch testing, equivalence partitioning etc. Random data is not a good method of testing.
      • jakevn 8y ago
        Except for the fact that it is exactly the method that has been used to discover a large number of critical bugs in the most popular OSS projects (including SQLite): http://lcamtuf.coredump.cx/afl/ http://lcamtuf.coredump.cx/afl/
        • Frizi 8y ago
          Fuzzing isn't really practical if all you do is just generate a totally random bit stream for input. There are many much more clever and robust strategies to hit as many edge cases as possible. Check AFL[1] for some details on generating smart random input files. You can also combine that with pretty advanced dynamic execution analysis to fuzz against unknown processor instruction sets, like in sandsifter[2]. [1]: http://lcamtuf.coredump.cx/afl/ http://lcamtuf.coredump.cx/afl/ [2]: https://github.com/xoreaxeaxeax/sandsifter https://github.com/xoreaxeaxeax/sandsifter
• thaumasiotes 8y ago
  Testing all possible branches (each considered individually) won't get you very far in terms of testing all possible logic flows. Consider a function which checks 5 simple if-statements in a row, always in the same order. Getting each branch means you tested 10 things. But there are 32 ways for 5 if-statements to jointly evaluate. If there is a logical dependency between the state checked by one if-statement and the state checked by another one, your perfect coverage may not pick up on that. If the if-statements might be checked in an arbitrary order... there are 120 ways to order 5 things. But you'll still get perfect branch coverage by checking 10 of them.
• archgoon 8y ago
  int returns_less_than_twelve(int input1, int input2) { int sum = 0; if (input >= 5) { sum += 5; } if (input2 >= 5) { sum += 8; } return sum; } The following test cases will pass and achieve 100% branch coverage. int x = returns_less_than_twelve(5,0); test_assert(x < 12); int y = returns_less_than_twelve(0,5); test_assert(x < 12); However, this does not cover the entire input space so returns_less_than_twelve(5,5); test_assert(x < 12); Will fail. However, branch coverage won't tell you that there's a hole in your test coverage. Generally however, writing full branch coverage will find a lot of issues, and also cause you to really think through how your code works; but still, it doesn't guarantee correctness. If you want that you need to start bringing tools that either exhaust your input space (a function which takes 5 booleans can be exhaustively tested for correctness in trivial amounts of time), or you start modeling your chosen language well enough that you can use a mathematical prover to demonstrate that your program or function is safe on all inputs. This of course requires you to come up with a definition of 'correct' or 'safe'. For the above program, it's clear how to define correctness. For things like "Don't let an unauthorized individual access this data or data that is derived from it in a way contrary to the desires of the owner of said data" it gets 'tricky' ;).