3 ms·

Not to disagree with the idea that you generally need more test cases than control flow paths to really test correctness well. Just a question about your exampl

Not to disagree with the idea that you generally need more test cases than control flow paths to really test correctness well. Just a question about your example -- let's say your requirement is a function that does this: void fn(int a, int b) { if (a == b) printf("equal"); else printf("not); } What are the 3 test cases you would write? What are the 9? fn(1, 1) -> "equal" fn(1, 0) -> "not" What more useful tests are there here? Other than just picking arbitrary numbers, maxint/minint/1/0/-1/etc are not significant edge cases here.

fiddlerwoaroof 5y ago
It’s not a problem here, but if the type doesn’t guarantee reference equality for equal value, you’d also need a test to differentiate between a == b And (in Java) a.equals(b) In order to catch misguided refactorings. (I’ve introduced bugs this way before I learned that == can give a false positive for strings if the strings are interned)
- merb 5y ago
  in java the method that takes two `int`s would still use ==. java has value types which don't need `equals`.
  - fiddlerwoaroof 5y ago
    Yeah, that’s why I said “it’s not a problem here”.
falcolas 5y ago
Let's say I mistype 'a == b' as 'a <= b' when writing fn (the purpose of tests being to validate that the code is correct, after all). The test values of fn(1, 1), and fn(1, 0) will still display "equal" and "not", even when the function is obviously incorrect. Let's add 1 to a before the equality comparison in order to meet a new business requirement. fn(1, 1) still works, but does fn(maxint, maxint)? Or does it suddenly throw an exception (or worse, silently roll over to minint)?
- throwawaylinux 5y ago
  Right but you could also accidentally mistype it as 'b == 1'. Or 'a == b*b'. Or '(a == 0 || a == 1 || a == maxint) && a == b'. I'm not saying more tests can't catch specific bugs you might come up with, I'm asking how you can choose numbers that have fundamental edge cases for this specific requirement without looking at the actual implementation. I don't think you really can. maxint/minint/0/-1/1 are generally common choices but only because they do tend to catch signdness errors or inequalities or division by zero or whatever, they aren't really fundamental to the requirement of this function though. If we just wanted to blindly blast those numbers in, it would be 25 individual test cases for all permutations. Doesn't seem to be reasonable.
  - falcolas 5y ago
    Is it reasonable? It depends on your goals. Do you want to ensure that, for the set of well known edge cases, your function is correct? Proving the function is correct for the known edge cases for integers and integer comparison is not excessively hard. Let's go for the worst case of edge cases, and: - Pick the input pairs that are problems with ints: the set of (minint, maxint). - Pick the input pairs that are problems with comparisons: the set of (-1, 0, 1). - Pick input pairs that show that it's correct where a < b, a = b, and a > b. That last set of cases is nicely covered by the previous two. That's a total of 25 input pairs(5^2). In my opinion, that's not an unreasonable number of test cases to test for correctness. Why do I think it's reasonable? Because while it's technically 25 test cases, it's really just one test that runs through an array of input values in a loop. Really simple, and adding more test cases as the business logic changes is equally simple. And let's be honest, given how quickly computers work these days, even testing all ints for both inputs isn't completely unreasonable (though I wouldn't make it a part of a unit test suite). Especially if your logic is more complicated than a simple equality test (which most function logic will be). EDIT: "Accidentally mistype" 'a==b' as '(a == 0 || a == 1 || a == maxint) && a == b' Love it.
    - throwawaylinux 5y ago
      I'm not sure that it is reasonable. It doesn't scale up to anything but the most trivial functions. It also does not prove the function is "actually correct", that is a fallacy. If you want that, you have to use formal. The problem is not this one particular function, it's transferring this methodology to something more complex. A reasonable set of tests that does not aim to prove correctness is quite entitled to assume a reasonable implementation. You wouldn't have to arbitrarily re-run all your test cases with values incremented by 42 just in case someone incorrectly subtracted 42 somewhere, for example.
deleted 5y ago
[deleted]
Supermancho 5y ago
I wouldn't even write this way because the else confuses what's happening. void fn(int a, int b) { string ret = "not"; if (a == b) ret = "equal"; printf(ret); } *I understand, this is not a good optimization in every language.
- ummonk 5y ago
  Your code is far less clear to me than the above code. Though I'd probably write: void fn(int a, int b) { printf(a == b ? "equal" : "not"); }
- throwawaylinux 5y ago
  How does the 'else' confuse what is happening?
  - Supermancho 5y ago
    The function will perform a printf, so nesting it in an if-else subconsciously implies it's optional. This is a quirk of human text comprehension. In a small program, it seems arbitrary because you can see it all at once. Make the branches do more than simply printf and you can see why you want to frontload the else, rather than pick through the if/else to see what always happens.
    - throwawaylinux 5y ago
      I can kind of see what you mean, but if you make branches do more then it's also quite possible the common actions become more tedious to factor into common code. And I don't think it's good practice to read control statements like that (obviously indented loops can't be assumed optional either). int pow2(int v, int pow) { int result; if (pow >= 0) result = v << pow; else result = v >> -pow; return result; } Looking at this, both sides of the if/else perform a multiply by 2^pow, you can't take the operation to possibly not happen. So I agree factoring common operations can be a good thing, but I think it's really a case by case basis and my code is fine. I would also caution against assuming something may not happen elsewhere in a function just because you see it inside a conditional block. This happens frequently to various degrees and I've never encountered a style policy preventing it or any compiler or linter warnings for it.

fiddlerwoaroof 5y ago

It’s not a problem here, but if the type doesn’t guarantee reference equality for equal value, you’d also need a test to differentiate between a == b And (in Java) a.equals(b) In order to catch misguided refactorings. (I’ve introduced bugs this way before I learned that == can give a false positive for strings if the strings are interned)

merb 5y ago
in java the method that takes two `int`s would still use ==. java has value types which don't need `equals`.
- fiddlerwoaroof 5y ago
  Yeah, that’s why I said “it’s not a problem here”.

falcolas 5y ago

Let's say I mistype 'a == b' as 'a <= b' when writing fn (the purpose of tests being to validate that the code is correct, after all). The test values of fn(1, 1), and fn(1, 0) will still display "equal" and "not", even when the function is obviously incorrect. Let's add 1 to a before the equality comparison in order to meet a new business requirement. fn(1, 1) still works, but does fn(maxint, maxint)? Or does it suddenly throw an exception (or worse, silently roll over to minint)?

throwawaylinux 5y ago
Right but you could also accidentally mistype it as 'b == 1'. Or 'a == b*b'. Or '(a == 0 || a == 1 || a == maxint) && a == b'. I'm not saying more tests can't catch specific bugs you might come up with, I'm asking how you can choose numbers that have fundamental edge cases for this specific requirement without looking at the actual implementation. I don't think you really can. maxint/minint/0/-1/1 are generally common choices but only because they do tend to catch signdness errors or inequalities or division by zero or whatever, they aren't really fundamental to the requirement of this function though. If we just wanted to blindly blast those numbers in, it would be 25 individual test cases for all permutations. Doesn't seem to be reasonable.
- falcolas 5y ago
  Is it reasonable? It depends on your goals. Do you want to ensure that, for the set of well known edge cases, your function is correct? Proving the function is correct for the known edge cases for integers and integer comparison is not excessively hard. Let's go for the worst case of edge cases, and: - Pick the input pairs that are problems with ints: the set of (minint, maxint). - Pick the input pairs that are problems with comparisons: the set of (-1, 0, 1). - Pick input pairs that show that it's correct where a < b, a = b, and a > b. That last set of cases is nicely covered by the previous two. That's a total of 25 input pairs(5^2). In my opinion, that's not an unreasonable number of test cases to test for correctness. Why do I think it's reasonable? Because while it's technically 25 test cases, it's really just one test that runs through an array of input values in a loop. Really simple, and adding more test cases as the business logic changes is equally simple. And let's be honest, given how quickly computers work these days, even testing all ints for both inputs isn't completely unreasonable (though I wouldn't make it a part of a unit test suite). Especially if your logic is more complicated than a simple equality test (which most function logic will be). EDIT: "Accidentally mistype" 'a==b' as '(a == 0 || a == 1 || a == maxint) && a == b' Love it.
  - throwawaylinux 5y ago
    I'm not sure that it is reasonable. It doesn't scale up to anything but the most trivial functions. It also does not prove the function is "actually correct", that is a fallacy. If you want that, you have to use formal. The problem is not this one particular function, it's transferring this methodology to something more complex. A reasonable set of tests that does not aim to prove correctness is quite entitled to assume a reasonable implementation. You wouldn't have to arbitrarily re-run all your test cases with values incremented by 42 just in case someone incorrectly subtracted 42 somewhere, for example.

deleted 5y ago

[deleted]

Supermancho 5y ago

I wouldn't even write this way because the else confuses what's happening. void fn(int a, int b) { string ret = "not"; if (a == b) ret = "equal"; printf(ret); } *I understand, this is not a good optimization in every language.

ummonk 5y ago
Your code is far less clear to me than the above code. Though I'd probably write: void fn(int a, int b) { printf(a == b ? "equal" : "not"); }
throwawaylinux 5y ago
How does the 'else' confuse what is happening?
- Supermancho 5y ago
  The function will perform a printf, so nesting it in an if-else subconsciously implies it's optional. This is a quirk of human text comprehension. In a small program, it seems arbitrary because you can see it all at once. Make the branches do more than simply printf and you can see why you want to frontload the else, rather than pick through the if/else to see what always happens.
  - throwawaylinux 5y ago
    I can kind of see what you mean, but if you make branches do more then it's also quite possible the common actions become more tedious to factor into common code. And I don't think it's good practice to read control statements like that (obviously indented loops can't be assumed optional either). int pow2(int v, int pow) { int result; if (pow >= 0) result = v << pow; else result = v >> -pow; return result; } Looking at this, both sides of the if/else perform a multiply by 2^pow, you can't take the operation to possibly not happen. So I agree factoring common operations can be a good thing, but I think it's really a case by case basis and my code is fine. I would also caution against assuming something may not happen elsewhere in a function just because you see it inside a conditional block. This happens frequently to various degrees and I've never encountered a style policy preventing it or any compiler or linter warnings for it.