4 ms·

Right but you could also accidentally mistype it as 'b == 1'. Or 'a == b*b'. Or '(a == 0 || a == 1 || a == maxint) && a == b'. I'm not saying more tests can't

Right but you could also accidentally mistype it as 'b == 1'. Or 'a == b*b'. Or '(a == 0 || a == 1 || a == maxint) && a == b'. I'm not saying more tests can't catch specific bugs you might come up with, I'm asking how you can choose numbers that have fundamental edge cases for this specific requirement without looking at the actual implementation. I don't think you really can. maxint/minint/0/-1/1 are generally common choices but only because they do tend to catch signdness errors or inequalities or division by zero or whatever, they aren't really fundamental to the requirement of this function though. If we just wanted to blindly blast those numbers in, it would be 25 individual test cases for all permutations. Doesn't seem to be reasonable.

falcolas 5y ago
Is it reasonable? It depends on your goals. Do you want to ensure that, for the set of well known edge cases, your function is correct? Proving the function is correct for the known edge cases for integers and integer comparison is not excessively hard. Let's go for the worst case of edge cases, and: - Pick the input pairs that are problems with ints: the set of (minint, maxint). - Pick the input pairs that are problems with comparisons: the set of (-1, 0, 1). - Pick input pairs that show that it's correct where a < b, a = b, and a > b. That last set of cases is nicely covered by the previous two. That's a total of 25 input pairs(5^2). In my opinion, that's not an unreasonable number of test cases to test for correctness. Why do I think it's reasonable? Because while it's technically 25 test cases, it's really just one test that runs through an array of input values in a loop. Really simple, and adding more test cases as the business logic changes is equally simple. And let's be honest, given how quickly computers work these days, even testing all ints for both inputs isn't completely unreasonable (though I wouldn't make it a part of a unit test suite). Especially if your logic is more complicated than a simple equality test (which most function logic will be). EDIT: "Accidentally mistype" 'a==b' as '(a == 0 || a == 1 || a == maxint) && a == b' Love it.
- throwawaylinux 5y ago
  I'm not sure that it is reasonable. It doesn't scale up to anything but the most trivial functions. It also does not prove the function is "actually correct", that is a fallacy. If you want that, you have to use formal. The problem is not this one particular function, it's transferring this methodology to something more complex. A reasonable set of tests that does not aim to prove correctness is quite entitled to assume a reasonable implementation. You wouldn't have to arbitrarily re-run all your test cases with values incremented by 42 just in case someone incorrectly subtracted 42 somewhere, for example.
  - falcolas 5y ago
    You're testing for the characteristics of your function - inputs and their outputs. If your function takes strings, there's a well known set of strings that are typically known to cause issues. It's the same for every data type. And for every operation against/between those data types, there's a known set of issues. You can be pretty certain of a function's correctness (or more accurately, the correctness of a refactor - what unit tests are most useful at, in my opinion) without having to re-write the function in a slightly different way. But to get back to the topic at hand - code coverage is a pretty bad measure of program correctness. What I was really trying to show is how bad code coverage is at predicting correctness for even the shortest and simplest of functions.
    - throwawaylinux 5y ago
      I know that's the idea, what I haven't seen is any real data showing that is a significant improvement in testing. You pick a few things, there are a lot more that can go wrong, off-by-one can come up in comparisons easily, shifting left or right can happen easily, assortment of bitwise operations can happen. And that's just integers. When you get into floats what are you going to plug in all common math constants as well as a range of values around each of those, various products and summations and factors of those things? No of course not. It's perfectly reasonable to simplify the test based on what the function is supposed to do. I'm not saying don't test any integer corner cases, I'm asking for the reason why the original poster says including those is good (and listing only 9 of possible 25 combinations to boot). In any case, why is that set of tests the golden one? That was just asserted as fact without any real reasoning I could see. If you tell me those 25 are the right set of tests, I can say you're wrong because 2 is also very common in computing so 2 and possibly -2 should be tested as well. My 49 tests are clearly superior. Then someone comes along and says well 10 is common in scientific or business logic, so better add 10 in there as well. Using the same arguments as you did to reach 25 we can now get to 81. And so on. Apply it to a function with 5 arguments and we're up to 60 thousand tests. Not reasonable.
amw-zero 5y ago
Now you understand - testing is extremely close to useless. You can't perfectly choose the test cases for code without looking at its implementation. And by that point, you are just testing the implementation and working backwards. What you want is a specification of the behavior, and to verify that the implementation satisfies the specification. This will be proven in the general case, and then you don't need to think about individual cases.
- throwawaylinux 5y ago
  I don't think testing is close to useless, I think it has a lot of value but trying to prove general correctness with your test cases is a fool's errand. Adding test cases intelligently for tricky cases in the specification and/or implementation, hard-to-hit error conditions, or adding tests for bugs you previously fixed, etc is perfectly reasonable without doggedly chasing 100% coverage or 100% correctness. And yes I'm aware of formal methods, and agree they seem like the better way to go for proving.

falcolas 5y ago

Is it reasonable? It depends on your goals. Do you want to ensure that, for the set of well known edge cases, your function is correct? Proving the function is correct for the known edge cases for integers and integer comparison is not excessively hard. Let's go for the worst case of edge cases, and: - Pick the input pairs that are problems with ints: the set of (minint, maxint). - Pick the input pairs that are problems with comparisons: the set of (-1, 0, 1). - Pick input pairs that show that it's correct where a < b, a = b, and a > b. That last set of cases is nicely covered by the previous two. That's a total of 25 input pairs(5^2). In my opinion, that's not an unreasonable number of test cases to test for correctness. Why do I think it's reasonable? Because while it's technically 25 test cases, it's really just one test that runs through an array of input values in a loop. Really simple, and adding more test cases as the business logic changes is equally simple. And let's be honest, given how quickly computers work these days, even testing all ints for both inputs isn't completely unreasonable (though I wouldn't make it a part of a unit test suite). Especially if your logic is more complicated than a simple equality test (which most function logic will be). EDIT: "Accidentally mistype" 'a==b' as '(a == 0 || a == 1 || a == maxint) && a == b' Love it.

throwawaylinux 5y ago
I'm not sure that it is reasonable. It doesn't scale up to anything but the most trivial functions. It also does not prove the function is "actually correct", that is a fallacy. If you want that, you have to use formal. The problem is not this one particular function, it's transferring this methodology to something more complex. A reasonable set of tests that does not aim to prove correctness is quite entitled to assume a reasonable implementation. You wouldn't have to arbitrarily re-run all your test cases with values incremented by 42 just in case someone incorrectly subtracted 42 somewhere, for example.
- falcolas 5y ago
  You're testing for the characteristics of your function - inputs and their outputs. If your function takes strings, there's a well known set of strings that are typically known to cause issues. It's the same for every data type. And for every operation against/between those data types, there's a known set of issues. You can be pretty certain of a function's correctness (or more accurately, the correctness of a refactor - what unit tests are most useful at, in my opinion) without having to re-write the function in a slightly different way. But to get back to the topic at hand - code coverage is a pretty bad measure of program correctness. What I was really trying to show is how bad code coverage is at predicting correctness for even the shortest and simplest of functions.
  - throwawaylinux 5y ago
    I know that's the idea, what I haven't seen is any real data showing that is a significant improvement in testing. You pick a few things, there are a lot more that can go wrong, off-by-one can come up in comparisons easily, shifting left or right can happen easily, assortment of bitwise operations can happen. And that's just integers. When you get into floats what are you going to plug in all common math constants as well as a range of values around each of those, various products and summations and factors of those things? No of course not. It's perfectly reasonable to simplify the test based on what the function is supposed to do. I'm not saying don't test any integer corner cases, I'm asking for the reason why the original poster says including those is good (and listing only 9 of possible 25 combinations to boot). In any case, why is that set of tests the golden one? That was just asserted as fact without any real reasoning I could see. If you tell me those 25 are the right set of tests, I can say you're wrong because 2 is also very common in computing so 2 and possibly -2 should be tested as well. My 49 tests are clearly superior. Then someone comes along and says well 10 is common in scientific or business logic, so better add 10 in there as well. Using the same arguments as you did to reach 25 we can now get to 81. And so on. Apply it to a function with 5 arguments and we're up to 60 thousand tests. Not reasonable.

amw-zero 5y ago

Now you understand - testing is extremely close to useless. You can't perfectly choose the test cases for code without looking at its implementation. And by that point, you are just testing the implementation and working backwards. What you want is a specification of the behavior, and to verify that the implementation satisfies the specification. This will be proven in the general case, and then you don't need to think about individual cases.

throwawaylinux 5y ago
I don't think testing is close to useless, I think it has a lot of value but trying to prove general correctness with your test cases is a fool's errand. Adding test cases intelligently for tricky cases in the specification and/or implementation, hard-to-hit error conditions, or adding tests for bugs you previously fixed, etc is perfectly reasonable without doggedly chasing 100% coverage or 100% correctness. And yes I'm aware of formal methods, and agree they seem like the better way to go for proving.