3 ms·

I'm not sure that it is reasonable. It doesn't scale up to anything but the most trivial functions. It also does not prove the function is "actually correct", t

I'm not sure that it is reasonable. It doesn't scale up to anything but the most trivial functions. It also does not prove the function is "actually correct", that is a fallacy. If you want that, you have to use formal. The problem is not this one particular function, it's transferring this methodology to something more complex. A reasonable set of tests that does not aim to prove correctness is quite entitled to assume a reasonable implementation. You wouldn't have to arbitrarily re-run all your test cases with values incremented by 42 just in case someone incorrectly subtracted 42 somewhere, for example.

falcolas 5y ago
You're testing for the characteristics of your function - inputs and their outputs. If your function takes strings, there's a well known set of strings that are typically known to cause issues. It's the same for every data type. And for every operation against/between those data types, there's a known set of issues. You can be pretty certain of a function's correctness (or more accurately, the correctness of a refactor - what unit tests are most useful at, in my opinion) without having to re-write the function in a slightly different way. But to get back to the topic at hand - code coverage is a pretty bad measure of program correctness. What I was really trying to show is how bad code coverage is at predicting correctness for even the shortest and simplest of functions.
- throwawaylinux 5y ago
  I know that's the idea, what I haven't seen is any real data showing that is a significant improvement in testing. You pick a few things, there are a lot more that can go wrong, off-by-one can come up in comparisons easily, shifting left or right can happen easily, assortment of bitwise operations can happen. And that's just integers. When you get into floats what are you going to plug in all common math constants as well as a range of values around each of those, various products and summations and factors of those things? No of course not. It's perfectly reasonable to simplify the test based on what the function is supposed to do. I'm not saying don't test any integer corner cases, I'm asking for the reason why the original poster says including those is good (and listing only 9 of possible 25 combinations to boot). In any case, why is that set of tests the golden one? That was just asserted as fact without any real reasoning I could see. If you tell me those 25 are the right set of tests, I can say you're wrong because 2 is also very common in computing so 2 and possibly -2 should be tested as well. My 49 tests are clearly superior. Then someone comes along and says well 10 is common in scientific or business logic, so better add 10 in there as well. Using the same arguments as you did to reach 25 we can now get to 81. And so on. Apply it to a function with 5 arguments and we're up to 60 thousand tests. Not reasonable.
  - mysterymath 5y ago
    Consider the input as a big space of points, where each point is a possible input. These points aren't unrelated; speaking loosely, foo(3.000000000001) is usually more likely to behave like foo(3) than foo(4) is. The different behaviors on this space of points are due to the control flow and mathematical expressions within the function. You could thus model the function's behavior as a partition of the input: the different sets of inputs that will behave "identically" WRT the control flow inside the function. You could probably formalize this with a suitable notion of continuity, but I haven't bothered. Accordingly, testing corresponds to searching this space for problematic partitions. It's a bit like playing battleship: if you know a lot about the possible shapes of errors in this mathematical space, then you can choose a few small points that can totally prove their absence. Even if you don't know the shapes though, by choosing your points carefully, you can rule out shapes that are "sufficiently large". This is the impetus behind boundary testing; you don't prove that errors don't exist, but you can prove that they are relatively "pointlike"; that there aren't arbitrarily wide swaths of the input space that all exhibit the same erroneous behavior. Since the input space is usually vastly multidimensional, there's usually pretty hard limits on how much you can actually reduce the set of possible "problem shapes," though. A lot of in-practice software testing out there (I can only really speak about the work I've seen at Google) is statistical in nature: you run a huge amount of tee-d or saved production traffic through a service, and run a massive diff of the output, after painstakingly making your service deterministic. By doing this, you can show that a "typical" query is unlikely to cause a "catastrophic" failure.

falcolas 5y ago

You're testing for the characteristics of your function - inputs and their outputs. If your function takes strings, there's a well known set of strings that are typically known to cause issues. It's the same for every data type. And for every operation against/between those data types, there's a known set of issues. You can be pretty certain of a function's correctness (or more accurately, the correctness of a refactor - what unit tests are most useful at, in my opinion) without having to re-write the function in a slightly different way. But to get back to the topic at hand - code coverage is a pretty bad measure of program correctness. What I was really trying to show is how bad code coverage is at predicting correctness for even the shortest and simplest of functions.

throwawaylinux 5y ago
I know that's the idea, what I haven't seen is any real data showing that is a significant improvement in testing. You pick a few things, there are a lot more that can go wrong, off-by-one can come up in comparisons easily, shifting left or right can happen easily, assortment of bitwise operations can happen. And that's just integers. When you get into floats what are you going to plug in all common math constants as well as a range of values around each of those, various products and summations and factors of those things? No of course not. It's perfectly reasonable to simplify the test based on what the function is supposed to do. I'm not saying don't test any integer corner cases, I'm asking for the reason why the original poster says including those is good (and listing only 9 of possible 25 combinations to boot). In any case, why is that set of tests the golden one? That was just asserted as fact without any real reasoning I could see. If you tell me those 25 are the right set of tests, I can say you're wrong because 2 is also very common in computing so 2 and possibly -2 should be tested as well. My 49 tests are clearly superior. Then someone comes along and says well 10 is common in scientific or business logic, so better add 10 in there as well. Using the same arguments as you did to reach 25 we can now get to 81. And so on. Apply it to a function with 5 arguments and we're up to 60 thousand tests. Not reasonable.
- mysterymath 5y ago
  Consider the input as a big space of points, where each point is a possible input. These points aren't unrelated; speaking loosely, foo(3.000000000001) is usually more likely to behave like foo(3) than foo(4) is. The different behaviors on this space of points are due to the control flow and mathematical expressions within the function. You could thus model the function's behavior as a partition of the input: the different sets of inputs that will behave "identically" WRT the control flow inside the function. You could probably formalize this with a suitable notion of continuity, but I haven't bothered. Accordingly, testing corresponds to searching this space for problematic partitions. It's a bit like playing battleship: if you know a lot about the possible shapes of errors in this mathematical space, then you can choose a few small points that can totally prove their absence. Even if you don't know the shapes though, by choosing your points carefully, you can rule out shapes that are "sufficiently large". This is the impetus behind boundary testing; you don't prove that errors don't exist, but you can prove that they are relatively "pointlike"; that there aren't arbitrarily wide swaths of the input space that all exhibit the same erroneous behavior. Since the input space is usually vastly multidimensional, there's usually pretty hard limits on how much you can actually reduce the set of possible "problem shapes," though. A lot of in-practice software testing out there (I can only really speak about the work I've seen at Google) is statistical in nature: you run a huge amount of tee-d or saved production traffic through a service, and run a massive diff of the output, after painstakingly making your service deterministic. By doing this, you can show that a "typical" query is unlikely to cause a "catastrophic" failure.