3 ms·

I know that's the idea, what I haven't seen is any real data showing that is a significant improvement in testing. You pick a few things, there are a lot more t

I know that's the idea, what I haven't seen is any real data showing that is a significant improvement in testing. You pick a few things, there are a lot more that can go wrong, off-by-one can come up in comparisons easily, shifting left or right can happen easily, assortment of bitwise operations can happen. And that's just integers. When you get into floats what are you going to plug in all common math constants as well as a range of values around each of those, various products and summations and factors of those things? No of course not. It's perfectly reasonable to simplify the test based on what the function is supposed to do. I'm not saying don't test any integer corner cases, I'm asking for the reason why the original poster says including those is good (and listing only 9 of possible 25 combinations to boot). In any case, why is that set of tests the golden one? That was just asserted as fact without any real reasoning I could see. If you tell me those 25 are the right set of tests, I can say you're wrong because 2 is also very common in computing so 2 and possibly -2 should be tested as well. My 49 tests are clearly superior. Then someone comes along and says well 10 is common in scientific or business logic, so better add 10 in there as well. Using the same arguments as you did to reach 25 we can now get to 81. And so on. Apply it to a function with 5 arguments and we're up to 60 thousand tests. Not reasonable.

mysterymath 5y ago
Consider the input as a big space of points, where each point is a possible input. These points aren't unrelated; speaking loosely, foo(3.000000000001) is usually more likely to behave like foo(3) than foo(4) is. The different behaviors on this space of points are due to the control flow and mathematical expressions within the function. You could thus model the function's behavior as a partition of the input: the different sets of inputs that will behave "identically" WRT the control flow inside the function. You could probably formalize this with a suitable notion of continuity, but I haven't bothered. Accordingly, testing corresponds to searching this space for problematic partitions. It's a bit like playing battleship: if you know a lot about the possible shapes of errors in this mathematical space, then you can choose a few small points that can totally prove their absence. Even if you don't know the shapes though, by choosing your points carefully, you can rule out shapes that are "sufficiently large". This is the impetus behind boundary testing; you don't prove that errors don't exist, but you can prove that they are relatively "pointlike"; that there aren't arbitrarily wide swaths of the input space that all exhibit the same erroneous behavior. Since the input space is usually vastly multidimensional, there's usually pretty hard limits on how much you can actually reduce the set of possible "problem shapes," though. A lot of in-practice software testing out there (I can only really speak about the work I've seen at Google) is statistical in nature: you run a huge amount of tee-d or saved production traffic through a service, and run a massive diff of the output, after painstakingly making your service deterministic. By doing this, you can show that a "typical" query is unlikely to cause a "catastrophic" failure.

mysterymath 5y ago

Consider the input as a big space of points, where each point is a possible input. These points aren't unrelated; speaking loosely, foo(3.000000000001) is usually more likely to behave like foo(3) than foo(4) is. The different behaviors on this space of points are due to the control flow and mathematical expressions within the function. You could thus model the function's behavior as a partition of the input: the different sets of inputs that will behave "identically" WRT the control flow inside the function. You could probably formalize this with a suitable notion of continuity, but I haven't bothered. Accordingly, testing corresponds to searching this space for problematic partitions. It's a bit like playing battleship: if you know a lot about the possible shapes of errors in this mathematical space, then you can choose a few small points that can totally prove their absence. Even if you don't know the shapes though, by choosing your points carefully, you can rule out shapes that are "sufficiently large". This is the impetus behind boundary testing; you don't prove that errors don't exist, but you can prove that they are relatively "pointlike"; that there aren't arbitrarily wide swaths of the input space that all exhibit the same erroneous behavior. Since the input space is usually vastly multidimensional, there's usually pretty hard limits on how much you can actually reduce the set of possible "problem shapes," though. A lot of in-practice software testing out there (I can only really speak about the work I've seen at Google) is statistical in nature: you run a huge amount of tee-d or saved production traffic through a service, and run a massive diff of the output, after painstakingly making your service deterministic. By doing this, you can show that a "typical" query is unlikely to cause a "catastrophic" failure.