3 ms·

I'm not sure I understand your point -- For instance, if an algorithm failed only when exactly all three inputs were 0, then it would fail only in 1 per (2^32)

I'm not sure I understand your point -- For instance, if an algorithm failed only when exactly all three inputs were 0, then it would fail only in 1 per (2^32)^3, despite not being exactly 0 at its failure rate. So exhaustive testing only works if you have a single input, since 2^96 is somewhat harder to exhaustively search, and statistical testing may not be enough, since 1/2^96 << 1/2^32. We can imagine a situation where it fails with each of the values being a number besides 0, making this nontrivial to check for. (For example, if the inputs are normalized somehow before being processed.)

conistonwater 9y ago
I wanted to make the point that very few formulas fail on really special inputs in practice.
- SomeStupidPoint 9y ago
  Thanks for replying. I'm not sure I agree with that, but I'll meet you halfway with those errors rarely matter in practice. (There are only 2^33 people, so getting 2^96 events (or significant portion) is well, impractical. So failure on a single, random one is unlikely to matter.)
- brucedawson 9y ago
  When rewriting sin() for Xbox 360 I found that the old function gave the wrong result for denormals - off by a factor of two. Since double has an eleven bit exponent that means that roughly one number out of 2,048 would hit this special failure case. For round/ceil/floor a common failure case is the number immediately preceding 0.5. If you're "lucky" then your algorithm fails for lots of numbers ending in 0.5, but if it's just the one number then a random search will never find it. And, an exhaustive search over all of the doubles from 0.0 to 1.0 turns out to cover a really big search space (one quarter of all doubles, roughly).
  - dummyfunny 9y ago
    > "but if it's just the one number then a random search will never find it" Minor, but I think it's not never just very rare. If you're sampling from all floats then the problematic inputs should be possible. I think the distinction people aren't seeing is what are you testing. If it's the sin routine, then I think you're making valid points. But if we're talking about testing for correctness and the overall logic of the program or the abstract expression of the calculation, there are only a limited number of variables. Trying to divide them and vary them under this kind of control seems like a valuable tactic. It might not be sufficient but I've seen way more bugs in shitty if else boolean soups than I have from bits getting flipped. If the output is so sensitive to the value of a certain calculation there should probably be a secondary or checksum. Unless you're doing games where... Framerates.
- IanCal 9y ago
  The most common ones (for example, when something is 0 or a list is empty) are also typically prioritised in tools like quickcheck.

conistonwater 9y ago

I wanted to make the point that very few formulas fail on really special inputs in practice.

SomeStupidPoint 9y ago
Thanks for replying. I'm not sure I agree with that, but I'll meet you halfway with those errors rarely matter in practice. (There are only 2^33 people, so getting 2^96 events (or significant portion) is well, impractical. So failure on a single, random one is unlikely to matter.)
brucedawson 9y ago
When rewriting sin() for Xbox 360 I found that the old function gave the wrong result for denormals - off by a factor of two. Since double has an eleven bit exponent that means that roughly one number out of 2,048 would hit this special failure case. For round/ceil/floor a common failure case is the number immediately preceding 0.5. If you're "lucky" then your algorithm fails for lots of numbers ending in 0.5, but if it's just the one number then a random search will never find it. And, an exhaustive search over all of the doubles from 0.0 to 1.0 turns out to cover a really big search space (one quarter of all doubles, roughly).
- dummyfunny 9y ago
  > "but if it's just the one number then a random search will never find it" Minor, but I think it's not never just very rare. If you're sampling from all floats then the problematic inputs should be possible. I think the distinction people aren't seeing is what are you testing. If it's the sin routine, then I think you're making valid points. But if we're talking about testing for correctness and the overall logic of the program or the abstract expression of the calculation, there are only a limited number of variables. Trying to divide them and vary them under this kind of control seems like a valuable tactic. It might not be sufficient but I've seen way more bugs in shitty if else boolean soups than I have from bits getting flipped. If the output is so sensitive to the value of a certain calculation there should probably be a secondary or checksum. Unless you're doing games where... Framerates.
IanCal 9y ago
The most common ones (for example, when something is 0 or a list is empty) are also typically prioritised in tools like quickcheck.