4 ms·

Your test suite seems broken (which is why test suites are hard to write in the first place). What if there is a bug when you have two bits changed -- one in th

Your test suite seems broken (which is why test suites are hard to write in the first place). What if there is a bug when you have two bits changed -- one in the 128th byte and one in the 256th byte? Here's a possible test suite that could catch those bugs. Take two files. Generate all possible combinations of byte-differences between the two files up to a length of 256 bytes and flag invalid comparisons. Now, what is the running time of this test case? I'll give you a hint: this test case is unlikely to finish before the heat-death of the universe. In general it's very hard to right comprehensive test suites and the only thing this proves is that Microsoft developers are humans, not gods. In fact, this is exactly the type of weird corner-case bug that I would expect to find in code written by good developers.

pzxc 16y ago
He didn't suggest that the test suite should account for all possible combinations of byte differences. He suggested that the test suite should account for all possible combinations of SINGLE byte differences. I.E., the test suite should have checked two files with only the first byte different, with only the second byte different, etc. Such a test suite would be linearly complex, not exponentially, and could easily be run before the heat death of anything. (However, to play my own devil's advocate, I'd have to say it's easy in retrospect to say "yes! there is an easy test for this that should have been written!" when in fact often the number of possible tests is astronomically large and it can be hard to pick the right ones. What if the bug was that FC.EXE didn't correctly register a difference when both the 127th and 128th bytes were the only differences? The proposed test suite would not have caught it.)
- Locke1689 16y ago
  Your devil's advocate argument seems to be just my post. I was trying to show that, while the parent's test suite was linear running time, the number of tests in a comprehensive test suite is exponential, therefore the runtime of any completely comprehensive test suite is exponential. Choosing the correct tests to use resources on is a very difficult problem.
pbhjpbhj 16y ago
>Your test suite seems broken (which is why test suites are hard to write in the first place). What if there is a bug when you have two bits changed I said "a test ... would include ...". This just seems the first most obvious pattern to check against. You need to be sure the program compares every bit, that just seems the simplest way to be sure that it's looping through every bit of both files and making a proper comparison. Knowing the internals of the program and functions used would give you the data length to check against. You seem to be saying that because a comprehensive test is impossible no test should have been performed. In any case wouldn't you see it in the ASCII read out of a watch routine or some such - hey look [made up variable] currentChunk has string terminators at the end which aren't in the chunks of the test file. You say good coders would miss this sort of bug, I'd think a good coder would realise the function they're using puts a string terminator in. Without seeing the code I guess we wouldn't know but direct comparison is surely one of the more basic operations to code (although I grant that doing it quickly maybe isn't). Surely the fundamental part is to XOR registers and look for 1s???

pzxc 16y ago

He didn't suggest that the test suite should account for all possible combinations of byte differences. He suggested that the test suite should account for all possible combinations of SINGLE byte differences. I.E., the test suite should have checked two files with only the first byte different, with only the second byte different, etc. Such a test suite would be linearly complex, not exponentially, and could easily be run before the heat death of anything. (However, to play my own devil's advocate, I'd have to say it's easy in retrospect to say "yes! there is an easy test for this that should have been written!" when in fact often the number of possible tests is astronomically large and it can be hard to pick the right ones. What if the bug was that FC.EXE didn't correctly register a difference when both the 127th and 128th bytes were the only differences? The proposed test suite would not have caught it.)

Locke1689 16y ago
Your devil's advocate argument seems to be just my post. I was trying to show that, while the parent's test suite was linear running time, the number of tests in a comprehensive test suite is exponential, therefore the runtime of any completely comprehensive test suite is exponential. Choosing the correct tests to use resources on is a very difficult problem.

pbhjpbhj 16y ago

>Your test suite seems broken (which is why test suites are hard to write in the first place). What if there is a bug when you have two bits changed I said "a test ... would include ...". This just seems the first most obvious pattern to check against. You need to be sure the program compares every bit, that just seems the simplest way to be sure that it's looping through every bit of both files and making a proper comparison. Knowing the internals of the program and functions used would give you the data length to check against. You seem to be saying that because a comprehensive test is impossible no test should have been performed. In any case wouldn't you see it in the ASCII read out of a watch routine or some such - hey look [made up variable] currentChunk has string terminators at the end which aren't in the chunks of the test file. You say good coders would miss this sort of bug, I'd think a good coder would realise the function they're using puts a string terminator in. Without seeing the code I guess we wouldn't know but direct comparison is surely one of the more basic operations to code (although I grant that doing it quickly maybe isn't). Surely the fundamental part is to XOR registers and look for 1s???