4 ms·
Let's say you've tested this thing 1 million times. Each time the output was automatically checked by five different and independently-developed test suites. Yo
by mgsouth 2y ago
Let's say you've tested this thing 1 million times. Each time the output was automatically checked by five different and independently-developed test suites. You're ready to swear there's no possible way for it to fail.
And then someone tries it with a ulimit of 16kB.
Does it run?
Do you _know_?
Do you even know what is correct behavior in this situation?
- adrianN 2y agoThe system it runs on is part of the specification. A program is correct if fulfills all specified requirements. You're saying a car is defective because it breaks when you put sugar in the tank.
- deleted 2y ago[deleted]
- odyssey7 2y agoThis is a no true Scotsman fallacy. Any time it runs incorrectly, it was invoked with the wrong operating conditions — which seem to be defined as any conditions that cause it to run incorrectly. Sugar in the tank is an agreeable example because of how obvious it is, but what about something more subtle? An odd condition that leads to the wrong resonant frequency. An unknown software bug that makes the brakes lock up on Tuesdays at 12:00 AM on one specific backroad in a remote part of Virginia. The combinatorial possibilities of operating conditions are too numerous to exhaustively test. I guess you could say that every function has every quality that it happens to have, so that functions need only exist in order to be “correct.”
- adrianN 2y agoYou typically write down the specification before claiming that your code is done so you can’t use it to claim that any undesired behavior is not a bug. I naturally agree that in general tests are insufficient to show correctness because the state space is of impractical size, but, as I said above, for certain programs they totally can be.