3 ms·
Bugs all passed the tests sounds like survior bias. Like how all the fighter planes that returned with damage had damage to the wings. Point is, damage anywhere
by fyrn_ 3y ago
Bugs all passed the tests sounds like survior bias.
Like how all the fighter planes that returned with damage had damage to the wings.
Point is, damage anywhere else was fatal, the takeaway should not have been to armor the wings more, which was the first thing which was tried.
Likewise thinking about bugs this way discounts potential bugs that _were_ stopped by tests.
- WesolyKubeczek 3y agoIt's not like the planes that have returned. A bug reported in the field could well have been a crash. A costly, all-hands-on-deck incident kind of crash. And that bug still passed your own tests, compiled well, went into the production. Occam's razor tells me that it's more indicative of test harness being not comprehensive enough. Fix it to reproduce the bug/crash, then fix the code, then ship the fix, then tentatively pat yourself on the back for potential bugs that were stopped by the test. Until the next crash.
- fyrn_ 3y agoOh, I agree with that. I thought you were arguing that tests were less valuable because all production bugs passed the tests.