4 ms·

Rich Hickey once said something in the midst of my “TypeScript, Haskell, la la la la la” phase: > I like to ask this question, what's true of every bug ever fo

Rich Hickey once said something in the midst of my “TypeScript, Haskell, la la la la la” phase: > I like to ask this question, what's true of every bug ever found in the field? (It got written?) Pff. It got written, yes. What's a more interesting fact? (pause) It passed the type checker! What else did it do? (the tests?) It passed all the tests. So now what do you do? The basic problem is that we really want tests to somehow specify the contract of the software, but when you are writing “given, when, then” the “givens” pin down too much of how it is done and the “when/then” is scoped to distinct sub-contracts of different parts so it pins down how the work was broken-down... It hits like a syllogism, right? All tests are software, all software has scope creep, scope creep in testing is contract creep.

fyrn_ 3y ago
Bugs all passed the tests sounds like survior bias. Like how all the fighter planes that returned with damage had damage to the wings. Point is, damage anywhere else was fatal, the takeaway should not have been to armor the wings more, which was the first thing which was tried. Likewise thinking about bugs this way discounts potential bugs that _were_ stopped by tests.
- WesolyKubeczek 3y ago
  It's not like the planes that have returned. A bug reported in the field could well have been a crash. A costly, all-hands-on-deck incident kind of crash. And that bug still passed your own tests, compiled well, went into the production. Occam's razor tells me that it's more indicative of test harness being not comprehensive enough. Fix it to reproduce the bug/crash, then fix the code, then ship the fix, then tentatively pat yourself on the back for potential bugs that were stopped by the test. Until the next crash.
  - fyrn_ 3y ago
    Oh, I agree with that. I thought you were arguing that tests were less valuable because all production bugs passed the tests.

fyrn_ 3y ago

Bugs all passed the tests sounds like survior bias. Like how all the fighter planes that returned with damage had damage to the wings. Point is, damage anywhere else was fatal, the takeaway should not have been to armor the wings more, which was the first thing which was tried. Likewise thinking about bugs this way discounts potential bugs that _were_ stopped by tests.

WesolyKubeczek 3y ago
It's not like the planes that have returned. A bug reported in the field could well have been a crash. A costly, all-hands-on-deck incident kind of crash. And that bug still passed your own tests, compiled well, went into the production. Occam's razor tells me that it's more indicative of test harness being not comprehensive enough. Fix it to reproduce the bug/crash, then fix the code, then ship the fix, then tentatively pat yourself on the back for potential bugs that were stopped by the test. Until the next crash.
- fyrn_ 3y ago
  Oh, I agree with that. I thought you were arguing that tests were less valuable because all production bugs passed the tests.