4 ms·
You don't know that the test code is free of bugs. You don't need to. Case 0: No bugs in the test code. All is well. Case 1: Bug in the test code that causes
by gjm11 8y ago
You don't know that the test code is free of bugs. You don't need to.
Case 0: No bugs in the test code. All is well.
Case 1: Bug in the test code that causes some bugs in the real code not to get caught. That's bad, but you're no worse off than if you didn't have the test at all.
Case 2: Bug in the test code that causes correct real code to look buggy. Result: the test fails, you look for problems, most likely you find that the problem is in the test code and fix it. Going forward, you have a working test.
Case 3: Bug in the test code that makes something else break. This can happen and is genuinely bad, but (1) it only affects testing, not your actual product, and (2) most bugs don't behave that way.
The test code is a net win if the bugs it catches in your real code are worth the effort of writing and debugging the test code. That's no less true on account of the possibility of bugs in the test code. It just means that when you estimate the benefit you have to be aware that sometimes the tests might be less effective because of bugs, and when you estimate the cost you have to be aware that you have to debug the code, not just write it the first time. And then you just ... decide what's the best tradeoff, just like everything else in engineering.
(And no, you don't need tests for your test code. The test code is a test for itself as well as for the code it's testing.)
- pintxo 8y agoI see 5 cases: Code | Test | Result -------------------------------------------------- fine | fine | a) We have a regression test, yay! fine | buggy | b) Someone breaks the code to make the test work. Oops. c) Someone fixes the test, we have a regression test, yay! buggy | fine | d) We'll fix the code, and we have a regression test, yay! buggy | buggy | e) Bug remains in code. Oops. If the probability of introducing an error is p, Code | Test | Probabilities | p = .01 (one in ten) -------------------------------------------------- fine | fine | a) (1-p) * (1-p) | .9801 fine | buggy | b) (1-p) * p * .5 | .00495 c) (1-p) * p * .5 | .00495 buggy | fine | d) p * (1-p) | .0099 buggy | buggy | e) p * p | .0001 So we see, probability for: no harm done .9801 (a) bugs found + fixed .01485 (c,d) bugs introduced / not found .00505 (b,e) The above completely ignores the fact, that depending on the code base there will be significantly more test code than production code. But then test code is quite often highly redundant, and might actually have a lower defect rate itself. Also the probability on introducing an error in the production code and the test code, might actually not be statistically independent, which I assumes here. So take with a grain of salt. [Edit] Actually d) could also end negatively. Guess a working model would have to take into account that on failing test cases, a sensible developer should take a step back and reason about why this happened. So the negative outcomes would be (hopefully) less likely than the positive ones here.
- peteradio 8y agoTop notch explanation here. P = 0.01 is 1 in 100 though right?
- pintxo 8y agoOops, yes.
- runald 8y agoI must have missed some steps, but why did you assign p = 0.01, and how did you arrive that a) get to be (1-p) * (1-p)? And why can't, say, a) be p * p?
- pintxo 8y agop is the probability of introducing in error into code. To get some actual numbers, I choose an error rate of 1 in 100 lines of code (0.01). This is totally subjective and probably larger then in reality, but it does not hurt to be pessimistic here. a) => the production code, nor the test code contains any bug. So if the probability for an error is `p` (.01), the probability for error free code `1 - p` (.99). If both events are independent, we can multiply them to get the probability of both happening at once `(1 - p) * (1 - p)`.