4 ms·
The question that always comes to mind whenever testing is discussed is "how do you know the test code is itself free of bugs?" I distinctively remember once p
by userbinator 8y ago
The question that always comes to mind whenever testing is discussed is "how do you know the test code is itself free of bugs?"
I distinctively remember once posing that question in a meeting about testing, and a manager replying --- seriously --- with "then perhaps the test code should itself have tests." Someone else must've come up with that before too, because (at a different job) I've also worked on a codebase where a surprising number of tests were basically testing the function of another test.
- gjm11 8y agoYou don't know that the test code is free of bugs. You don't need to. Case 0: No bugs in the test code. All is well. Case 1: Bug in the test code that causes some bugs in the real code not to get caught. That's bad, but you're no worse off than if you didn't have the test at all. Case 2: Bug in the test code that causes correct real code to look buggy. Result: the test fails, you look for problems, most likely you find that the problem is in the test code and fix it. Going forward, you have a working test. Case 3: Bug in the test code that makes something else break. This can happen and is genuinely bad, but (1) it only affects testing, not your actual product, and (2) most bugs don't behave that way. The test code is a net win if the bugs it catches in your real code are worth the effort of writing and debugging the test code. That's no less true on account of the possibility of bugs in the test code. It just means that when you estimate the benefit you have to be aware that sometimes the tests might be less effective because of bugs, and when you estimate the cost you have to be aware that you have to debug the code, not just write it the first time. And then you just ... decide what's the best tradeoff, just like everything else in engineering. (And no, you don't need tests for your test code. The test code is a test for itself as well as for the code it's testing.)
- pintxo 8y agoI see 5 cases: Code | Test | Result -------------------------------------------------- fine | fine | a) We have a regression test, yay! fine | buggy | b) Someone breaks the code to make the test work. Oops. c) Someone fixes the test, we have a regression test, yay! buggy | fine | d) We'll fix the code, and we have a regression test, yay! buggy | buggy | e) Bug remains in code. Oops. If the probability of introducing an error is p, Code | Test | Probabilities | p = .01 (one in ten) -------------------------------------------------- fine | fine | a) (1-p) * (1-p) | .9801 fine | buggy | b) (1-p) * p * .5 | .00495 c) (1-p) * p * .5 | .00495 buggy | fine | d) p * (1-p) | .0099 buggy | buggy | e) p * p | .0001 So we see, probability for: no harm done .9801 (a) bugs found + fixed .01485 (c,d) bugs introduced / not found .00505 (b,e) The above completely ignores the fact, that depending on the code base there will be significantly more test code than production code. But then test code is quite often highly redundant, and might actually have a lower defect rate itself. Also the probability on introducing an error in the production code and the test code, might actually not be statistically independent, which I assumes here. So take with a grain of salt. [Edit] Actually d) could also end negatively. Guess a working model would have to take into account that on failing test cases, a sensible developer should take a step back and reason about why this happened. So the negative outcomes would be (hopefully) less likely than the positive ones here.
- peteradio 8y agoTop notch explanation here. P = 0.01 is 1 in 100 though right?
- pintxo 8y agoOops, yes.
- runald 8y agoI must have missed some steps, but why did you assign p = 0.01, and how did you arrive that a) get to be (1-p) * (1-p)? And why can't, say, a) be p * p?
- pintxo 8y agop is the probability of introducing in error into code. To get some actual numbers, I choose an error rate of 1 in 100 lines of code (0.01). This is totally subjective and probably larger then in reality, but it does not hurt to be pessimistic here. a) => the production code, nor the test code contains any bug. So if the probability for an error is `p` (.01), the probability for error free code `1 - p` (.99). If both events are independent, we can multiply them to get the probability of both happening at once `(1 - p) * (1 - p)`.
- zerogvt 8y agoAdd to this countless hours going into engineering tests that have to bent over backwards and mock the universe itself to test a tiny bit of logic that doesn't make sense to test in the first place. Then multiply with the hours getting lost trying to debug frail tests that break time-expensive continuous integration pipelines. Good unit and integration tests are a rarity. Instead tests like the above ones which actually are being developed like a side-project to the real project are the norm and they are bogging down the whole project. But you cannot deliver any code that is not "covered" because that would be against the current 100%-TDD bible/best practice/call it however you want. So the next dev/maintainer has to work out a badly written test (probably bearing untrue assumptions about the programs intended business logic) and after some fun hair pulling he does the reasonable thing which is working around the test or pampering the test to get it to pass. And that is how you end up with test code that is actually buggy and problematic in itself and does not really test much but it is increasing the test-coverage holly counter.