5 ms·
TDD is definitely helpful, but I don't ever use it as a metric for code fitness. At best it proves a limited number of inputs produce a limited number of expect
by covidthrow 6y ago
TDD is definitely helpful, but I don't ever use it as a metric for code fitness. At best it proves a limited number of inputs produce a limited number of expected outputs.
Consider, for example, a bug recently introduced during a refactor at my work:
The programmer optimized a conditional based on a regex by transforming it into a simple string compare. All the tests passed, code/branch coverage was good. Except that he missed that the regular expression tested case-insensitively, and our test suite didn't test upper and lower case scenarios.
This simple mistake outlines a few flaws:
- coverage was not robust, because it didn't take into consideration the branches inside the regex (which one could interpret as a form of macro expansion)
- the limited number of inputs used to test failed to capture the broader domain of possible input
- the test did not reflect a successful refactor
Obviously this is a fairly complicated example despite a pretty simple change, and several pieces had to fail in order for this change to fail. But it affirms that even basic changes to code aren't necessarily adequately covered by TDD.
Yes, the developer who forgot to test different letter cases made a mistake. Yes, regular expressions bring their own problems. But fundamentally, the result was that passing the tests did not affirm fitness of the change. Rather, it only proved a limited subset of conditions were error-free.
Effectively, tests are loaded with false negatives, so trusting them to identify problems should be done with a massive grain of salt.
If the developer had simply copy/pasted the code he changed and compared the two, then it's exceedingly more likely that he would have noticed that his code didn't capture the full breadth of conditions in the previous code.
A bug existed in the test suite, to start, and then a bug was introduced into the codebase. A human being looking at the two lines of code as it was rewritten probably would have noticed the regression. But even a code review missed it because it was a fundamentally small change among the other, more "make sure this looks good" code.
Personally, I very much value copy/paste/compare changes and never treat the test suite for anything other than "well we haven't broken anything in any exceedingly obvious ways." Maybe you're a superhuman programmer, but I'd lean more towards "you've probably added more bugs than you realize".
- deleted 6y ago[deleted]
- bfung 6y agoThere’s def. ways to write really good tests as well to avoid the limited input issue - write the test in a property-based test manner, like haskell’s quickcheck, and the test can catch entire classes of bugs. The caps vs. no caps issue would be easily caught using randomString() as the test case.