4 ms·
TDD doesn't really seem to be useful, at least, not as clearly useful as its proponents generally advocate. That is, it's generally advocated as a clear win, b
by eesmith 9y ago
TDD doesn't really seem to be useful, at least, not as clearly useful as its proponents generally advocate.
That is, it's generally advocated as a clear win, but of the various attempts to quantify it, it's a mix at best.
For example, https://www.computer.org/csdl/trans/ts/2013/06/tts2013060835.html https://www.computer.org/csdl/trans/ts/2013/06/tts2013060835... says "This paper provides a systematic meta-analysis of 27 studies that investigate the impact of Test-Driven Development (TDD) on external code quality and productivity. The results indicate that, in general, TDD has a small positive effect on quality but little to no discernible effect on productivity."
It then goes on about subgroup analysis, but we must take that with a grain of salt as subgroup analysis is more likely to have "statistically significant" correlations due to random fluctuations.
When I try TDD on my own projects, I find that I lock in the internal API too quickly, that is, I test against internal functions, because they are easy to test. Then when I refactor, I find I avoid refactoring which might, for example, merge a couple of functions into one or otherwise change that internal API, because of the effort of rewriting the tests to meet the new API.
I work best with "Spike and Stabilize", that is, a spike solution, flesh it out, then once the API is mostly finished, develop the tests. I also use coverage testing to help identify which tests are missing.
Side note: I hate it when TDD people say that TDD code naturally has 100% coverage. It doesn't. At a process level, the refactor can include "replace algorithm", but the new algorithm may have different edge cases than the original algorithm, so require additional tests to ensure correctness.
For example, replace a simple quicksort with timsort, because it's possible to build a test case which causes quicksort to hit quadratic worst-case performance. But if the input has 63 elements or fewer then only the insertion sort part of timsort will be tested. You need a very different test suite to really exercise timsort, compared to what's needed to test a simple quicksort.
Or for a simpler example, performance analysis may say that the case n=1 takes 90% of the time, so rather than use the slower general-purpose algorithm, the new strategy is to special-case n=1. However, deeper in the code there was already special support for n=1, which is now no longer used. There's no way TDD can identify this dead code.
Has there been any analysis of coverage analysis on TDD-based projects, where coverage was never evaluated before, to see what the coverage actually is?
- lancerkind 9y agoRegarding coverage comment, your right in that code can contain capabilities (or bugs) beyond the TDD coded conceived. (If they weren't conceived then it could be a bug or things that don't happen in practical operations.) The TDD people are right in that if you follow the TDD process, all your code comes up as covered by a code coverage tool. But this is only coverage. Not all possible functionality coverage. I imagine "all possible functionality coverage" is an intractable problem in the same class as the Halting Problem. Branch/line coverage should be near 100% for TDD code. (Some I/O coupling areas are intentionally saved for other types testing as they aren't unit testable.) It's an easy thing to measure. It's a common industry metric. It's not the whole picture but it is something. Measuring code coverage on TDD created software that was built without monitoring the code coverage, then later use code coverage at the end as an evaluation: I've only a few anecdotal (two) instances where I did this and I found the code coverage to be well above 95% for the code base and at 100% for modules that were pure logic. In one of those instances, after many rounds of defect injection, I discovered one or two locations where another unit test could be added. All in all, I've been pretty happy with TDD. But who cares? I want to hear from the industry why others aren't doing it. :-) Thank you eesmith, I enjoyed the discussion! How about hearing some more from others? What's stopping you from doing TDD? Is it because you don't believe it? If you don't believe it, how was it presented to you? Did you try it?
- eesmith 9y ago"your right in that code can contain capabilities (or bugs) beyond the TDD coded conceived" I do not understand what you mean by my being right. I did not say that. As you wrote it, it's a trivially true statement, and I didn't think what I wrote was so trite. I will try to reword it. In the "Red - Green - Refactor" style of TDD you are allowed to refactor the code so long as the tests remain green. This assumes there are sufficient tests to catch errors in the refactoring. However, one of the allowed refactorings is "Substitute Algorithm"[0]. The new algorithm may have have corner cases which the old algorithm did not have, and so wasn't originally tested. R-G-R omits the part where you add new unit tests which are expected to be green, but added in order to verify that the new algorithm really does work correctly for new corner cases. It should be "Red-Green-Refactor-Yellow-Green", where "Yellow" means "do the failure analysis to ensure that the refactoring didn't add new failure cases." Yet TDD people don't teach that. And I think the reason why is that this "Yellow" stage is the normal part of non-TDD programming. If you drop "Red-Green" and just iterate over "Code-Green-Yellow-Green" (write the code, make sure the old tests work, do the failure analysis, and make the new tests work), you'll still end up with good quality code. This comes back to another aspect of TDD. Suppose that the programmers in a company are in their 20s and 30s while middle management is in their 50s. There is an asymmetry as middle management has much more experience in applying pressure than the younger programmers. Managers, after all, have more experience trying to manipulate people. Some programmers in this situation use TDD as a tactic to resist external pressure to release code with insufficient testing. "The code's done, but it isn't fully tested." "It's done? Ship it, and we'll deal with the problems later." With TDD this becomes "we're using TDD because <famous people> say it's a best industry practice". This gives the programmers a way to externalize the resistance to releasing poorly tested software. However, it's also unprofessional. Some code can be lightly tested, perhaps manually tested instead of a unit test suite. TDD shuts off the dialog about how to make those trade-offs at the business level, in the name of either technical purity or an unwillingness or inability to really negotiate. "if you follow the TDD process, all your code comes up as covered by a code coverage tool" No, that is incorrect. I have given two examples of how TDD code can end up with less than 100% statement coverage, much less 100% branch coverage. Ken Beck wrote "TDD followed religiously should result in 100% statement coverage." You backed that off a little with "should be near 100%". But my point is that these are assertions of faith, which TDD proponents repeat without evidence. If they repeat this without evidence, what else are they repeating without evidence? I myself have written non-TDD code, then tests, then verified code coverage, and ended with 98% coverage. (I wasn't able to test malloc() failures, though I coded in handlers for them and manually exercised those code paths to make sure they worked.) [0] As Fowler originally described it, "Substitute Algorithm" is used "to replace an algorithm with one that is clearer." However, if a new test appears to hang, and analysis shows it will take several days to finish, then you may need to use a more complex algorithm with better scaling performance. If this isn't a refactoring, then what is it? Others, like https://refactoring.guru/substitute-algorithm https://refactoring.guru/substitute-algorithm , don't require that the new algorithm be simpler.