I read "An Initial Investigation of Test Driven Development in Industry" by Boby George and Laurie Williams, and "Realizing quality improvement through test dr

I read "An Initial Investigation of Test Driven Development in Industry" by Boby George and Laurie Williams, and "Realizing quality improvement through test driven development: results and experiences of four industrial teams" by Naggapan et al. and I am not too impressed. The first study has 6 TDD teams and 6 control teams. Our null hypothesis is that these teams produce exactly the same quality of code, measured in defect rate. Now, any sane study would take a look into (statistical) power, ie the probability the test will reject the null hypothesis. If we further look at the box-plots, we see that the overlap between the "treatment" and control group samples is pretty big. Hence, we will need to test for any statistic significance before coming to the conclusion. In fact, it may even be that we can reject the idea that writing in a TDD style takes 16% more time on average. The second study is much worse. It quotes studies secondhand, only has 4 studies (with 4 "controls" being completely different systems). It generally fails to the problem of lurking variables, or confounding: That some unmeasured external factor affects the study. For instance, it is put forth that the Microsoft teams in the study employ many methods for minimizing program defects including static analysis. The other grande problem here is that there are no assesment of the control-group systems at all. They will, invariably differ in more than one variable and the question is: Have we measured to benefits of TDD or have we measured the benefits of something else? We can't say for sure. Finally, we only test statically typed languages with a subtyping construct (Java, C#, C++). I think (and I stress it is a hypothesis) dynamically typed languages would benefit considerably more with a TDD system in place, and I hypothesize more expressively typed languages (ML, Haskell) would benefit less. But I have no good idea of how to design an experiment that can show this hypothesis true.