3 ms·

I fear the problems on this empirical approach are deeper. The main problem is that software metrics are imprecise and non-objective. Lines of code, functional

I fear the problems on this empirical approach are deeper. The main problem is that software metrics are imprecise and non-objective. Lines of code, functional points, code coverage, counting code paths, ... we can't have a metric that can be accepted by everyone, all of them have big flaws. And the metrics are the basis for any reliable analysis, if we can't trust them we can't trust anything. The second main problem is that it is very hard to isolate things under examination. How can we analyse TDD without taking into account the developer's grasp of good design (coupling and cohesion), Dependency Injection and Inversion of Control, refactoring techniques and tools, etc? Software Engineering is a lot harder because it is much more akin to the fuzzy social studies (e.g.: economics, sociology, management) than to hard sciences (e.g.: computing science).

davidcuddeback 14y ago
To be fair, the metrics you mentioned are objective. It's just debatable how relevant they are. Other metrics that are used are measures of coupling, cohesion, lines per method, methods per class, etc. There are probably more measures that are used. None of them are perfect, but they all help paint a picture that adds to our understanding. To address your second point, it's actually not too difficult to isolate factors like TDD. The standard way is to have a control group and a test group. With a large enough of a sample size, you can determine statistical significance with standard tests. Unfortunately, the test subjects are often university students, who are less experienced than professionals. The fact that data collected on students might not generalize to professionals is a threat to external validity, but should be made explicit in most papers. Most of the time, I think companies aren't very happy about having researchers use their engineers for experiments on the company's dime, but it does happen. So there are some papers out there reporting results with professionals.

davidcuddeback 14y ago

To be fair, the metrics you mentioned are objective. It's just debatable how relevant they are. Other metrics that are used are measures of coupling, cohesion, lines per method, methods per class, etc. There are probably more measures that are used. None of them are perfect, but they all help paint a picture that adds to our understanding. To address your second point, it's actually not too difficult to isolate factors like TDD. The standard way is to have a control group and a test group. With a large enough of a sample size, you can determine statistical significance with standard tests. Unfortunately, the test subjects are often university students, who are less experienced than professionals. The fact that data collected on students might not generalize to professionals is a threat to external validity, but should be made explicit in most papers. Most of the time, I think companies aren't very happy about having researchers use their engineers for experiments on the company's dime, but it does happen. So there are some papers out there reporting results with professionals.