One solution to this is that you can measure similar projects. "The Mythical Man-Month" is based on this idea. If you compare large projects to large projects

One solution to this is that you can measure similar projects. "The Mythical Man-Month" is based on this idea. If you compare large projects to large projects (or a large project to itself with different methods, different number of people, etc), you get a decently reliable, if imperfect, metric which can give good results. Similarly, compare elite six-man teams to other elite six man teams (not very reliable, but interesting if results are extreme) or to themselves on other projects, possibly with slightly different methodologies. Yes, it's difficult. It would be nice if we had some simple means of measuring productivity, but much of programming is very artlike -- we're not solving identical problems, and we may solve them in very different ways, and a lot of the resulting quality depends on the taste of the viewer. We're not going to get a really excellent, predictable way to measure quality. Waiting until we do means no research, which is awfully fatalistic.