4 ms·
Someone needs to defend McConnell here. I'll do it. He's one of the most meticulous people to work with this material. That comes through pretty clearly in the
by gruseom 14y ago
Someone needs to defend McConnell here. I'll do it. He's one of the most meticulous people to work with this material. That comes through pretty clearly in the post you linked to. The problem is not McConnell, it's – and this is the dirty secret of software engineering – that the research literature itself is so weak and spotty. Most studies consist of arbitrary metrics on small data sets with no replication, rarely (ever?) making the actual data available. Massive assumptions and confounding factors are typically obvious and ignored (things like: how familiar were the programmers with the tools they were using). The overwhelming factor is the interpretive preferences of the authors, who knit their conclusions out of this dodgy material. As science, it not only does not hold up, it's a joke.
On the other hand, what they are trying to study is incredibly complex - not just people, not just software, but people working on software, and even social systems of people working on software. How do you even begin to reliably measure that? Psychology looks like physics by comparison. Yet the resources available to do these studies are a drop in the bucket. This is not something society values – it's not even something our own industry values. So it's unsurprising that what we have so far is at best a smattering. You can argue that these toy studies ought not to be dressed up in formal scientific wear, and I don't disagree – it gives an altogether misleading impression of how solid their conclusions are. But no doubt that's true of a lot of papers.
(Oh and on top of all that, much of the literature is hard to track down. Didn't anyone notice that the OP wasn't able to even find half of the citations he was trying to check? Based on that alone he shouldn't be claiming that "the table is completely false", only that he was unable to rebuild it from data.)
The real question is whether we should throw it all out as junk or try to make something of it. I think there's a good argument for junking. But it's also reasonable to say: hold on, for all their weaknesses these studies are all we have and they're not nothing. At least someone is trying to work with actual data. So let's consider them, but carefully. Like walking on a swamp.
McConnell is one of the few people who have made serious efforts to walk the swamp and filter it somewhat and convey its findings to professional programmers. Of all these people (that I've read), he actually stands out as the most meticulous. To fire cowboy accusations of dishonesty at him personally is likely unfair and misses the point. I do think McConnell overestimates how solid the swamp is, but he's not ignorant about it (read that whole post) and the claim that he outright falsifies citations, or anything like that, demands a high burden of proof. I haven't read Bossavit's critique (should I?) but anyone taking that position had better first make sure he's not standing on a swamp too. That's harder than it sounds.
- Morendil 14y ago> I haven't read Bossavit's critique (should I?) but anyone taking that position had better first make sure he's not standing on a swamp too. Author of "Leprechauns" here. I'm not sure I follow you in the above sentence. What happened time and again in my investigations was that I would find a citation (in McConnell or in Boehm or elsewhere) that was metaphorically accompanied with the statement "this here is a solid piece of land in the swamp". When I got there, however, what I found was, in fact, just more swamp. It takes a higher standard of proof to demonstrate that something is solid ground than to demonstrate there's a flaw in it. That may be unfair, but it's why scientists need a lot of training. To take just one example, Graham mentions the Hughes Aircraft study, cited (by McConnell and others) in support of the usual "exponential" curve for defect cost (as "Willis, Ron R., et al, 1998"). When you actually read that paper - which is both one of the easiest to obtain and one that has the more detailed data about what was studied and how - and look for the raw data, you find numbers that do not obey an exponential law OR a monotonic increase. For one column for instance the numbers in fact vary within a narrow range .36 to 2.00 with two maxima at the "Coding" and "Functional test" phases, dropping off before, between and after. In a different column the costs vary only by a factor of two between Unit Test and System Test, by a factor of less than four between Code and System Test. The exponential rise is generally not true for the pre-1991 period; some post-1991 measurements get closer, but many do not. It's hard for me to see this evidence as doing anything else but undermining the original claim of the exponential rise as a generally reliable regularity in software development. Given this, I don't think that citing this paper in support of the claim is appropriate.
- gruseom 14y ago"I'm not sure I follow you in the above sentence." I'm saying that if one is going to attack someone personally for dishonesty, as opposed to critiquing the state of the field in general, the bar for that is high and one had better be more than careful with one's own particulars. Reasonable people can interpret this stuff differently. The post linked to upthread [1] didn't strike me as dishonest (though of course it's only one side). I'm glad you've been looking for solid ground in the swamp and finding it swampy. I think that's valuable. What bothers me are the hints in the OP, comments here, and elsewhere I've seen (including people talking about your work) that this is about one guy making shit up when in reality the problem is endemic to the entire field. Unless he's been egregiously dishonest, which I doubt, it's a distraction. On another note, you seem like a good person to ask: is there any finding in the software engineering literature that you think holds up? i.e. have you found any solid ground in the swamp? I'm not sure I have (but I haven't looked nearly as hard as you). If there really isn't anything, that alone is kind of shocking. [1] http://forums.construx.com/blogs/stevemcc/archive/2011/01/09/origins-of-10x-how-valid-is-the-underlying-research.aspx http://forums.construx.com/blogs/stevemcc/archive/2011/01/09...