7 ms·

It looks like it was a nice talk, but unfortunately it failed to address the reason why there is so little reliable science when it comes to software developmen

by swombat 17y ago

It looks like it was a nice talk, but unfortunately it failed to address the reason why there is so little reliable science when it comes to software development. The reason is, of course, that we have yet to come up with a reliable, meaningful metric for programmer productivity, one that would apply to the lone ruby on rails coder as well as to the enterprise java cog-in-the-great-machine or the ThoughtWorks elite agile programmer in a 6-people iterative team. Until there is a meaningful way to compare the productivity of different software methodologies, it is pointless to berate the lack of scientific studies comparing the productivity of these methodologies. As a final nail in that coffin, I don't think anyone can claim that Google is lacking in the "let's measure everything" department, with their reputation for intensive A/B testing of every single pixel change. And yet Google has yet to resolve this problem... which would tend to indicate that this is a real, non-trivial problem, not just a case of people not trying hard enough.

nopassrecover 17y ago
I'm of the opinion that the reason there is no quantitative metric for programmers is the same reason there is no quantitative metric for writers, artists or musicians.
- arohner 17y ago
  I agree this is a key point. Software bug rates are not comparable to scurvy incidence rates, and developer productivity is not similar to the number of bricks a laborer can move in an hour.
- gaius 17y ago
  Yet there are quantitative metrics for engineers. Is your Ruby on Rails application really more complex than the software flying an airliner?
  - prospero 17y ago
    There are metrics for problems which are studied in-depth before engineers are let loose to solve it. That's not true of most software projects.
    - gaius 17y ago
      Who do you think does the studying? Engineers...
      - derefr 17y ago
        You're agreeing with the parent. There are metrics to measure solving well-known problems, but no metrics for researching problems to make them well-known.
        gaius 17y ago
        But the first person or team who realized that you could put a computer in an aircraft weren't solving a known problem - however they were using engineering methods, which is why it's very rare for avionics to (literally!) crash. Here's my theory: we do know how to build very high quality software in a repeatable way. However very few organizations are willing to pay what it costs to do that, and that is why most software is so bad. Admittedly a lot of people writing code these days are "cowboys" (they don't care and neither do their clients, they just want speed) but that doesn't mean that the body of knowledge doesn't exist.
        pmichaud 17y ago
        I don't know why this comment was voted down, but I think he could be right -- given the correct resources (time mainly), and assuming the right talent level, I think we could consistently build rock-solid software. The constraint seems to be a (logical and necessary) trade off between total cost and total utility. As we all know, something that works 95% of the time at 5% the price is very often better than a "perfect" solution. With critical systems like plane navigation, the tradeoff doesn't make sense, so you pay full price for near-perfection... your niche e-commerce site? 95% is sufficient.
  - apotheon 17y ago
    I don't think it's a matter of complexity so much as a matter of difficulty defining the problem.
    - natevc2 17y ago
      I agree, But what if we looked at a job where productivity should be much easier to define, like a roofer. An obvious performance metric would be "Time it takes to install a roof that will pass a quality inspection". That metric is pretty useless alone though. Can he fix a nail gun that breaks? How safe is he? Can he work as fast in cold weather? are just a few factors. Say you could get metrics for ALL productivity factors, you still haven't taken into account personality issues that cannot be quantified. And this is for a single person working on a measurable task with only a few tools. What chance is there for metrics to be accurate about a task that could require any number of tools, multiple correct answers, and collaboration?
  - yters 17y ago
    What's your point? Does airline software have a good metric? The main difference is that other engineering disciplines don't have to deal with the halting problem.
  - Hexstream 17y ago
    It's more open-ended because you can mostly make your own rules, unlike in the physical world where you have lots of predefined constraints. The resulting solution space being much bigger, you have to use your taste to narrow down to the better solutions, so of course it's more subjective.
  - nopassrecover 17y ago
    "Does it work" isn't a quantitative metric, it's a qualitative one.
- teeja 17y ago
  For sure. Before 'hit parades' music could sit around in a desk drawer for decades (Schubert) before being heard. Because innovation drove away audiences wanting to hear the familiar. The people who made sure that Schubert's stuff lived was other composers ... not audiences. Software's the same ... the people who cheer great code (art, science) are people who understand it well enough to recognize great code. The opinions of peers is an important part of achievement metrics. And that artful kind of code pretty much counts for nothing by any standard 'industry' measures. But it's influential.
angelbob 17y ago
One solution to this is that you can measure similar projects. "The Mythical Man-Month" is based on this idea. If you compare large projects to large projects (or a large project to itself with different methods, different number of people, etc), you get a decently reliable, if imperfect, metric which can give good results. Similarly, compare elite six-man teams to other elite six man teams (not very reliable, but interesting if results are extreme) or to themselves on other projects, possibly with slightly different methodologies. Yes, it's difficult. It would be nice if we had some simple means of measuring productivity, but much of programming is very artlike -- we're not solving identical problems, and we may solve them in very different ways, and a lot of the resulting quality depends on the taste of the viewer. We're not going to get a really excellent, predictable way to measure quality. Waiting until we do means no research, which is awfully fatalistic.
ejames 17y ago
Still, there's a lot of room between "no evidence" and "double-blind clinical trials", and reason to believe that statements about software development can profitably be moved towards the latter. I would take Code Complete as a good example. There are sections on methodology that run along the lines of "Company A tried this method and got a B% reduction in defects compared to the first version of the product. Company X also used it and only got a Y% improvement compared to the previous year, but that's still worth using." Studies don't have to be totally conclusive in order to provide meaningful back-up to anecdotal reports and subjective experience. In other words, the perfect is the enemy of the good, and good evidentiary metrics may be achievable even if perfect ones aren't.
Maciek416 17y ago
Good point. I attended this talk, and it was easily one of the best talks at Stackoverflow (Devdays) Toronto, and got a very rousing response from the crowd. To answer your concern, the question of "why" was actually one of the overall points of his discussion. Wilson would enthusiastically agree with you that there aren't very good metrics out there not only for programmer productivity, but a number of other things that you'd want to measure in software. At the beginning of the talk he contrasted the software sciences against other, older disciplines just to show how incredibly nascent it is. His most damning point of the current state of affairs was that there is a lot of metrics/methodology pseudoscience being spouted by people and businesses in our profession, and that we often eat it up unchallenged. His optimistic counter-point was that things are turning around and measurement, experimentation and skepticism are spreading, and as you point out, for some companies like Google, it's yielding a lot of gains, even if they haven't cracked any universal code of productivity (does anyone actually believe we ever will? This will always be a moving target). The most frequent phrase used by Wilson in his talk was "wouldn't that be a useful thing to know?" (with respect to the result of some experiment that would show a verifiable result). Essentially, his talk was a rallying cry to attempt to stir the audience into becoming more critical about their own methods and to boost their skepticism of the things they're told about methodology.
- 3pt14159 17y ago
  I also attended this conference in Toronto. I didn't like the last two thirds of the talk one bit. There is plenty of evidence that explains why less females are in software development (and in general, science) there is also plenty of science that disputes anthropogenic global warming (and also plenty of science that supports it). The speaker made highly emotionally charged statements that were in the opposite spirit of his whole thesis: DATA and ANALYSIS should drive opinions and decisions, not emotions. All of this is besides the point. What does global warming or Rush Limbaugh have to do with the science behind software development organization? While he is absolutely correct that software development lacks rigorous testing of effective methods (agile vs waterfall), he allows, openly, volunteers (!) to be part of his test group ("I'll post my email") after already telling them what he is testing. This makes his inquisition of the most effective method even worse than nothing at all. By taking in biased volunteers he makes his numbers look like they achieve statistically validly, while failing to take a true cross-section of software developers. My favorite talk of the night was the Ruby guy. Clean, unemotional, understandable, interesting, informative. I almost expected him to end with "live long and prosper."
mustpax 17y ago
Not only is there no metric, but when we force artificial metrics onto programming teams, we are forcing them to pursue a goal other than the quality of the software itself. Any benefits derived from measurability will be eclipsed by the bad practices brought about by the metrics. Take the two of the most common pseudo-metrics: lines-of-code, bugs-closed-per-month. Lines-of-code encourages verbosity and complexity while discouraging code reuse. It also ignores essential code maintenance tasks that usually remove lines-of-code rather than adding them. Bugs-closed-per-month encourages the creation of bugs in the first place. Not that anyone intentionally makes mistakes, but it lets the programmer get away with getting it less-right the first time. Even worse, it assures continued dividends from an error-prone implementation. It would be useful to seek macroscopic measures of programmer productivity in our discipline as a whole. Let's just keep in mind the potential adverse effects of applying these metrics on the ground. Realistically, any effective macro metric will be abused my PHBs all over the world in managing their 10 man teams.
- pohl 17y ago
  Somehow even your use of the word "force" here still manages to understate the severity of the problem. The "Performance Management" craze is sweeping corporate management, along with the pernicious use of the "Balanced Scorecard" it advocates. Unable to identify meaningful metrics to build these scorecards with, management delegates to programmers the task of inventing metrics. Programmers, who know very well how any metric can be gamed, know that they're obligated to hand management a loaded gun, and hope they can fill the scorecards with a gun of the least-damaging calibre. This will continue unabated until a better science appears, and there's a very real possibility that one never will (if programming is more akin to an art and/or craft). I would love to have seen this presentation.
  - apotheon 17y ago
    > This will continue unabated until a better science appears . . . or until a better theory of management appears and gains widespread acceptance.
- srn 17y ago
stonemetal 17y ago
What I find odd is that there used to be quite a bit of research in this area. Read older software development books they reference papers research more often. i.e. Code Complete vs. agile programming books.
raganwald 17y ago
I freely admit that I have no idea how to measure productivity. And until I do, I really can't tell you that what I choose to do is anything other than consistent with my superstitions about software development. In other words, I choose some superstitions and I can build up a consistent kind of world based on those superstitions and I can test various ideas to see if the ideas are consistent with my superstitions or the logical consequences of those superstitions. But really, I have no idea whether I'm directing development traffic or whether I have a pair of coconuts clamped to my ears and I'm waving palm fronds under an empty sky. http://github.com/raganwald/homoiconic/blob/master/2009-02-12/a_question.md#readme http://github.com/raganwald/homoiconic/blob/master/2009-02-1...
InclinedPlane 17y ago
Worse than that. We can't even measure success. In the extremes of ultimate financial success or failure in the long term we can determine success, but that gives us something like 1 bit of data per 3-5 years per major software project. Is twitter successful? MySpace? Facebook? Was Netscape successful? Geocities? (The stock used to buy Geocities is currently worth about $2 billion dollars.) For each of those there are even more examples that are more difficult to determine. Modern software comes in a series of releases, how much can you attribute the success of a particular release to the existing code base, and how much can you attribute it to just the particular diffs for that release? It's a tricky problem with no easy answers.
nradov 17y ago
In principle it should be practical to compare the productivity of different software methodologies by counting Function Points (or something similar) as well as defects. http://www.ifpug.org/ http://www.ifpug.org/ But in practice that doesn't work very well. It's awkward to count Function Points for maintenance and refactoring work in a meaningful way. And for most organizations, the cost of just gathering the data and doing the counting is so high as to outweigh any possible benefit.

by swombat 17y ago

nopassrecover 17y ago

I'm of the opinion that the reason there is no quantitative metric for programmers is the same reason there is no quantitative metric for writers, artists or musicians.

arohner 17y ago
I agree this is a key point. Software bug rates are not comparable to scurvy incidence rates, and developer productivity is not similar to the number of bricks a laborer can move in an hour.
gaius 17y ago
Yet there are quantitative metrics for engineers. Is your Ruby on Rails application really more complex than the software flying an airliner?
- prospero 17y ago
  There are metrics for problems which are studied in-depth before engineers are let loose to solve it. That's not true of most software projects.
  - gaius 17y ago
    Who do you think does the studying? Engineers...
    - derefr 17y ago
      You're agreeing with the parent. There are metrics to measure solving well-known problems, but no metrics for researching problems to make them well-known.
      - gaius 17y ago
        But the first person or team who realized that you could put a computer in an aircraft weren't solving a known problem - however they were using engineering methods, which is why it's very rare for avionics to (literally!) crash. Here's my theory: we do know how to build very high quality software in a repeatable way. However very few organizations are willing to pay what it costs to do that, and that is why most software is so bad. Admittedly a lot of people writing code these days are "cowboys" (they don't care and neither do their clients, they just want speed) but that doesn't mean that the body of knowledge doesn't exist.
        pmichaud 17y ago
        I don't know why this comment was voted down, but I think he could be right -- given the correct resources (time mainly), and assuming the right talent level, I think we could consistently build rock-solid software. The constraint seems to be a (logical and necessary) trade off between total cost and total utility. As we all know, something that works 95% of the time at 5% the price is very often better than a "perfect" solution. With critical systems like plane navigation, the tradeoff doesn't make sense, so you pay full price for near-perfection... your niche e-commerce site? 95% is sufficient.
- apotheon 17y ago
  I don't think it's a matter of complexity so much as a matter of difficulty defining the problem.
  - natevc2 17y ago
    I agree, But what if we looked at a job where productivity should be much easier to define, like a roofer. An obvious performance metric would be "Time it takes to install a roof that will pass a quality inspection". That metric is pretty useless alone though. Can he fix a nail gun that breaks? How safe is he? Can he work as fast in cold weather? are just a few factors. Say you could get metrics for ALL productivity factors, you still haven't taken into account personality issues that cannot be quantified. And this is for a single person working on a measurable task with only a few tools. What chance is there for metrics to be accurate about a task that could require any number of tools, multiple correct answers, and collaboration?
- yters 17y ago
  What's your point? Does airline software have a good metric? The main difference is that other engineering disciplines don't have to deal with the halting problem.
- Hexstream 17y ago
  It's more open-ended because you can mostly make your own rules, unlike in the physical world where you have lots of predefined constraints. The resulting solution space being much bigger, you have to use your taste to narrow down to the better solutions, so of course it's more subjective.
- nopassrecover 17y ago
  "Does it work" isn't a quantitative metric, it's a qualitative one.
teeja 17y ago
For sure. Before 'hit parades' music could sit around in a desk drawer for decades (Schubert) before being heard. Because innovation drove away audiences wanting to hear the familiar. The people who made sure that Schubert's stuff lived was other composers ... not audiences. Software's the same ... the people who cheer great code (art, science) are people who understand it well enough to recognize great code. The opinions of peers is an important part of achievement metrics. And that artful kind of code pretty much counts for nothing by any standard 'industry' measures. But it's influential.

angelbob 17y ago

One solution to this is that you can measure similar projects. "The Mythical Man-Month" is based on this idea. If you compare large projects to large projects (or a large project to itself with different methods, different number of people, etc), you get a decently reliable, if imperfect, metric which can give good results. Similarly, compare elite six-man teams to other elite six man teams (not very reliable, but interesting if results are extreme) or to themselves on other projects, possibly with slightly different methodologies. Yes, it's difficult. It would be nice if we had some simple means of measuring productivity, but much of programming is very artlike -- we're not solving identical problems, and we may solve them in very different ways, and a lot of the resulting quality depends on the taste of the viewer. We're not going to get a really excellent, predictable way to measure quality. Waiting until we do means no research, which is awfully fatalistic.

ejames 17y ago

Still, there's a lot of room between "no evidence" and "double-blind clinical trials", and reason to believe that statements about software development can profitably be moved towards the latter. I would take Code Complete as a good example. There are sections on methodology that run along the lines of "Company A tried this method and got a B% reduction in defects compared to the first version of the product. Company X also used it and only got a Y% improvement compared to the previous year, but that's still worth using." Studies don't have to be totally conclusive in order to provide meaningful back-up to anecdotal reports and subjective experience. In other words, the perfect is the enemy of the good, and good evidentiary metrics may be achievable even if perfect ones aren't.

Maciek416 17y ago

Good point. I attended this talk, and it was easily one of the best talks at Stackoverflow (Devdays) Toronto, and got a very rousing response from the crowd. To answer your concern, the question of "why" was actually one of the overall points of his discussion. Wilson would enthusiastically agree with you that there aren't very good metrics out there not only for programmer productivity, but a number of other things that you'd want to measure in software. At the beginning of the talk he contrasted the software sciences against other, older disciplines just to show how incredibly nascent it is. His most damning point of the current state of affairs was that there is a lot of metrics/methodology pseudoscience being spouted by people and businesses in our profession, and that we often eat it up unchallenged. His optimistic counter-point was that things are turning around and measurement, experimentation and skepticism are spreading, and as you point out, for some companies like Google, it's yielding a lot of gains, even if they haven't cracked any universal code of productivity (does anyone actually believe we ever will? This will always be a moving target). The most frequent phrase used by Wilson in his talk was "wouldn't that be a useful thing to know?" (with respect to the result of some experiment that would show a verifiable result). Essentially, his talk was a rallying cry to attempt to stir the audience into becoming more critical about their own methods and to boost their skepticism of the things they're told about methodology.

3pt14159 17y ago
I also attended this conference in Toronto. I didn't like the last two thirds of the talk one bit. There is plenty of evidence that explains why less females are in software development (and in general, science) there is also plenty of science that disputes anthropogenic global warming (and also plenty of science that supports it). The speaker made highly emotionally charged statements that were in the opposite spirit of his whole thesis: DATA and ANALYSIS should drive opinions and decisions, not emotions. All of this is besides the point. What does global warming or Rush Limbaugh have to do with the science behind software development organization? While he is absolutely correct that software development lacks rigorous testing of effective methods (agile vs waterfall), he allows, openly, volunteers (!) to be part of his test group ("I'll post my email") after already telling them what he is testing. This makes his inquisition of the most effective method even worse than nothing at all. By taking in biased volunteers he makes his numbers look like they achieve statistically validly, while failing to take a true cross-section of software developers. My favorite talk of the night was the Ruby guy. Clean, unemotional, understandable, interesting, informative. I almost expected him to end with "live long and prosper."

mustpax 17y ago

Not only is there no metric, but when we force artificial metrics onto programming teams, we are forcing them to pursue a goal other than the quality of the software itself. Any benefits derived from measurability will be eclipsed by the bad practices brought about by the metrics. Take the two of the most common pseudo-metrics: lines-of-code, bugs-closed-per-month. Lines-of-code encourages verbosity and complexity while discouraging code reuse. It also ignores essential code maintenance tasks that usually remove lines-of-code rather than adding them. Bugs-closed-per-month encourages the creation of bugs in the first place. Not that anyone intentionally makes mistakes, but it lets the programmer get away with getting it less-right the first time. Even worse, it assures continued dividends from an error-prone implementation. It would be useful to seek macroscopic measures of programmer productivity in our discipline as a whole. Let's just keep in mind the potential adverse effects of applying these metrics on the ground. Realistically, any effective macro metric will be abused my PHBs all over the world in managing their 10 man teams.

pohl 17y ago
Somehow even your use of the word "force" here still manages to understate the severity of the problem. The "Performance Management" craze is sweeping corporate management, along with the pernicious use of the "Balanced Scorecard" it advocates. Unable to identify meaningful metrics to build these scorecards with, management delegates to programmers the task of inventing metrics. Programmers, who know very well how any metric can be gamed, know that they're obligated to hand management a loaded gun, and hope they can fill the scorecards with a gun of the least-damaging calibre. This will continue unabated until a better science appears, and there's a very real possibility that one never will (if programming is more akin to an art and/or craft). I would love to have seen this presentation.
- apotheon 17y ago
  > This will continue unabated until a better science appears . . . or until a better theory of management appears and gains widespread acceptance.
srn 17y ago

stonemetal 17y ago

What I find odd is that there used to be quite a bit of research in this area. Read older software development books they reference papers research more often. i.e. Code Complete vs. agile programming books.

raganwald 17y ago

I freely admit that I have no idea how to measure productivity. And until I do, I really can't tell you that what I choose to do is anything other than consistent with my superstitions about software development. In other words, I choose some superstitions and I can build up a consistent kind of world based on those superstitions and I can test various ideas to see if the ideas are consistent with my superstitions or the logical consequences of those superstitions. But really, I have no idea whether I'm directing development traffic or whether I have a pair of coconuts clamped to my ears and I'm waving palm fronds under an empty sky. http://github.com/raganwald/homoiconic/blob/master/2009-02-12/a_question.md#readme http://github.com/raganwald/homoiconic/blob/master/2009-02-1...

InclinedPlane 17y ago

Worse than that. We can't even measure success. In the extremes of ultimate financial success or failure in the long term we can determine success, but that gives us something like 1 bit of data per 3-5 years per major software project. Is twitter successful? MySpace? Facebook? Was Netscape successful? Geocities? (The stock used to buy Geocities is currently worth about $2 billion dollars.) For each of those there are even more examples that are more difficult to determine. Modern software comes in a series of releases, how much can you attribute the success of a particular release to the existing code base, and how much can you attribute it to just the particular diffs for that release? It's a tricky problem with no easy answers.

nradov 17y ago

In principle it should be practical to compare the productivity of different software methodologies by counting Function Points (or something similar) as well as defects. http://www.ifpug.org/ http://www.ifpug.org/ But in practice that doesn't work very well. It's awkward to count Function Points for maintenance and refactoring work in a meaningful way. And for most organizations, the cost of just gathering the data and doing the counting is so high as to outweigh any possible benefit.