5 ms·

The most practically relevant finding of the original paper remains in effect: the choice of language has an effect of less than 1% on the bug rate[1]. The orig

by pron 7y ago

The most practically relevant finding of the original paper remains in effect: the choice of language has an effect of less than 1% on the bug rate[1]. The original paper explains: "With effects coding, each coefficient indicates the relative effect of the use of a particular language on the response as compared to the weighted mean of the dependent variable across all projects." So the entire discussion is on the relative effect of particular languages relative to the total effect of language choice, which is less than 1%. The original paper reported stronger relative effects for particular languages, while this one found the effects to be smaller, and in many cases they disappeared altogether, thus summarizing the effects of the languages as "exceedingly small." So even the comparison of C++ vs. Haskell, that has ~25% difference, is within that 1%. This is similar to a study comparing the effect of running shoes on running speed that shows that Nike positively affects your running speed 25% more than Adidas, to the extent running shoes affect your speed at all, which is less than 1%. ----- [1] One should take care not to overestimate the impact of language on defects. While these relationships are statistically significant, the effects are quite small. In the analysis of deviance... we see that activity in a project accounts for the majority of explained deviance... The next closest predictor, which accounts for less than one percent of the total deviance, is language. (https://web.cs.ucdavis.edu/~filkov/papers/lang_github.pdf https://web.cs.ucdavis.edu/~filkov/papers/lang_github.pdf)

scott_s 7y ago
I disagree that the combination of the two studies supports an effect of language choice on bug rate. Given that the original study showed a negative association for about half the languages; the replication has significant disagreement on about half of the original positive associations; and the extremely small associations across all languages, I think we're looking at noise.
- pron 7y ago
  But if we consider as the main result that the total effect of language choice is less than 1%, even "looking at noise" means looking at an effect of 0, and either way, the effect is tiny to nonexistent. So if someone had the hypothesis that the effect of language choice is large, because lack of evidence of a big effect is evidence of lack of one, this study is some evidence against such a hypothesis. That this finding agrees with the observations in the industry lends it even more credence.
  - jrumbut 7y ago
    If I were sitting on a thing that could decrease bugs 1% I'd be really excited. The only reason this looks like a paltry reduction is that it's hugely expensive and difficult to change languages midstream. This is evidence against a shop retraining C++ devs in Haskell (which I'm sure someone somewhere wants to do). But if it was something like switching editor fonts? I'd do it for 1%.
    - deleted 7y ago
      [deleted]
yogthos 7y ago
Two things this studies haven't measured are time to deliver features, and the number of developers necessary. For example, if a project in Haskell can be delivered faster by a smaller team than an equivalent project in C++, that would be an important factor.
- bunderbunder 7y ago
  There's a concept I discovered in the realm of political science, the "fundamentally unanswerable question" - ones where it's not actually possible to generate the kinds of counterfactuals you'd need to try and run an experiment. I think this is probably one of those. Haskell and C++ are very different languages, that are designed for very different purposes, and are popular in very different problem domains. To start with, that would make it difficult to come up with a task that wouldn't handicap one language or the other. (For example, a video game would favor C++.) Even if you found that, you'd have a hard time coming up with a good test. Under random assignment, Haskell is presumably going to be handicapped relative to C++ by the fact that most languages are imperative and use Algol-style syntax. You could try to avoid that by using participants with zero programming experience, but then you're studying learning curve for absolute beginners rather than productivity among experts. And so on and so on. Long story short, probably best to leave this one to the spittle-flecked Internet arguments.
  - yogthos 7y ago
    I really don't see how that would be the case. Both Haskell and C++ are general purpose languages, there are plenty of projects written in many overlapping domains. The test would to treat projects as black boxes, and look for trends across many projects. If you saw a trend that Haskell projects are consistently delivered faster by smaller teams, then you could make a hypothesis that Haskell is a factor. At the end of the day, we've come up with ways to test hypothesis like quantum mechanics. I'm pretty sure measuring impact of a language on software development process is a tractable problem.
    - bunderbunder 7y ago

by pron 7y ago

scott_s 7y ago

I disagree that the combination of the two studies supports an effect of language choice on bug rate. Given that the original study showed a negative association for about half the languages; the replication has significant disagreement on about half of the original positive associations; and the extremely small associations across all languages, I think we're looking at noise.

pron 7y ago
But if we consider as the main result that the total effect of language choice is less than 1%, even "looking at noise" means looking at an effect of 0, and either way, the effect is tiny to nonexistent. So if someone had the hypothesis that the effect of language choice is large, because lack of evidence of a big effect is evidence of lack of one, this study is some evidence against such a hypothesis. That this finding agrees with the observations in the industry lends it even more credence.
- jrumbut 7y ago
  If I were sitting on a thing that could decrease bugs 1% I'd be really excited. The only reason this looks like a paltry reduction is that it's hugely expensive and difficult to change languages midstream. This is evidence against a shop retraining C++ devs in Haskell (which I'm sure someone somewhere wants to do). But if it was something like switching editor fonts? I'd do it for 1%.
  - deleted 7y ago
    [deleted]

yogthos 7y ago

Two things this studies haven't measured are time to deliver features, and the number of developers necessary. For example, if a project in Haskell can be delivered faster by a smaller team than an equivalent project in C++, that would be an important factor.

bunderbunder 7y ago
There's a concept I discovered in the realm of political science, the "fundamentally unanswerable question" - ones where it's not actually possible to generate the kinds of counterfactuals you'd need to try and run an experiment. I think this is probably one of those. Haskell and C++ are very different languages, that are designed for very different purposes, and are popular in very different problem domains. To start with, that would make it difficult to come up with a task that wouldn't handicap one language or the other. (For example, a video game would favor C++.) Even if you found that, you'd have a hard time coming up with a good test. Under random assignment, Haskell is presumably going to be handicapped relative to C++ by the fact that most languages are imperative and use Algol-style syntax. You could try to avoid that by using participants with zero programming experience, but then you're studying learning curve for absolute beginners rather than productivity among experts. And so on and so on. Long story short, probably best to leave this one to the spittle-flecked Internet arguments.
- yogthos 7y ago
  I really don't see how that would be the case. Both Haskell and C++ are general purpose languages, there are plenty of projects written in many overlapping domains. The test would to treat projects as black boxes, and look for trends across many projects. If you saw a trend that Haskell projects are consistently delivered faster by smaller teams, then you could make a hypothesis that Haskell is a factor. At the end of the day, we've come up with ways to test hypothesis like quantum mechanics. I'm pretty sure measuring impact of a language on software development process is a tractable problem.
  - bunderbunder 7y ago