13 ms·

On the Impact of Programming Languages on Code Quality

264 points by lelf 7y ago

umass_alum 7y ago
This is great (to see something I know about on HN). As an undergrad at UMass (where Emery Berger teaches), I remember the exact paper they are talking about, and how it was presented at UMass (this was part of a lunch and learn with the faculty, grad, and undergrad students and the author of the original paper). I remember Emery being quite confused at the methodology in the paper, and he was very confused with one aspect of it, which was how they came to conclude that C was more bug prone (something about void * being able to cast to everything? It was a while ago...) Glad to see he wrote a rebuttal, a little awkward watching a professor debate with the paper's author :)
- mcguire 7y ago
  I knew Emery at UT Austin; he's a good researcher, but I'd imagine "awkward" is an understatement.
melling 7y ago
...deleted I skimmed the paper. I misunderstood the results.
- smoe 7y ago
  Your quote was the conclusion of the original work by Ray et al., not the one from the reproduction paper linked here. That one in short is: "Unfortunately, our work has identified numerous problems in the FSE study that invalidated its key result."
voidhorse 7y ago
Interesting! Excepting typescript, the “safer” languages also happen to be the ones I gravitate towards on a daily basis and actually enjoy writing programs in, while the others... It’d be interesting to see whether programmers’ preferences uphold the division posited by the study, or whether they are widely variable—eg is it likely that if you prefer ruby you’ll also prefer typescript over JavaScript, if you prefer Haskell you’d choose clojure over c++ etc.
- scott_s 7y ago
  I think you're confusing the conclusions from the original paper with the conclusions from this paper. This paper is a reproduction of the original study, and for the most part, it is not able to reproduce the findings.
  - voidhorse 7y ago
    How right you are. That’s what I get for skimming news piecemeal way too quickly in the mornings!
panpanna 7y ago
Maybe off topic, but isn't the language in the abstract a bit aggressive? I mean, this type of study is somewhat open to interpretation. It's not like everyone should arrive to the exact same numbers..
- omginternets 7y ago
  Which part? This kind of direct language is both standard and encouraged in academia. The idea is to take a clear position, which can be attacked if wrong.
- idoubtit 7y ago
  The conclusion makes it clear that this work has no aggressive intent: Unfortunately, our work has identified numerous problems in the FSE study that invalidated its key result. Our intent is not to blame, performing statistical analysis of programming languages based on large-scale code repositories is hard. [...] Acknowledgments. We thank Baishakhi Ray and Vladimir Filkov for sharing the data and code of their FSE paper. Had they not preserved the original files and part of their code, reproduction would have been more challenging
jph 7y ago
(Edited-- I was reading the original paper, which showed good results for TypeScript; the new paper linked above refutes that result. My mistake. Thanks to the posters who corrected me!) The winners in this paper are Clojure, Haskell, Scala. I look forward to seeing Rust added. The authors go into substantial detail about what they consider errors, and in my personal experience Rust solves many of them. See e.g. "Some defect type like memory error, concurrency errors also depend on language primitives."
- vidro3 7y ago
  How so? Typescript, along with most other languages is listed as 'No Statistically Significant Association" see fig 2. ETA: They actually did not assess Typescript in their reanalysis as most of the files were not actually Typescript, section 4.1.2
- faitswulff 7y ago
  That's not at all what I got from the abstract: > ...only four languages are found to have a statistically significant association with defects, and even for those the effect size is exceedingly small. Emphasis mine.
- scott_s 7y ago
  I believe you are confusing the conclusions from the original paper with the claims from this paper. I think this is the case because this paper removes TypeScript from their dataset entirely: "4.1.2 Removal of TypeScript. In the original dataset, the first commit for TypeScript was recorded on 2003-03-21, several years before the language was created. Upon inspection, we found that the file extension .ts is used for XML files containing human language translations. Out of 41 projects labeled as TypeScript, only 16 contained TypeScript. This reduced the number of commits from 10,063 to an even smaller 3,782. Unfortunately, the three largest remaining projects (typescript-node-definitions, DefinitelyTyped, and the deprecated tsd) contained only declarations and no code. They accounted for 34.6% of the remaining TypeScript commits. Given the small size of the remaining corpus, we removed it from consideration as it is not clear that we have sufficient data to draw useful conclusions."
- xvilka 7y ago
salmonellaeater 7y ago
Of note is that these are all associations. It's possible the associations are all due to selection effects (or even that the "better" language are actually worse but their effects are reversed by selecting for programmers who are better). In the early days of Python, some employers used it as a way to filter for candidates who were ahead of the curve. I believe Paul Graham mentioned Lisp being good for attracting better programmers, although I can't find the reference anymore [1]. If there's any value in this kind of filter, the effects will show up in the programmers' output. 1. I'm having a hard time finding references for these assertions; the best I've found is some Reddit posts and Joel Spolsky's post on finding great developers: https://www.joelonsoftware.com/2006/09/06/finding-great-developers-2/ https://www.joelonsoftware.com/2006/09/06/finding-great-deve...
- deleted 7y ago
  [deleted]
- hannofcart 7y ago
  Indeed, and in addition to selection bias towards better programmers, there is another selection bias implicit in this. A shop that makes the decision to write their stack in Haskell do it with an intention to focus on correctness. I reckon that such companies would consequently invest in other practices like more tests, higher code coverage and more intensive code reviews resulting in code with fewer errors. This is in stark contrast to a startup that may have hacked together a POC in Python, and once it worked, with a little bit more spit and polish put it into production.
- notacoward 7y ago
  > It's possible the associations are all due to selection effects FWIW, the original authors (Ray et al) said as much.
- goto11 7y ago
  The paper goes further though. It states: Not only is it not possible to establish a causal link between programming language and code quality based on data at hand, but even their correlation proves questionable. I find that a surprising result! This might also call into question the theory that some languages attract better developers.
idoubtit 7y ago
My own take of this: - This study tries to enhance the original work by better metrics (e.g. confusion with C/C++), using better statistics (e.g. uncertainty). - Automatically evaluating the quality of a code is hard and time-consuming. They had to use peer review of labellisation. - Most of the old claims are not confirmed. - Only 2 languages in this study had both a significant impact (impact coeff) and enough fiability (p-value): Clojure and Haskell. But the impact is still small, and there may be other influencing factors. - They propose several ways to further enhance their work (e.g. taking regression tests into account).
- msluyter 7y ago
  "... and there may be other influencing factors." Selection bias? Consider the type of person likely to be attracted to languages like Haskell or Clojure. (Note, didn't read the original study so I don't know the selection criteria. Were code samples in different languages from the same programmers compared, or were there different sets of programmers per language? My comment really only applies to the latter case.)
  - sidlls 7y ago
    What is the type of person likely to be attracted to languages like Haskell or Clojure?
    - maxkwallace 7y ago
      I can think of selection bias effects both in-favor of more defects and against it. On one hand, Haskell or Clojure probably attract more academic types who have a reputation for (consistent with my experience) on-average worse-quality code. On other hand, the barrier-to-entry for these languages is higher so they tend to be programmed by more experienced folks. I assume that people pick functional languages in part because they believe the functional paradigm leads to higher-quality code, so the sample probably cares more about code quality to begin with.
      - 7y ago
dahart 7y ago
One of the simplest but more important conclusions that affects me all the time... it’s easy to search for things, but hard to know whether you’ve got all the answers you want and hard to know if all the answers you got are answers you want. 6.3 Grep considered harmful Simple analysis techniques may be too blunt to provide useful answers. This problem was compounded by the fact that the search for keywords did not look for words and instead captured substrings wholly unrelated to software defects. When the accuracy of classification is as low as 36%, it becomes difficult to argue that results with small effect sizes are meaningful as they may be indistinguishable from noise. If such classification techniques are to be employed, then a careful post hoc validation by hand should be conducted by domain experts.
- stcredzero 7y ago
  One of the simplest but more important conclusions that affects me all the time... it’s easy to search for things, but hard to know whether you’ve got all the answers you want and hard to know if all the answers you got are answers you want. Smalltalk was wonderful in this regard. If you wanted to search for how something was used, you just right-click search for "senders." If you wanted to find implementors, the same. This big difference occurred with chains of senders. In other languages, where there's 2 or 3 ways to get to something or call something, you have this O(2^n) or O(2.5^n) blowup in the work you have to do to properly follow chains of senders. With Smalltalk, it's just O(n). On top of that, there were tools that could let you compose really sophisticated SQL like queries of the code base, dynamically, then pop that up in a browser.
coldcode 7y ago
Testing Objective-C is sort of pointless, they should have picked Swift.
- chriswarbo 7y ago
  Apple didn't invent Objective-C, it just-so-happened to be the language of choice at Next, which carried on into OSX and iOS. The fact that Swift is now Apple's language of choice doesn't make analysis of Objective-C "sort of pointless", any more than e.g. Facebook creating Hack would make anything involving PHP "sort of pointless".
- pmiller2 7y ago
  I disagree. If anything, they should have tested Obj-C and Swift. I would also have liked to see Scheme, Common Lisp, or something else lisp-y besides Clojure. Kotlin would have been interesting. R or Mathematica would have been fun, too. I guess it just comes down to needing a sufficiently large corpus of code and commits and having to eventually publish.
  - deleted 7y ago
    [deleted]
scott_s 7y ago
A common thread I'm seeing in the discussion here is confusing the claims from the original paper with conclusions made by this paper. This is a replication study, and it starts by stating what the original paper found. What the original paper found is not what this paper finds. For the most part, this is a replication failure, for a myriad of reasons, including several kinds of misclassification. From this paper: The reanalysis failed to validate most of the claims of [18]. As Table 6(d-f) shows, the multiple steps of data cleaning and improved statistical modeling have invalidated the significance of 7 out of 11 languages. Even when the associations are statistically significant, their practical significance is small. The reference [18] is https://dl.acm.org/citation.cfm?doid=2635868.2635922 https://dl.acm.org/citation.cfm?doid=2635868.2635922, which is the original paper that this paper is a replication study of.
- narag 7y ago
  So the old adage stands: you can write crap in any language.
  - TheOtherHobbes 7y ago
    And if you do, chances are good it'll end up on GitHub. 5.3 and 5.4 really nail it, IMO. And you can't even use bug fixes as a metric, because there's a difference between semantic bugs which will be significantly language-influenced, architectural bugs which are partly language-influenced, and user-expectation/experience bugs, which are broad-brush logical errors or misunderstandings. (There are probably other categories, but those are the obvious ones I can think of right now.) The real takeaway is there is as yet no objective metric for software quality. The best a working dev shop can do is implement a small but non-trivial sample project with known-good logic in a handful of candidate languages, and see what kind of trade-offs fall out.
    - jrumbut 7y ago
      I'll agree with you for sure no objective metric for quality (that is broad enough to capture a satisfying amount of what we mean by quality). I do think user experience/expectation bugs are very language dependent though. First you have the quality of the String and Datetime libraries. Ruby has all kinds of high quality string functions that are a source of pain and bugs in C. Say your project wants to display the different between two datetimes as "5 minutes ago" or "last month" (in several languages). Saying "1.27 days ago" is correct to an extent but also not pretty. There exist high quality libraries for this, but not in every language, and a lot of devs would stumble on this feature. The other is UI creation, I would say it is easier and faster to create a really nice UI with HTML/CSS/JS than with native Android. Maybe I'm wrong on this example but I bet there is a pair of UI frameworks where this is true.
ggregoire 7y ago
Casual observation: adding a framework to any language seems to improve the overall code quality. It adds and teaches structure, patterns and good practices that developers usually follow. I've seen terrible codebases written in vanilla PHP, JavaScript, Python (probably the worst ones). But I've seen very good looking, easy to understand, easy to maintain codebases written in Symfony, React and Django. (React is an interesting case because it's not actually a framework but a view library, which doesn't force you to follow any structure to your project. But its declarative, component-based paradigm seem to help developers structuring their projects and writing simple, reusable and composable components in a good way).
- tabtab 7y ago
  I agree. Our framework, libraries, and/or interfaces can and should make a bigger difference than programming language. If you have to use the "weird parts" of a language to get typical work done, you are doing something wrong. And if you are not using the weird parts, then programming languages look and do pretty much the same thing. Braggings such as, "look! My language can do double recursive lambda backflips while blindfolded and chewing gum!" are mostly just 1950's car racing done in cubicles. We buy cars to get to work and shops as cheaply and conveniently as possible, not beat Fonzy.
  - syastrov 7y ago
    I think it really depends on how you use the “weird parts”. Some of those let you write code much more concisely and effectively. Used in the wrong context, it’s a unnecessary distraction. It also depends on if you want to write “typical code” or code that the competition can’t match. See Paul Graham’s essay on the Blub Paradox [0], which discusses why some languages seem to have “weird parts” from the perspective of someone who knows a less-powerful language. [0] http://www.paulgraham.com/avg.html http://www.paulgraham.com/avg.html
    - tabtab 7y ago
      You make it hard to hire if you limit the field to those who know the esoteric techniques. Programming itself is mostly a dead-end career; you either move into management or quasi-management, or face ageism on average. There is thus not a lot of incentive to "perfect the art".
gilbetron 7y ago
Unfortunately, it's the original paper that will keep getting quoted at me forever. My favorite line from this paper is: "Correlation is not causality, but it is tempting to confuse them." I think most people believe "correlation is not causality, but most of the time it is", when in reality it is, "correlation is not causality, and almost never is." There was a great analysis of this in a book or paper I read years ago that I haven't been able to find, but the tldr is that if you have a set of events, if causality were truly common, the events would for a seized up network of causality.
baby 7y ago
I’ve read a lot of code as a consultant. The best codebase I’ve read had almost no comments and was written in erlang. Besides that every codebase I’ve read in Golang were super clear and of excellent quality for the exception of one that was written by java developers. I can say I hated reading C because of maccros and different build systems. I hated C++ codebases because people abused generics. And I hated Java the most because of the verbosity and the directory structure and all the layers of abstractions and all the factories and all the singletons and all the...
- Insanity 7y ago
  That is always a risk if a team decides to pick up a new language but does not make the effort to learn the language properly. At work we are going through a similar Java - Go transition, so I know :(
  - noir_lord 7y ago
    It's always a problem when people shoe-horn a non-idiomatic approach from one language into another. It makes me wish the language creators wrote a single none-trivial 'this is how we intended it to look' project. When learning a new language I find as many big projects written in it as I can and spend time exploring them. It helps but there is often no guarantee those projects are doing it idiomatically either.
    - chubot 7y ago
      Looking through the Python standard library or Go standard library should be a clue not to use Java idioms, but yes I realize those habits are hard to break. I have seen Java-in-Python and it is indeed painful.
      - noir_lord 7y ago
        The problem with reading framework and library code is that it's very coupled to what it is, so while it's interesting and valuable to read it's not really the type of code you'd see in a business app or whatever.
kazinator 7y ago
The problem is that "code quality" isn't just bugs per line of code. Maintainability, extensibility, readability, expressivity all factor into it. Suppose I have to write 100,000 lines in language B to write something doable in 10,000 lines of A. Who cares if the defect rate is twice as high per line of A: I still have five times fewer defects due to ten times fewer LOC's. Another thing to consider is the nature of defects. A defect that allows a remote execution exploit is not the same as an incorrect result being ignored and propagating to later calculations, is not the same as terminating with a diagnostic.
- caust1c 7y ago
  Very much agree. This type of analysis is hard to quantify but anecdotally I don't know many people who have written Go, Rust or Typescript that would prefer python, ruby, java, javascript or c++ for their daily work after using them.
- imtringued 7y ago
  If the defect rate is the same across languages then you're not measuring the defect rate of the programming language. You're measuring the defect rate of the developers!
robbrit 7y ago
Seems like they're just doing associations, and not controlling for a lot of factors. For example, you can come up with some narratives to explain a lot of these findings: * Python, JS, PHP, Java, C++ are all very common first languages; I'd imagine the average experience level for these is dragged down a bit compared to others like Golang, Haskell, or Scala. One of the recent Stack Overflow Developer surveys found Golang use to be strongly correlated with age, and since I'm assuming bug count would negatively correlate with age you'd see a confounding factor here. * Ruby's community has an obsessive culture towards unit testing, that could impact bug levels. * The trade-offs that make C and C++ attractive such as memory performance are double-edged swords and will naturally correlate with more bugs. For example, I work a fair bit in embedded systems which use MISRA C out of necessity; I've also worked in large scale data processing which chooses C++ for its memory mapping performance over Java or Go. If "number of defects" is your primary measure then you're ignoring some of the reasons languages are chosen. Sometimes a language that minimizes bugs is not a viable option. * I'd guess there is a skill correlation here: an engineer who is willing to seek out a specific less-used language and learn to use it effectively is probably more likely (not guaranteed) to do a better job than one who is just going with the default choice. It'd be very interesting to see a regression done that accounts for these factors.
- cestith 7y ago
  That would be interesting. I also have a feeling it would be incredibly difficult to accurately control for social differences in language communities.
  - robbrit 7y ago
    Yeah you'd have to take them one-by-one and see if there is some proxy if you can't measure the behaviour directly. For example, with testing you could use code-coverage or even a binary variable for the existence of tests - not perfect, but would give better results than no control at all.
  - shados 7y ago
    Definitely. Heck, in .NET's early days, there was a distinct difference in quality of the software and of the team using VB.net vs C#, when they were 99% the same language (they had some features unique to each, but they were very few). But C# was used a lot in new projects or by people familiar with Java while VB.net was used in VB6 migrations or teams coming from VB6. That had a drastic effect on the end result, but the language had very little to do with it.
alkonaut 7y ago
The language is what makes me apply to work at your company with 15 years experience but paid 20% less than I would be if I had to do OO. That’s the driver. Is the code quality better? Well maybe. Probably. But dollar for dollar you’ll do better with a good language because you’ll get much better people for your dollar.
yogthos 7y ago
It looks like the key takeaways are: Overall language effects do not appear to be a dominant factor in software quality. Manual memory management is error prone. Functional languages appear to produce better results than OO/imperaive ones. Static typing does not appear to play a measurable role. In fact, Clojure and Erlang were in a category of their own, beating out statically typed counterparts.
- Skunkleton 7y ago
  > Clojure and Erlang were in a category of their own I wonder how they controlled for developer skill?
  - yogthos 7y ago
    It's likely that Clojure and Erlang communities have a higher ratio of experienced developers than mainstream languages. However, this would be true for Scala and Haskell communities as well presumably. Meanwhile, the fact that experienced developers choosing these languages appear to produce better code is an interesting data point of itself.
tsegratis 7y ago
I wonder if this lack of significant link is because number_of_bugs = time_spent_programming / time_fixing_bugs So choice of language does affect the number of potential bugs. But this ratio of effort a programmer gives to bugs vs features is almost completely independent. Actual bugs making it past the programmer depend on how much effort the programmer cares to expend So a safer language doesn't reduce bugs. It's like farmers with back pain from driving tractors. Giving them comfier rides doesn't reduce their pain; it just means they drive faster upto the same pain limit If this is true, then bugs can be reduced by making bugs more painful for the programmer But also a safer language will let the programmer drive at a higher speed, releasing a greater percentage of safe code (even if the total number of bugs is the same)
nullc 7y ago
Good to see better statistics being applied. However, the base methodology appears to be really weak. The presence of fixes doesn't necessarily reflect the presence of underlying bugs: highly conscientious projects make many "fixes" that improve the software quality against an abstract ideal without directly fixing any defect, while highly unconscientious projects don't bother fixing anything. There are also many confounders. For example, I've seen project institute restrictions on refactoring changes in response to the huge amount of brownian-code-motion popular projects can receive, only to then have some contributors start misleading describing their commits, "by splitting this function up, we avoid the risk that someone would later introduce an overflow!". Considering how important software quality is to contemporary engineering, including life safety critical systems, it's disappointing that we don't see more randomize studies. E.g. take a pool of developers randomly assign them to teams using different tools each to accomplish the same task, then compare the results against each other and a hidden test suite.
pron 7y ago
The most practically relevant finding of the original paper remains in effect: the choice of language has an effect of less than 1% on the bug rate[1]. The original paper explains: "With effects coding, each coefficient indicates the relative effect of the use of a particular language on the response as compared to the weighted mean of the dependent variable across all projects." So the entire discussion is on the relative effect of particular languages relative to the total effect of language choice, which is less than 1%. The original paper reported stronger relative effects for particular languages, while this one found the effects to be smaller, and in many cases they disappeared altogether, thus summarizing the effects of the languages as "exceedingly small." So even the comparison of C++ vs. Haskell, that has ~25% difference, is within that 1%. This is similar to a study comparing the effect of running shoes on running speed that shows that Nike positively affects your running speed 25% more than Adidas, to the extent running shoes affect your speed at all, which is less than 1%. ----- [1] One should take care not to overestimate the impact of language on defects. While these relationships are statistically significant, the effects are quite small. In the analysis of deviance... we see that activity in a project accounts for the majority of explained deviance... The next closest predictor, which accounts for less than one percent of the total deviance, is language. (https://web.cs.ucdavis.edu/~filkov/papers/lang_github.pdf https://web.cs.ucdavis.edu/~filkov/papers/lang_github.pdf)
- scott_s 7y ago
  I disagree that the combination of the two studies supports an effect of language choice on bug rate. Given that the original study showed a negative association for about half the languages; the replication has significant disagreement on about half of the original positive associations; and the extremely small associations across all languages, I think we're looking at noise.
  - pron 7y ago
    But if we consider as the main result that the total effect of language choice is less than 1%, even "looking at noise" means looking at an effect of 0, and either way, the effect is tiny to nonexistent. So if someone had the hypothesis that the effect of language choice is large, because lack of evidence of a big effect is evidence of lack of one, this study is some evidence against such a hypothesis. That this finding agrees with the observations in the industry lends it even more credence.

264 points by lelf 7y ago

This is great (to see something I know about on HN). As an undergrad at UMass (where Emery Berger teaches), I remember the exact paper they are talking about, and how it was presented at UMass (this was part of a lunch and learn with the faculty, grad, and undergrad students and the author of the original paper). I remember Emery being quite confused at the methodology in the paper, and he was very confused with one aspect of it, which was how they came to conclude that C was more bug prone (something about void * being able to cast to everything? It was a while ago...) Glad to see he wrote a rebuttal, a little awkward watching a professor debate with the paper's author :)

mcguire 7y ago
I knew Emery at UT Austin; he's a good researcher, but I'd imagine "awkward" is an understatement.

melling 7y ago

...deleted I skimmed the paper. I misunderstood the results.

smoe 7y ago
Your quote was the conclusion of the original work by Ray et al., not the one from the reproduction paper linked here. That one in short is: "Unfortunately, our work has identified numerous problems in the FSE study that invalidated its key result."

voidhorse 7y ago

Interesting! Excepting typescript, the “safer” languages also happen to be the ones I gravitate towards on a daily basis and actually enjoy writing programs in, while the others... It’d be interesting to see whether programmers’ preferences uphold the division posited by the study, or whether they are widely variable—eg is it likely that if you prefer ruby you’ll also prefer typescript over JavaScript, if you prefer Haskell you’d choose clojure over c++ etc.

scott_s 7y ago
I think you're confusing the conclusions from the original paper with the conclusions from this paper. This paper is a reproduction of the original study, and for the most part, it is not able to reproduce the findings.
- voidhorse 7y ago
  How right you are. That’s what I get for skimming news piecemeal way too quickly in the mornings!

panpanna 7y ago

Maybe off topic, but isn't the language in the abstract a bit aggressive? I mean, this type of study is somewhat open to interpretation. It's not like everyone should arrive to the exact same numbers..

omginternets 7y ago
Which part? This kind of direct language is both standard and encouraged in academia. The idea is to take a clear position, which can be attacked if wrong.
idoubtit 7y ago
The conclusion makes it clear that this work has no aggressive intent: Unfortunately, our work has identified numerous problems in the FSE study that invalidated its key result. Our intent is not to blame, performing statistical analysis of programming languages based on large-scale code repositories is hard. [...] Acknowledgments. We thank Baishakhi Ray and Vladimir Filkov for sharing the data and code of their FSE paper. Had they not preserved the original files and part of their code, reproduction would have been more challenging

jph 7y ago

(Edited-- I was reading the original paper, which showed good results for TypeScript; the new paper linked above refutes that result. My mistake. Thanks to the posters who corrected me!) The winners in this paper are Clojure, Haskell, Scala. I look forward to seeing Rust added. The authors go into substantial detail about what they consider errors, and in my personal experience Rust solves many of them. See e.g. "Some defect type like memory error, concurrency errors also depend on language primitives."

vidro3 7y ago
How so? Typescript, along with most other languages is listed as 'No Statistically Significant Association" see fig 2. ETA: They actually did not assess Typescript in their reanalysis as most of the files were not actually Typescript, section 4.1.2
faitswulff 7y ago
That's not at all what I got from the abstract: > ...only four languages are found to have a statistically significant association with defects, and even for those the effect size is exceedingly small. Emphasis mine.
scott_s 7y ago
I believe you are confusing the conclusions from the original paper with the claims from this paper. I think this is the case because this paper removes TypeScript from their dataset entirely: "4.1.2 Removal of TypeScript. In the original dataset, the first commit for TypeScript was recorded on 2003-03-21, several years before the language was created. Upon inspection, we found that the file extension .ts is used for XML files containing human language translations. Out of 41 projects labeled as TypeScript, only 16 contained TypeScript. This reduced the number of commits from 10,063 to an even smaller 3,782. Unfortunately, the three largest remaining projects (typescript-node-definitions, DefinitelyTyped, and the deprecated tsd) contained only declarations and no code. They accounted for 34.6% of the remaining TypeScript commits. Given the small size of the remaining corpus, we removed it from consideration as it is not clear that we have sufficient data to draw useful conclusions."
xvilka 7y ago

salmonellaeater 7y ago

Of note is that these are all associations. It's possible the associations are all due to selection effects (or even that the "better" language are actually worse but their effects are reversed by selecting for programmers who are better). In the early days of Python, some employers used it as a way to filter for candidates who were ahead of the curve. I believe Paul Graham mentioned Lisp being good for attracting better programmers, although I can't find the reference anymore [1]. If there's any value in this kind of filter, the effects will show up in the programmers' output. 1. I'm having a hard time finding references for these assertions; the best I've found is some Reddit posts and Joel Spolsky's post on finding great developers: https://www.joelonsoftware.com/2006/09/06/finding-great-developers-2/ https://www.joelonsoftware.com/2006/09/06/finding-great-deve...

deleted 7y ago
[deleted]
hannofcart 7y ago
Indeed, and in addition to selection bias towards better programmers, there is another selection bias implicit in this. A shop that makes the decision to write their stack in Haskell do it with an intention to focus on correctness. I reckon that such companies would consequently invest in other practices like more tests, higher code coverage and more intensive code reviews resulting in code with fewer errors. This is in stark contrast to a startup that may have hacked together a POC in Python, and once it worked, with a little bit more spit and polish put it into production.
notacoward 7y ago
> It's possible the associations are all due to selection effects FWIW, the original authors (Ray et al) said as much.
goto11 7y ago
The paper goes further though. It states: Not only is it not possible to establish a causal link between programming language and code quality based on data at hand, but even their correlation proves questionable. I find that a surprising result! This might also call into question the theory that some languages attract better developers.

idoubtit 7y ago

My own take of this: - This study tries to enhance the original work by better metrics (e.g. confusion with C/C++), using better statistics (e.g. uncertainty). - Automatically evaluating the quality of a code is hard and time-consuming. They had to use peer review of labellisation. - Most of the old claims are not confirmed. - Only 2 languages in this study had both a significant impact (impact coeff) and enough fiability (p-value): Clojure and Haskell. But the impact is still small, and there may be other influencing factors. - They propose several ways to further enhance their work (e.g. taking regression tests into account).

msluyter 7y ago
"... and there may be other influencing factors." Selection bias? Consider the type of person likely to be attracted to languages like Haskell or Clojure. (Note, didn't read the original study so I don't know the selection criteria. Were code samples in different languages from the same programmers compared, or were there different sets of programmers per language? My comment really only applies to the latter case.)
- sidlls 7y ago
  What is the type of person likely to be attracted to languages like Haskell or Clojure?
  - maxkwallace 7y ago
    I can think of selection bias effects both in-favor of more defects and against it. On one hand, Haskell or Clojure probably attract more academic types who have a reputation for (consistent with my experience) on-average worse-quality code. On other hand, the barrier-to-entry for these languages is higher so they tend to be programmed by more experienced folks. I assume that people pick functional languages in part because they believe the functional paradigm leads to higher-quality code, so the sample probably cares more about code quality to begin with.
    - 7y ago

dahart 7y ago

One of the simplest but more important conclusions that affects me all the time... it’s easy to search for things, but hard to know whether you’ve got all the answers you want and hard to know if all the answers you got are answers you want. 6.3 Grep considered harmful Simple analysis techniques may be too blunt to provide useful answers. This problem was compounded by the fact that the search for keywords did not look for words and instead captured substrings wholly unrelated to software defects. When the accuracy of classification is as low as 36%, it becomes difficult to argue that results with small effect sizes are meaningful as they may be indistinguishable from noise. If such classification techniques are to be employed, then a careful post hoc validation by hand should be conducted by domain experts.

stcredzero 7y ago
One of the simplest but more important conclusions that affects me all the time... it’s easy to search for things, but hard to know whether you’ve got all the answers you want and hard to know if all the answers you got are answers you want. Smalltalk was wonderful in this regard. If you wanted to search for how something was used, you just right-click search for "senders." If you wanted to find implementors, the same. This big difference occurred with chains of senders. In other languages, where there's 2 or 3 ways to get to something or call something, you have this O(2^n) or O(2.5^n) blowup in the work you have to do to properly follow chains of senders. With Smalltalk, it's just O(n). On top of that, there were tools that could let you compose really sophisticated SQL like queries of the code base, dynamically, then pop that up in a browser.

coldcode 7y ago

Testing Objective-C is sort of pointless, they should have picked Swift.

chriswarbo 7y ago
Apple didn't invent Objective-C, it just-so-happened to be the language of choice at Next, which carried on into OSX and iOS. The fact that Swift is now Apple's language of choice doesn't make analysis of Objective-C "sort of pointless", any more than e.g. Facebook creating Hack would make anything involving PHP "sort of pointless".
pmiller2 7y ago
I disagree. If anything, they should have tested Obj-C and Swift. I would also have liked to see Scheme, Common Lisp, or something else lisp-y besides Clojure. Kotlin would have been interesting. R or Mathematica would have been fun, too. I guess it just comes down to needing a sufficiently large corpus of code and commits and having to eventually publish.
- deleted 7y ago
  [deleted]

scott_s 7y ago

A common thread I'm seeing in the discussion here is confusing the claims from the original paper with conclusions made by this paper. This is a replication study, and it starts by stating what the original paper found. What the original paper found is not what this paper finds. For the most part, this is a replication failure, for a myriad of reasons, including several kinds of misclassification. From this paper: The reanalysis failed to validate most of the claims of [18]. As Table 6(d-f) shows, the multiple steps of data cleaning and improved statistical modeling have invalidated the significance of 7 out of 11 languages. Even when the associations are statistically significant, their practical significance is small. The reference [18] is https://dl.acm.org/citation.cfm?doid=2635868.2635922 https://dl.acm.org/citation.cfm?doid=2635868.2635922, which is the original paper that this paper is a replication study of.

narag 7y ago
So the old adage stands: you can write crap in any language.
- TheOtherHobbes 7y ago
  And if you do, chances are good it'll end up on GitHub. 5.3 and 5.4 really nail it, IMO. And you can't even use bug fixes as a metric, because there's a difference between semantic bugs which will be significantly language-influenced, architectural bugs which are partly language-influenced, and user-expectation/experience bugs, which are broad-brush logical errors or misunderstandings. (There are probably other categories, but those are the obvious ones I can think of right now.) The real takeaway is there is as yet no objective metric for software quality. The best a working dev shop can do is implement a small but non-trivial sample project with known-good logic in a handful of candidate languages, and see what kind of trade-offs fall out.
  - jrumbut 7y ago
    I'll agree with you for sure no objective metric for quality (that is broad enough to capture a satisfying amount of what we mean by quality). I do think user experience/expectation bugs are very language dependent though. First you have the quality of the String and Datetime libraries. Ruby has all kinds of high quality string functions that are a source of pain and bugs in C. Say your project wants to display the different between two datetimes as "5 minutes ago" or "last month" (in several languages). Saying "1.27 days ago" is correct to an extent but also not pretty. There exist high quality libraries for this, but not in every language, and a lot of devs would stumble on this feature. The other is UI creation, I would say it is easier and faster to create a really nice UI with HTML/CSS/JS than with native Android. Maybe I'm wrong on this example but I bet there is a pair of UI frameworks where this is true.

ggregoire 7y ago

Casual observation: adding a framework to any language seems to improve the overall code quality. It adds and teaches structure, patterns and good practices that developers usually follow. I've seen terrible codebases written in vanilla PHP, JavaScript, Python (probably the worst ones). But I've seen very good looking, easy to understand, easy to maintain codebases written in Symfony, React and Django. (React is an interesting case because it's not actually a framework but a view library, which doesn't force you to follow any structure to your project. But its declarative, component-based paradigm seem to help developers structuring their projects and writing simple, reusable and composable components in a good way).

tabtab 7y ago
I agree. Our framework, libraries, and/or interfaces can and should make a bigger difference than programming language. If you have to use the "weird parts" of a language to get typical work done, you are doing something wrong. And if you are not using the weird parts, then programming languages look and do pretty much the same thing. Braggings such as, "look! My language can do double recursive lambda backflips while blindfolded and chewing gum!" are mostly just 1950's car racing done in cubicles. We buy cars to get to work and shops as cheaply and conveniently as possible, not beat Fonzy.
- syastrov 7y ago
  I think it really depends on how you use the “weird parts”. Some of those let you write code much more concisely and effectively. Used in the wrong context, it’s a unnecessary distraction. It also depends on if you want to write “typical code” or code that the competition can’t match. See Paul Graham’s essay on the Blub Paradox [0], which discusses why some languages seem to have “weird parts” from the perspective of someone who knows a less-powerful language. [0] http://www.paulgraham.com/avg.html http://www.paulgraham.com/avg.html
  - tabtab 7y ago
    You make it hard to hire if you limit the field to those who know the esoteric techniques. Programming itself is mostly a dead-end career; you either move into management or quasi-management, or face ageism on average. There is thus not a lot of incentive to "perfect the art".

gilbetron 7y ago

Unfortunately, it's the original paper that will keep getting quoted at me forever. My favorite line from this paper is: "Correlation is not causality, but it is tempting to confuse them." I think most people believe "correlation is not causality, but most of the time it is", when in reality it is, "correlation is not causality, and almost never is." There was a great analysis of this in a book or paper I read years ago that I haven't been able to find, but the tldr is that if you have a set of events, if causality were truly common, the events would for a seized up network of causality.

baby 7y ago

I’ve read a lot of code as a consultant. The best codebase I’ve read had almost no comments and was written in erlang. Besides that every codebase I’ve read in Golang were super clear and of excellent quality for the exception of one that was written by java developers. I can say I hated reading C because of maccros and different build systems. I hated C++ codebases because people abused generics. And I hated Java the most because of the verbosity and the directory structure and all the layers of abstractions and all the factories and all the singletons and all the...

Insanity 7y ago
That is always a risk if a team decides to pick up a new language but does not make the effort to learn the language properly. At work we are going through a similar Java - Go transition, so I know :(
- noir_lord 7y ago
  It's always a problem when people shoe-horn a non-idiomatic approach from one language into another. It makes me wish the language creators wrote a single none-trivial 'this is how we intended it to look' project. When learning a new language I find as many big projects written in it as I can and spend time exploring them. It helps but there is often no guarantee those projects are doing it idiomatically either.
  - chubot 7y ago
    Looking through the Python standard library or Go standard library should be a clue not to use Java idioms, but yes I realize those habits are hard to break. I have seen Java-in-Python and it is indeed painful.
    - noir_lord 7y ago
      The problem with reading framework and library code is that it's very coupled to what it is, so while it's interesting and valuable to read it's not really the type of code you'd see in a business app or whatever.

kazinator 7y ago

The problem is that "code quality" isn't just bugs per line of code. Maintainability, extensibility, readability, expressivity all factor into it. Suppose I have to write 100,000 lines in language B to write something doable in 10,000 lines of A. Who cares if the defect rate is twice as high per line of A: I still have five times fewer defects due to ten times fewer LOC's. Another thing to consider is the nature of defects. A defect that allows a remote execution exploit is not the same as an incorrect result being ignored and propagating to later calculations, is not the same as terminating with a diagnostic.

caust1c 7y ago
Very much agree. This type of analysis is hard to quantify but anecdotally I don't know many people who have written Go, Rust or Typescript that would prefer python, ruby, java, javascript or c++ for their daily work after using them.
imtringued 7y ago
If the defect rate is the same across languages then you're not measuring the defect rate of the programming language. You're measuring the defect rate of the developers!

robbrit 7y ago

Seems like they're just doing associations, and not controlling for a lot of factors. For example, you can come up with some narratives to explain a lot of these findings: * Python, JS, PHP, Java, C++ are all very common first languages; I'd imagine the average experience level for these is dragged down a bit compared to others like Golang, Haskell, or Scala. One of the recent Stack Overflow Developer surveys found Golang use to be strongly correlated with age, and since I'm assuming bug count would negatively correlate with age you'd see a confounding factor here. * Ruby's community has an obsessive culture towards unit testing, that could impact bug levels. * The trade-offs that make C and C++ attractive such as memory performance are double-edged swords and will naturally correlate with more bugs. For example, I work a fair bit in embedded systems which use MISRA C out of necessity; I've also worked in large scale data processing which chooses C++ for its memory mapping performance over Java or Go. If "number of defects" is your primary measure then you're ignoring some of the reasons languages are chosen. Sometimes a language that minimizes bugs is not a viable option. * I'd guess there is a skill correlation here: an engineer who is willing to seek out a specific less-used language and learn to use it effectively is probably more likely (not guaranteed) to do a better job than one who is just going with the default choice. It'd be very interesting to see a regression done that accounts for these factors.

cestith 7y ago
That would be interesting. I also have a feeling it would be incredibly difficult to accurately control for social differences in language communities.
- robbrit 7y ago
  Yeah you'd have to take them one-by-one and see if there is some proxy if you can't measure the behaviour directly. For example, with testing you could use code-coverage or even a binary variable for the existence of tests - not perfect, but would give better results than no control at all.
- shados 7y ago
  Definitely. Heck, in .NET's early days, there was a distinct difference in quality of the software and of the team using VB.net vs C#, when they were 99% the same language (they had some features unique to each, but they were very few). But C# was used a lot in new projects or by people familiar with Java while VB.net was used in VB6 migrations or teams coming from VB6. That had a drastic effect on the end result, but the language had very little to do with it.

alkonaut 7y ago

The language is what makes me apply to work at your company with 15 years experience but paid 20% less than I would be if I had to do OO. That’s the driver. Is the code quality better? Well maybe. Probably. But dollar for dollar you’ll do better with a good language because you’ll get much better people for your dollar.

yogthos 7y ago

It looks like the key takeaways are: Overall language effects do not appear to be a dominant factor in software quality. Manual memory management is error prone. Functional languages appear to produce better results than OO/imperaive ones. Static typing does not appear to play a measurable role. In fact, Clojure and Erlang were in a category of their own, beating out statically typed counterparts.

Skunkleton 7y ago
> Clojure and Erlang were in a category of their own I wonder how they controlled for developer skill?
- yogthos 7y ago
  It's likely that Clojure and Erlang communities have a higher ratio of experienced developers than mainstream languages. However, this would be true for Scala and Haskell communities as well presumably. Meanwhile, the fact that experienced developers choosing these languages appear to produce better code is an interesting data point of itself.

tsegratis 7y ago

I wonder if this lack of significant link is because number_of_bugs = time_spent_programming / time_fixing_bugs So choice of language does affect the number of potential bugs. But this ratio of effort a programmer gives to bugs vs features is almost completely independent. Actual bugs making it past the programmer depend on how much effort the programmer cares to expend So a safer language doesn't reduce bugs. It's like farmers with back pain from driving tractors. Giving them comfier rides doesn't reduce their pain; it just means they drive faster upto the same pain limit If this is true, then bugs can be reduced by making bugs more painful for the programmer But also a safer language will let the programmer drive at a higher speed, releasing a greater percentage of safe code (even if the total number of bugs is the same)

nullc 7y ago

Good to see better statistics being applied. However, the base methodology appears to be really weak. The presence of fixes doesn't necessarily reflect the presence of underlying bugs: highly conscientious projects make many "fixes" that improve the software quality against an abstract ideal without directly fixing any defect, while highly unconscientious projects don't bother fixing anything. There are also many confounders. For example, I've seen project institute restrictions on refactoring changes in response to the huge amount of brownian-code-motion popular projects can receive, only to then have some contributors start misleading describing their commits, "by splitting this function up, we avoid the risk that someone would later introduce an overflow!". Considering how important software quality is to contemporary engineering, including life safety critical systems, it's disappointing that we don't see more randomize studies. E.g. take a pool of developers randomly assign them to teams using different tools each to accomplish the same task, then compare the results against each other and a hidden test suite.

pron 7y ago

The most practically relevant finding of the original paper remains in effect: the choice of language has an effect of less than 1% on the bug rate[1]. The original paper explains: "With effects coding, each coefficient indicates the relative effect of the use of a particular language on the response as compared to the weighted mean of the dependent variable across all projects." So the entire discussion is on the relative effect of particular languages relative to the total effect of language choice, which is less than 1%. The original paper reported stronger relative effects for particular languages, while this one found the effects to be smaller, and in many cases they disappeared altogether, thus summarizing the effects of the languages as "exceedingly small." So even the comparison of C++ vs. Haskell, that has ~25% difference, is within that 1%. This is similar to a study comparing the effect of running shoes on running speed that shows that Nike positively affects your running speed 25% more than Adidas, to the extent running shoes affect your speed at all, which is less than 1%. ----- [1] One should take care not to overestimate the impact of language on defects. While these relationships are statistically significant, the effects are quite small. In the analysis of deviance... we see that activity in a project accounts for the majority of explained deviance... The next closest predictor, which accounts for less than one percent of the total deviance, is language. (https://web.cs.ucdavis.edu/~filkov/papers/lang_github.pdf https://web.cs.ucdavis.edu/~filkov/papers/lang_github.pdf)

scott_s 7y ago
I disagree that the combination of the two studies supports an effect of language choice on bug rate. Given that the original study showed a negative association for about half the languages; the replication has significant disagreement on about half of the original positive associations; and the extremely small associations across all languages, I think we're looking at noise.
- pron 7y ago
  But if we consider as the main result that the total effect of language choice is less than 1%, even "looking at noise" means looking at an effect of 0, and either way, the effect is tiny to nonexistent. So if someone had the hypothesis that the effect of language choice is large, because lack of evidence of a big effect is evidence of lack of one, this study is some evidence against such a hypothesis. That this finding agrees with the observations in the industry lends it even more credence.