5 ms·

A common thread I'm seeing in the discussion here is confusing the claims from the original paper with conclusions made by this paper. This is a replication stu

A common thread I'm seeing in the discussion here is confusing the claims from the original paper with conclusions made by this paper. This is a replication study, and it starts by stating what the original paper found. What the original paper found is not what this paper finds. For the most part, this is a replication failure, for a myriad of reasons, including several kinds of misclassification. From this paper: The reanalysis failed to validate most of the claims of [18]. As Table 6(d-f) shows, the multiple steps of data cleaning and improved statistical modeling have invalidated the significance of 7 out of 11 languages. Even when the associations are statistically significant, their practical significance is small. The reference [18] is https://dl.acm.org/citation.cfm?doid=2635868.2635922 https://dl.acm.org/citation.cfm?doid=2635868.2635922, which is the original paper that this paper is a replication study of.

narag 7y ago
So the old adage stands: you can write crap in any language.
- TheOtherHobbes 7y ago
  And if you do, chances are good it'll end up on GitHub. 5.3 and 5.4 really nail it, IMO. And you can't even use bug fixes as a metric, because there's a difference between semantic bugs which will be significantly language-influenced, architectural bugs which are partly language-influenced, and user-expectation/experience bugs, which are broad-brush logical errors or misunderstandings. (There are probably other categories, but those are the obvious ones I can think of right now.) The real takeaway is there is as yet no objective metric for software quality. The best a working dev shop can do is implement a small but non-trivial sample project with known-good logic in a handful of candidate languages, and see what kind of trade-offs fall out.
  - jrumbut 7y ago
    I'll agree with you for sure no objective metric for quality (that is broad enough to capture a satisfying amount of what we mean by quality). I do think user experience/expectation bugs are very language dependent though. First you have the quality of the String and Datetime libraries. Ruby has all kinds of high quality string functions that are a source of pain and bugs in C. Say your project wants to display the different between two datetimes as "5 minutes ago" or "last month" (in several languages). Saying "1.27 days ago" is correct to an extent but also not pretty. There exist high quality libraries for this, but not in every language, and a lot of devs would stumble on this feature. The other is UI creation, I would say it is easier and faster to create a really nice UI with HTML/CSS/JS than with native Android. Maybe I'm wrong on this example but I bet there is a pair of UI frameworks where this is true.
  - mark-r 7y ago
    > user-expectation/experience bugs I consider those to be bugs in the spec, not the code. I once worked at a place that simply swept those under the rug - if it worked as documented, it didn't matter how many users couldn't get it to work. The bug report was closed as "not a bug"; you can imagine how frustrating that was.
pmiller2 7y ago
Yes, definitely. This is mostly a non-replication of the original study. I was a little disappointed that they didn't look at defects per line of code, even though that's already a fairly well-studied metric. I think the lesson here is that you can write good or bad code in any language, but choose one that's fit for purpose. Using proper abstractions (including those builtin to languages) probably does more to reduce the incidence of bugs than anything you can do other than writing automated tests.
- neves 7y ago
  The practice that most reduces the incidence of tests is code review: https://kevin.burke.dev/kevin/the-best-ways-to-find-bugs-in-your-code/ https://kevin.burke.dev/kevin/the-best-ways-to-find-bugs-in-...
  - JamesBarney 7y ago
    Emphasis on "Formal code review" which is practiced in far fewer shops than "code review". For those who don't happen to know the difference, formal code review basically uses a check list of common best practices and possible defects to look for versus doing each code review ad-hoc.
    - codazoda 7y ago
      I thrive on checklists and hate the ad-hoc way I do code reviews on my team. This sounds great, I'm going to go look for some basic formal checklists. I need language agnostic versions and/or versions for javascript, php, and python, if anyone has suggestions.
      - treebog 7y ago
        The Art of Software Testing by Glenford Myers has an excellent language-agnostic checklist. See the chapter on “walkthroughs”
        7y ago
brlewis 7y ago
tl;dr Reanalysis showed positive association between defects and C++, negative association for Clojure, Haskell and Ruby. The original analysis found positive association for C++, Objective-C, C, PHP, Python, JavaScript. Negative association for TypeScript, Clojure, Scala, Haskell, Ruby.
- DannyBee 7y ago
  It also found the effect is very small in all cases.

narag 7y ago

So the old adage stands: you can write crap in any language.

TheOtherHobbes 7y ago
And if you do, chances are good it'll end up on GitHub. 5.3 and 5.4 really nail it, IMO. And you can't even use bug fixes as a metric, because there's a difference between semantic bugs which will be significantly language-influenced, architectural bugs which are partly language-influenced, and user-expectation/experience bugs, which are broad-brush logical errors or misunderstandings. (There are probably other categories, but those are the obvious ones I can think of right now.) The real takeaway is there is as yet no objective metric for software quality. The best a working dev shop can do is implement a small but non-trivial sample project with known-good logic in a handful of candidate languages, and see what kind of trade-offs fall out.
- jrumbut 7y ago
  I'll agree with you for sure no objective metric for quality (that is broad enough to capture a satisfying amount of what we mean by quality). I do think user experience/expectation bugs are very language dependent though. First you have the quality of the String and Datetime libraries. Ruby has all kinds of high quality string functions that are a source of pain and bugs in C. Say your project wants to display the different between two datetimes as "5 minutes ago" or "last month" (in several languages). Saying "1.27 days ago" is correct to an extent but also not pretty. There exist high quality libraries for this, but not in every language, and a lot of devs would stumble on this feature. The other is UI creation, I would say it is easier and faster to create a really nice UI with HTML/CSS/JS than with native Android. Maybe I'm wrong on this example but I bet there is a pair of UI frameworks where this is true.
- mark-r 7y ago
  > user-expectation/experience bugs I consider those to be bugs in the spec, not the code. I once worked at a place that simply swept those under the rug - if it worked as documented, it didn't matter how many users couldn't get it to work. The bug report was closed as "not a bug"; you can imagine how frustrating that was.

pmiller2 7y ago

Yes, definitely. This is mostly a non-replication of the original study. I was a little disappointed that they didn't look at defects per line of code, even though that's already a fairly well-studied metric. I think the lesson here is that you can write good or bad code in any language, but choose one that's fit for purpose. Using proper abstractions (including those builtin to languages) probably does more to reduce the incidence of bugs than anything you can do other than writing automated tests.

neves 7y ago
The practice that most reduces the incidence of tests is code review: https://kevin.burke.dev/kevin/the-best-ways-to-find-bugs-in-your-code/ https://kevin.burke.dev/kevin/the-best-ways-to-find-bugs-in-...
- JamesBarney 7y ago
  Emphasis on "Formal code review" which is practiced in far fewer shops than "code review". For those who don't happen to know the difference, formal code review basically uses a check list of common best practices and possible defects to look for versus doing each code review ad-hoc.
  - codazoda 7y ago
    I thrive on checklists and hate the ad-hoc way I do code reviews on my team. This sounds great, I'm going to go look for some basic formal checklists. I need language agnostic versions and/or versions for javascript, php, and python, if anyone has suggestions.
    - treebog 7y ago
      The Art of Software Testing by Glenford Myers has an excellent language-agnostic checklist. See the chapter on “walkthroughs”
      - 7y ago

brlewis 7y ago

tl;dr Reanalysis showed positive association between defects and C++, negative association for Clojure, Haskell and Ruby. The original analysis found positive association for C++, Objective-C, C, PHP, Python, JavaScript. Negative association for TypeScript, Clojure, Scala, Haskell, Ruby.

DannyBee 7y ago
It also found the effect is very small in all cases.