3 ms·

> this evidence has been around for ages Evidence of what? This is not a study of software development at all, but of prototyping using a program with no more

by pron 7y ago

> this evidence has been around for ages Evidence of what? This is not a study of software development at all, but of prototyping using a program with no more than a few hundred lines. I'm talking about effects on software development. The issue of prototyping/tiny programs is a completely separate one. > Substantiate that I don't know how I can substantiate the lack of evidence, but here's a recent failed attempt to find evidence: https://arxiv.org/pdf/1901.10220.pdf https://arxiv.org/pdf/1901.10220.pdf > Lack of knowledge by an HN poster is not evidence of lack of evidence. True. While I have been following the subject for at least the past 20 years (and I have read your PDF several times already, including shortly after it was published), it is certainly possible I have missed something. If there's any evidence of a large effect you believe you've found, let me know. > Evidence please? Sure. Here's code review, for example (I don't have time to discuss each paper -- as they're quite different -- but they all paint a similar picture). BTW, while programming language studies are debating effects in the 0-15% range, these report effects in the 30-80%: * What We Have Learned About Fighting Defects, 2002 -- https://www.cs.umd.edu/~mvz/pub/eworkshop02.pdf https://www.cs.umd.edu/~mvz/pub/eworkshop02.pdf * The Impact of Design and Code Reviews on Software Quality: An Empirical Study Based on PSP Data, 2009 -- https://www.pitt.edu/~ckemerer/PSP_Data.pdf https://www.pitt.edu/~ckemerer/PSP_Data.pdf * Large-Scale Analysis of Modern Code Review Practices and Software Security in Open Source Software, 2017 -- https://www2.eecs.berkeley.edu/Pubs/TechRpts/2017/EECS-2017-217.pdf https://www2.eecs.berkeley.edu/Pubs/TechRpts/2017/EECS-2017-... * What Types of Defects Are Really Discovered in Code Reviews?, 2003 -- https://ieeexplore.ieee.org/document/4604671 https://ieeexplore.ieee.org/document/4604671 * Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix?, 2014 -- https://www.testroots.org/assets/papers/2014_beller_bacchelli_zaidman_juergens_modern_code_reviews_in_open-source_projects_which_problems_do_they_fix.pdf https://www.testroots.org/assets/papers/2014_beller_bacchell... * The impact of code review coverage and code review participation on software quality: a case study of the qt, VTK, and ITK projects, 2014 -- https://dl.acm.org/citation.cfm?id=2597076 https://dl.acm.org/citation.cfm?id=2597076 * Best Kept Secrets of Peer Code Review, 2013 -- https://smartbear.com/SmartBear/media/pdfs/Best-Kept-Secrets-of-Peer-Code-Review_Redirected.pdf https://smartbear.com/SmartBear/media/pdfs/Best-Kept-Secrets... * Modern Code Review: A Case Study at Google, 2018 -- https://storage.googleapis.com/pub-tools-public-publication-data/pdf/80735342aebcbfc8af4878373f842c25323cb985.pdf https://storage.googleapis.com/pub-tools-public-publication-... If we summarize our current state of knowledge (not myth) it is this: process matters a lot; language matters little, if at all (with all the caveats I mentioned in other comments, such as diminishing returns etc.).

tempguy9999 7y ago
You're moving goalposts. You denied language mattered: > then the reality is that no big effect has been found for the choice of programming language on software quality Then I show some evidence giving small programs rapidly developed, which you then say doesn't count because it's 'tiny' and 'prototyping'. You also don't define quality which gives you lots of wriggle room. Nope, 1000 line programs aren't tiny. They aren't industry monsters but you can't dismiss them because it contradicts you. It's not proof, but it is strong evidence. ...arxiv paper... Interesting, thanks. It would have been helpful to have posted that in your original post. However from <https://soarsmu.github.io/papers/A_Large_Scale_Study_of_Multiple_Programming_Languages_and_Code_Quality.pdf> https://soarsmu.github.io/papers/A_Large_Scale_Study_of_Mult..., and note this paper is not mentioned, as in not debunked, in the above arxiv paper. "Bhattacharya et al. study four open-source projects which use C and C++, i.e., Firefox, Blender, VLC Media Player and MySQL to understand the impact of languages on software quality [20]. They compute several statistical measures while controlling for factors, such as developer competence and software process. They find that applications previously written in C are migrating to C++ and C++ code is often of higher quality, less prone to bugs, and easier to maintain than C code." and from that same paper: "As can be seen in Tables 8, 9, and 10, the mean de- fect density values for the C sets can be up to an order of magnitude higher than the mean values for the C++ sets." > Sure. Here's code review... I was talking about your claim about languages not mattering, I did not mention code review. All these papers are about code review. This is relevant to my code review comment (edit: I meant process comment), they don't say anything about language vs bugs (unless you wish to point out a paper that does). > process matters a lot And I agreed with you, I just said it didn't deliver quality, it just prepared the ground for it. With enough process you can close any hole in a language, but it gets exponentially expensive.
- pron 7y ago
  > You're moving goalposts. Only if you insist on uncharitable reading. I'm talking about software development; if I mistakenly assumed that could be left implicit, I'm sorry. And ~500 line programs are positively minuscule. JQuery is ~50KLOC, and an average business system is ~5MLOC. We're talking four orders of magnitude between those prototypes and a rather average industry system size (systems commonly run to tens and even more than 100 MLOC). > It's not proof, but it is strong evidence. I don't think it matters if it's strong or weak evidence, as it's not even about software development. > the C sets can be up to an order of magnitude higher than the mean values for the C++ sets. As I mentioned in another comment, C is not a good example. It's a ~50-year-old language, and the theory that correctly predicted that languages won't make a big difference was based on diminishing returns. I.e. not that no two languages have ever been or could ever have a big difference, but that over time the ability to affect quality with language would severely diminish. The original prediction of that theory was that no 10x improvement would be made by a single language improvement over a decade, and was called overly pessimistic by PL fans. It's not been over thirty years, and we have doubtfully made a 3x boost with all language features combined. If you want a more precise statement: no theory or empirical evidence supports the claim that a reasonable choice among production languages developed in the past three decades or so has a big impact on any measurable bottom-line metric. > I just said it didn't deliver quality But those papers show that, unlike language, process does have a big impact on quality. > With enough process you can close any hole in a language, but it gets exponentially expensive. You are now making an unsubstantiated claim that contradicts a substantiated claim. Programming languages (with the caveats above) have not been found to have a big effect (nor is there a theory that suggests they do), while process has.

by pron 7y ago

tempguy9999 7y ago

You're moving goalposts. You denied language mattered: > then the reality is that no big effect has been found for the choice of programming language on software quality Then I show some evidence giving small programs rapidly developed, which you then say doesn't count because it's 'tiny' and 'prototyping'. You also don't define quality which gives you lots of wriggle room. Nope, 1000 line programs aren't tiny. They aren't industry monsters but you can't dismiss them because it contradicts you. It's not proof, but it is strong evidence. ...arxiv paper... Interesting, thanks. It would have been helpful to have posted that in your original post. However from <https://soarsmu.github.io/papers/A_Large_Scale_Study_of_Multiple_Programming_Languages_and_Code_Quality.pdf> https://soarsmu.github.io/papers/A_Large_Scale_Study_of_Mult..., and note this paper is not mentioned, as in not debunked, in the above arxiv paper. "Bhattacharya et al. study four open-source projects which use C and C++, i.e., Firefox, Blender, VLC Media Player and MySQL to understand the impact of languages on software quality [20]. They compute several statistical measures while controlling for factors, such as developer competence and software process. They find that applications previously written in C are migrating to C++ and C++ code is often of higher quality, less prone to bugs, and easier to maintain than C code." and from that same paper: "As can be seen in Tables 8, 9, and 10, the mean de- fect density values for the C sets can be up to an order of magnitude higher than the mean values for the C++ sets." > Sure. Here's code review... I was talking about your claim about languages not mattering, I did not mention code review. All these papers are about code review. This is relevant to my code review comment (edit: I meant process comment), they don't say anything about language vs bugs (unless you wish to point out a paper that does). > process matters a lot And I agreed with you, I just said it didn't deliver quality, it just prepared the ground for it. With enough process you can close any hole in a language, but it gets exponentially expensive.

pron 7y ago
> You're moving goalposts. Only if you insist on uncharitable reading. I'm talking about software development; if I mistakenly assumed that could be left implicit, I'm sorry. And ~500 line programs are positively minuscule. JQuery is ~50KLOC, and an average business system is ~5MLOC. We're talking four orders of magnitude between those prototypes and a rather average industry system size (systems commonly run to tens and even more than 100 MLOC). > It's not proof, but it is strong evidence. I don't think it matters if it's strong or weak evidence, as it's not even about software development. > the C sets can be up to an order of magnitude higher than the mean values for the C++ sets. As I mentioned in another comment, C is not a good example. It's a ~50-year-old language, and the theory that correctly predicted that languages won't make a big difference was based on diminishing returns. I.e. not that no two languages have ever been or could ever have a big difference, but that over time the ability to affect quality with language would severely diminish. The original prediction of that theory was that no 10x improvement would be made by a single language improvement over a decade, and was called overly pessimistic by PL fans. It's not been over thirty years, and we have doubtfully made a 3x boost with all language features combined. If you want a more precise statement: no theory or empirical evidence supports the claim that a reasonable choice among production languages developed in the past three decades or so has a big impact on any measurable bottom-line metric. > I just said it didn't deliver quality But those papers show that, unlike language, process does have a big impact on quality. > With enough process you can close any hole in a language, but it gets exponentially expensive. You are now making an unsubstantiated claim that contradicts a substantiated claim. Programming languages (with the caveats above) have not been found to have a big effect (nor is there a theory that suggests they do), while process has.