6 ms·

> then the reality is that no big effect has been found for the choice of programming language on software quality Substantiate that. I say it's garbage. <htt

> then the reality is that no big effect has been found for the choice of programming language on software quality Substantiate that. I say it's garbage. <https://www.cse.iitk.ac.in/users/karkare/courses/2010/cs653/Papers/hudak_haskell_sw_prototype.pdf> https://www.cse.iitk.ac.in/users/karkare/courses/2010/cs653/..., and this BTW is from 1994 so this evidence has been around for ages Language LOC Documentation Lines Dev time (hours) (1)Haskell 85 465 10 (2)Ada 767 714 23 (3)Ada9X 800 200 28 (4)C++ 1105 130 – (5)Awk/Nawk 250 150 – (6)Rapide 157 0 54 (7)Griffin 251 0 34 (8)Proteus 293 79 26 (9)RelationalLisp 274 12 3 (10)Haskell 156 112 8 (Edit: all aligned with tabs but looks like they've been stripped. See paper instead) comments of note: (2) The Ada solution was written by a lead programmer at NSWC; in this sense it represents the “control” group. The developer initially reported a line count of only 249; this was the numberof lines of imperativestatements as reported by theSun Adacompiler, anddid notinclude declarations, all of which were essential for proper execution. The line count of 767 is based on the actual code in and does not include lines with only termination characters. (4) TheC++solutionwaswrittenbyanONRprogrammanagerafterhavingfirstwrittentheAwk solution described in the next paragraph. In addition to these 1105 lines of code, the developer also wrote a 595-line “test harness.” No development times were reported [Note: C++ has evolved a lot since then, but perhaps so has haskell] (10) Intermetrics, independently and without the knowledge of NSWC or Yale University, con-ductedan experimentof its own: theHaskell Report was given to anewly hiredcollege graduate, who was then given 8 days to learn Haskell. This new hire received no formal training, but was allowed to ask an experienced Haskell programmer questions as issues came up during the self study. After this training period, the new hire was handed the geo-server specification and asked to write aprototype in Haskell. Theresulting metrics shown in row 10 ofthe tableare perhaps the most stunning of the lot, suggesting the ease with which Haskell may be learned and effectively utilized. Also mentioned was "The use of higher-order functions [in haskell] is noteworthy" so language features did help. Actual evidence. > As lack of evidence in favor of a big effect is evidence of its lack yeah right. Let me rephrase that for you - Lack of knowledge by an HN poster is not evidence of lack of evidence. > What has been shown to drive quality up or down is process Evidence please? My experience is that process can be rigorous but useless crap. My old boss said "process is not sufficient to produce quality, but it is a necessary" There's plenty more in that PDF, please read it.

pron 7y ago
> this evidence has been around for ages Evidence of what? This is not a study of software development at all, but of prototyping using a program with no more than a few hundred lines. I'm talking about effects on software development. The issue of prototyping/tiny programs is a completely separate one. > Substantiate that I don't know how I can substantiate the lack of evidence, but here's a recent failed attempt to find evidence: https://arxiv.org/pdf/1901.10220.pdf https://arxiv.org/pdf/1901.10220.pdf > Lack of knowledge by an HN poster is not evidence of lack of evidence. True. While I have been following the subject for at least the past 20 years (and I have read your PDF several times already, including shortly after it was published), it is certainly possible I have missed something. If there's any evidence of a large effect you believe you've found, let me know. > Evidence please? Sure. Here's code review, for example (I don't have time to discuss each paper -- as they're quite different -- but they all paint a similar picture). BTW, while programming language studies are debating effects in the 0-15% range, these report effects in the 30-80%: * What We Have Learned About Fighting Defects, 2002 -- https://www.cs.umd.edu/~mvz/pub/eworkshop02.pdf https://www.cs.umd.edu/~mvz/pub/eworkshop02.pdf * The Impact of Design and Code Reviews on Software Quality: An Empirical Study Based on PSP Data, 2009 -- https://www.pitt.edu/~ckemerer/PSP_Data.pdf https://www.pitt.edu/~ckemerer/PSP_Data.pdf * Large-Scale Analysis of Modern Code Review Practices and Software Security in Open Source Software, 2017 -- https://www2.eecs.berkeley.edu/Pubs/TechRpts/2017/EECS-2017-217.pdf https://www2.eecs.berkeley.edu/Pubs/TechRpts/2017/EECS-2017-... * What Types of Defects Are Really Discovered in Code Reviews?, 2003 -- https://ieeexplore.ieee.org/document/4604671 https://ieeexplore.ieee.org/document/4604671 * Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix?, 2014 -- https://www.testroots.org/assets/papers/2014_beller_bacchelli_zaidman_juergens_modern_code_reviews_in_open-source_projects_which_problems_do_they_fix.pdf https://www.testroots.org/assets/papers/2014_beller_bacchell... * The impact of code review coverage and code review participation on software quality: a case study of the qt, VTK, and ITK projects, 2014 -- https://dl.acm.org/citation.cfm?id=2597076 https://dl.acm.org/citation.cfm?id=2597076 * Best Kept Secrets of Peer Code Review, 2013 -- https://smartbear.com/SmartBear/media/pdfs/Best-Kept-Secrets-of-Peer-Code-Review_Redirected.pdf https://smartbear.com/SmartBear/media/pdfs/Best-Kept-Secrets... * Modern Code Review: A Case Study at Google, 2018 -- https://storage.googleapis.com/pub-tools-public-publication-data/pdf/80735342aebcbfc8af4878373f842c25323cb985.pdf https://storage.googleapis.com/pub-tools-public-publication-... If we summarize our current state of knowledge (not myth) it is this: process matters a lot; language matters little, if at all (with all the caveats I mentioned in other comments, such as diminishing returns etc.).

pron 7y ago

> this evidence has been around for ages Evidence of what? This is not a study of software development at all, but of prototyping using a program with no more than a few hundred lines. I'm talking about effects on software development. The issue of prototyping/tiny programs is a completely separate one. > Substantiate that I don't know how I can substantiate the lack of evidence, but here's a recent failed attempt to find evidence: https://arxiv.org/pdf/1901.10220.pdf https://arxiv.org/pdf/1901.10220.pdf > Lack of knowledge by an HN poster is not evidence of lack of evidence. True. While I have been following the subject for at least the past 20 years (and I have read your PDF several times already, including shortly after it was published), it is certainly possible I have missed something. If there's any evidence of a large effect you believe you've found, let me know. > Evidence please? Sure. Here's code review, for example (I don't have time to discuss each paper -- as they're quite different -- but they all paint a similar picture). BTW, while programming language studies are debating effects in the 0-15% range, these report effects in the 30-80%: * What We Have Learned About Fighting Defects, 2002 -- https://www.cs.umd.edu/~mvz/pub/eworkshop02.pdf https://www.cs.umd.edu/~mvz/pub/eworkshop02.pdf * The Impact of Design and Code Reviews on Software Quality: An Empirical Study Based on PSP Data, 2009 -- https://www.pitt.edu/~ckemerer/PSP_Data.pdf https://www.pitt.edu/~ckemerer/PSP_Data.pdf * Large-Scale Analysis of Modern Code Review Practices and Software Security in Open Source Software, 2017 -- https://www2.eecs.berkeley.edu/Pubs/TechRpts/2017/EECS-2017-217.pdf https://www2.eecs.berkeley.edu/Pubs/TechRpts/2017/EECS-2017-... * What Types of Defects Are Really Discovered in Code Reviews?, 2003 -- https://ieeexplore.ieee.org/document/4604671 https://ieeexplore.ieee.org/document/4604671 * Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix?, 2014 -- https://www.testroots.org/assets/papers/2014_beller_bacchelli_zaidman_juergens_modern_code_reviews_in_open-source_projects_which_problems_do_they_fix.pdf https://www.testroots.org/assets/papers/2014_beller_bacchell... * The impact of code review coverage and code review participation on software quality: a case study of the qt, VTK, and ITK projects, 2014 -- https://dl.acm.org/citation.cfm?id=2597076 https://dl.acm.org/citation.cfm?id=2597076 * Best Kept Secrets of Peer Code Review, 2013 -- https://smartbear.com/SmartBear/media/pdfs/Best-Kept-Secrets-of-Peer-Code-Review_Redirected.pdf https://smartbear.com/SmartBear/media/pdfs/Best-Kept-Secrets... * Modern Code Review: A Case Study at Google, 2018 -- https://storage.googleapis.com/pub-tools-public-publication-data/pdf/80735342aebcbfc8af4878373f842c25323cb985.pdf https://storage.googleapis.com/pub-tools-public-publication-... If we summarize our current state of knowledge (not myth) it is this: process matters a lot; language matters little, if at all (with all the caveats I mentioned in other comments, such as diminishing returns etc.).