6 ms·

Ask HN: Best empirical papers on software development?

• eimrine 4y ago
  Mythical man-month - this is a short book written dozens of years ago but from my opinion it is timeless.
  • liamconnell 4y ago
    Not sure why you were downvoted. Parts of the book are indeed timeless! My favorite but is the idea that out of the jumble of documents produced, a few emerge as the pivot around which the majority of the team’s communication revolves. I’ve seen very effective project leads use this to great effect. Rather than robotically using boilerplate document templates they discover the most precise format for the project at hand.
    • hwayne 4y ago
      True, but it's not empirical research.
  • affgrff2 4y ago
    But is that an empirical paper?
  • snovv_crash 4y ago
    On this topic, Peopleware as well. It references a lot of studies and is extremely well researched.
• mtmail 4y ago
  Can you list the ones you know/found?
  • KingOfCoders 4y ago
    You found some in (the very old) "Facts and Fallacies of Software Engineering"
    • teddyh 4y ago
      2003 is “very old”? I would have expected that description about something like Structured Programming from 1972. Not 2003.
      • ThePadawan 4y ago
        If not in number of years, more so in half-lifes of knowledge [0]. [0] https://en.wikipedia.org/wiki/Half-life_of_knowledge https://en.wikipedia.org/wiki/Half-life_of_knowledge
      • KingOfCoders 4y ago
        "Very old" is subjective, some people think the Antikythera to be very old. But when I remember developing software in my startup around 2000, the world of software development was completely different. So, hence, for me, very old.
      • rileyphone 4y ago
        Software is a pop culture.
• bramblerose 4y ago
  I quite like https://neverworkintheory.org/reviews/ https://neverworkintheory.org/reviews/ which has summaries of recent empirical research papers
  • KingOfCoders 4y ago
    Thanks, yes, I put the very interesting https://neverworkintheory.org/2022/08/30/software-engineering-research-questions.html https://neverworkintheory.org/2022/08/30/software-engineerin... in my newsletter yesterday :-)
• okasaki 4y ago
  You might like to review the book Evidence-based Software Engineering, which is freely available online: http://www.knosof.co.uk/ESEUR/ http://www.knosof.co.uk/ESEUR/
  • KingOfCoders 4y ago
    Great resource!
  • loquisgon 4y ago
    This looked promising, while I browsed the index. Especially the first chapters that discuss human cognition. Since software is still a human creative endeavor cognition is key to understand software engineering. However, continuing to read the index I am discouraged and I expect that this is a difficult read. I was expecting more like a handbook, more succint, that I could use to explain software engineering to non technical leaders. Also, why in the world somebody would choose R as a vehicle to present software engineering fundamentals? This by itself almost disqualifies the book for me. It would have been much more useful, IMO, to use some imperative or object oriented, more common, language.
• Qualadore 4y ago
  In terms of methodological quality, "Fixing Faults in C and Java Source Code: Abbreviated vs. Full-Word Identifier Names" [0] is a favorite of Hillel Wayne's [1]. > Manageable scope. > They did their homework. > They mix qualitative and quantitative methods. > Objective measure of Defects. > Really, _really_ good experimental setup. > And then an ethnography. [0] http://www2.unibas.it/gscanniello/Giuseppe_Scanniello%40unibas/Home_files/TOSEM.pdf http://www2.unibas.it/gscanniello/Giuseppe_Scanniello%40unib... [1] https://www.hillelwayne.com/post/the-best-se-paper/ https://www.hillelwayne.com/post/the-best-se-paper/
• pid-1 4y ago
  Here's a few I like: Why Do Computers Stop and What Can Be Done About It? https://pages.cs.wisc.edu/~remzi/Classes/739/Fall2018/Papers/gray85-easy.pdf https://pages.cs.wisc.edu/~remzi/Classes/739/Fall2018/Papers... Not a paper, but the whole book "Accelerate" presents many empirical findings related to automating software operations. Hidden Technical Debt in Machine Learning Systems - NIPS https://papers.nips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html https://papers.nips.cc/paper/2015/hash/86df7dcfd896fcaf2674f...
• ta238911 4y ago
  A great recap of some papers: Software engineering's greatest hits. * https://www.youtube.com/watch?v=HrVtA-ue-x0 https://www.youtube.com/watch?v=HrVtA-ue-x0 * slides: https://third-bit.com/talks/greatest-hits/#1 https://third-bit.com/talks/greatest-hits/#1
• smaddox 4y ago
  Technically not a paper, but Casey Muratori takes a very empirical approach to software development, which I find extremely practical: - https://caseymuratori.com/blog_0015 https://caseymuratori.com/blog_0015 - https://caseymuratori.com/contents https://caseymuratori.com/contents
  • jesse__ 4y ago
    +1 for this. I love the way Casey works
  • KingOfCoders 4y ago
    Thanks!
• eclarkso 4y ago
  I have always appreciated https://www.cs.cmu.edu/~NatProg/papers/Ko2008JavaWhyline.pdf https://www.cs.cmu.edu/~NatProg/papers/Ko2008JavaWhyline.pdf - an empirical analysis of debugging and a tool based on those findings, and an evaluation of that tool.
• deleted 4y ago
  [deleted]
• divbzero 4y ago
  Are there studies comparing total development effort to build and maintain software using types vs. no types? You would need to use closely related languages like TypeScript vs. JavaScript, or something like Python with vs. without type hints.
  • hwayne 4y ago
    There's some, but it's not... great. The two highest profile studies I know of are: "A Large-Scale Study of Programming Languages and Code Quality in Github" [1]: Finds significant differences in defect rate between typed and untyped languages. It stood for a while before replicators tore it apart. Among other things, they counted the v8 engine as a javascript project, counted seventeen forks of Bitcoin as distinct projects, and counted commits like "add infix operator" as a sign of a defect (because it has "fix" in it). I wrote a bunch about the study here: https://www.hillelwayne.com/post/this-is-how-science-happens/ https://www.hillelwayne.com/post/this-is-how-science-happens... "To Type Or Not to Type" [2]: Finds that Typescript catches 15% of bugs missed in Javascript repos. Haven't dug into this as much, but I'm suspicious of it because they didn't seem to do any quality control. One of the codebases they studied, for example, is a junior dev's minesweeper game.[3] Right now there's a lot of case studies of companies happy with typescript/mypy, which I take as tentative evidence that type systems are on the whole beneficial, but I'm also open to the tide changing in 4-5 years. [1] https://cacm.acm.org/magazines/2017/10/221326-a-large-scale-study-of-programming-languages-and-code-quality-in-github/fulltext https://cacm.acm.org/magazines/2017/10/221326-a-large-scale-... [2] https://earlbarr.com/publications/typestudy.pdf https://earlbarr.com/publications/typestudy.pdf [3] https://github.com/anabarasan/Minesweeper https://github.com/anabarasan/Minesweeper
• hwayne 4y ago
  These days I've been really keen on qualitative studies, where scientists work on building a better understanding of software instead of trying to validate theories. Some examples of that: https://link.springer.com/article/10.1007/s10664-016-9464-2 https://link.springer.com/article/10.1007/s10664-016-9464-2 — scientists followed a team of developers for three years and recorded all of their sprint retrospectives and people kept forgetting what they already learned. https://jlubin.net/assets/oopsla21.pdf https://jlubin.net/assets/oopsla21.pdf — researchers compiled 23 hours of Zoom sessions and 15 hours of Twitch programming livestreams to study how people write code in static FP languages. In addition to that, there's a lot of good work on how we teach programming. "Commonsense Computing" [1] found that students understand concurrency a lot faster when presented as a "human" problem, such as selling tickets in a concert. I'd recommend reading Teaching Tech Together (http://teachtogether.tech http://teachtogether.tech), which references a lot of empirical papers on teaching programming. [1]: https://cseweb.ucsd.edu/classes/fa08/cse599/Papers/ICERConcurrencyFINAL.pdf https://cseweb.ucsd.edu/classes/fa08/cse599/Papers/ICERConcu...
  • blacklite 4y ago
    This paper on retros is hilarious gold.
• arijr 4y ago
  Hard question, there's a lot; one favourite I came across years ago on one uni. course when I was finishing my SW Eng. MSc and while had I worked in Global SW Dev for years. The research was so well put together on a difficult to measure topic: An empirical study of speed and communication in globally distributed software development http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.324.3157&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.324...
• westurner 4y ago
  From https://en.wikipedia.org/wiki/Experimental_software_engineering https://en.wikipedia.org/wiki/Experimental_software_engineer... : > Experimental software engineering involves running experiments on the processes and procedures involved in the creation of software systems, with the intent that the data be used as the basis of theories about the processes involved in software engineering (theory backed by data is a fundamental tenet of the scientific method). A number of research groups primarily use empirical and experimental techniques. > The term empirical software engineering emphasizes the use of empirical studies of all kinds to accumulate knowledge. Methods used include experiments, case studies, surveys, and using whatever data is available. (CS) Papers We Love > https://github.com/papers-we-love/papers-we-love#other-good-places-to-find-papers https://github.com/papers-we-love/papers-we-love#other-good-... : - "Systematic Review in Software Engineering" (2005) -- "The Developed Template for Systematic Reviews in Software Engineering" - "Happiness and the productivity of software engineers" (2019) DevTech Research Group (Kibo, Scratch Jr,) > Publications https://sites.bc.edu/devtech/publications/ https://sites.bc.edu/devtech/publications/ ' > Empirical Research, instruments: https://sites.bc.edu/devtech/about-devtech/empirical-research/ https://sites.bc.edu/devtech/about-devtech/empirical-researc... "SafeScrum: Agile Development of Safety-Critical Software" (2018) > A Summary of Research https://scholar.google.com/scholar?cites=9208467786713301421&as_sdt=2005&sciodt=0,5&hl=en https://scholar.google.com/scholar?cites=9208467786713301421... (Gscholar features: cited by, Related Articles) https://link.springer.com/chapter/10.1007/978-3-319-99334-8_12 https://link.springer.com/chapter/10.1007/978-3-319-99334-8_... Re: Safety-Critical systems, awesome-safety-critical, and Formal Verification as the ultimate empirical study: https://news.ycombinator.com/item?id=28709239 https://news.ycombinator.com/item?id=28709239
• Pelam 4y ago
  The studies referred in this book https://www.amazon.com/Accelerate-Software-Performing-Technology-Organizations/dp/1942788339 https://www.amazon.com/Accelerate-Software-Performing-Techno... Nicole Forsgren PhD and 2 more Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations
• geekjock 4y ago
  I write and share summaries of papers related to developer productivity here: https://abinoda.substack.com https://abinoda.substack.com Two of my favorites so far are https://homepages.dcc.ufmg.br/~figueiredo/disciplinas/papers/icse17sedano.pdf https://homepages.dcc.ufmg.br/~figueiredo/disciplinas/papers... and https://faculty.washington.edu/ajko/papers/Li2019WhatDistinguishesEngineers.pdf https://faculty.washington.edu/ajko/papers/Li2019WhatDisting...