5 ms·

The Epistemology of Software Quality

93 points by signa11 4y ago

nathias 4y ago
Poorly titled, I expected an article about how we can know software quality more objectively or something like that, as quality is always somewhat subjective or culturally dependent.
nathias 4y ago
I'm more on the technical solution team, mostly because the problems of the whole process (including your sleepless nights) will be determined by the particularities of your technologies. You can choose your technology well or you can choose poorly and go down the rabbit hole of new and new secondary problems and issues that only exist because of that poor decision. Practical wisdom is very effective. That said of course life/work balance is an important aspect and we should strive to avoid work cultures that value time invested versus output, because they reward mostly the appearance of bussiness instead of useful work, so you get both negatives workers are tired from working hard on appearing to be busy and the production becomes secondary, lower quality and slower.
okaleniuk 4y ago
What baffles me is that everyone talks about software quality but very little organization actually measure their software quality. Not just some made up metrics but exactly how well the software meets its requirements. And not that we don't know how. For any ML model, we have a validation data set and it is imperative that we measure how well a model performs on this data set and not the training data set. We know that without a validation, a machine will overfit its model into the data it has. Programmers... do the same thing. We're very good at passing tests and, unless we have a separate independent validation and verification process, we convince ourselves that green tests = quality. So our tests are always green but our backlogs are always red. And nobody seems to notice the contradiction.
- flappyeagle 4y ago
  What would you have us do? There is no such validation process.
  - _a_a_a_ 4y ago
    Agree. A positive suggestion would be welcome and very interesting to me.
    - okaleniuk 4y ago
      Sorry, I should have been more clear. Independent validation and verification is a thing: https://csrc.nist.gov/glossary/term/independent_verification_and_validation https://csrc.nist.gov/glossary/term/independent_verification.... We still use it in, for instance, NPP automation, avionics, and defense. Which not only makes sense but usually required by law. Interestingly, outside these few domains, we usually omit it as too costly as if doing the first and only independent validation with our own users is not.
      - _a_a_a_ 4y ago
        Useful to know, thanks. Trouble is, it's going to be damn expensive and I know no shortage of bosses who will become angry at having failures pointed out. It really takes money and a good mindset. > we usually omit it as too costly as if doing the first and only independent validation with our own users is not Users will accept absolute shite and that is definitely a cost saving. Unfortunately. End users complain a lot but in the end they'll just work around bugs and this is why I blame a lot of the deficiency in current software development on end users.
  - okaleniuk 4y ago
    Literally independent validation and verification. https://csrc.nist.gov/glossary/term/independent_verification_and_validation https://csrc.nist.gov/glossary/term/independent_verification... In some domains, doing IV&V to see how well your software really works is required by regulations, in others... we just guess.
flurgle 4y ago
There was no epistemology in this article.
- whitemary 4y ago
  Yeah, it gives a brief critique of existing ontologies but does not proceed into epistemology.
- actionfromafar 4y ago
  I guess we are it then?
- deleted 4y ago
  [deleted]
_match 4y ago
> Static typing? One study, presented at FSE 2014, found no evidence that static typing is helpful—or harmful And yet the abstract of the linked paper says: > Most notably, it does appear that strong typing is modestly better than weak typing, and among functional languages, static typing is also somewhat better than dynamic typing.
- luu 4y ago
  If you click through the link in that sentence to https://danluu.com/empirical-pl/ https://danluu.com/empirical-pl/ or read the study itself, you'll see that the paper doesn't support the claims made in the abstract at all. It used automatic classification that's obviously wrong. Table 1 gives a list of "top" projects for each language and many of them are simply misclassified. > ... the "top three" TypeScript projects are bitcoin, litecoin, and qBittorrent). These are C++ projects. So the intermediate result appears to not be that TypeScript is reliable, but that projects mis-identified as TypeScript are reliable. Those projects are reliable because Qt translation files are identified as TypeScript and it turns out that, per line of code, giant dumps of config files from another project don't cause a lot of bugs. It's like saying that a project has few bugs per line of code because it has a giant README. This is the most blatant classification error, but it's far from the only one. > For example, of what they call the "top three" perl projects, one is showdown, a javascript project, and one is rails-dev-box, a shell script and a vagrant file used to launch a Rails dev environment. Without knowing anything about the latter project, one might expect it's not a perl project from its name, rails-dev-box, which correctly indicates that it's a rails related project. There are other major problems with the study, but that one is sufficient to make the results invalid.
  - _match 4y ago
    Are you the author of the meta analysis? If so, thanks for your work on that. But I was not commenting on the quality of the studies. I did not think the author of "The Epistemology of Software Quality" was either.
smokel 4y ago
This article is nice to read, but I believe that some of the arguments are quite flawed. I doubt that empirical evidence is helpful in this context. Some references to studies are used to make bold statements. The ones on static vs dynamic typing for example, may look scientific, but scanning GitHub repositories or comparing student assignments does not convince me much about professional applications. It just seems nearly impossible to say something evidence based about this topic. Perhaps rewriting the entire Linux kernel along with all its drivers in Rust might be a nice approach. If it were done in parallel, starting in 1991. Also, applying results from studies in one domain to another is a red flag to me: if productivity drops for construction crews working extra hours, I fail to see how one can conclude that the same holds for software developers. Of course the conclusion is likely, but I don't need a scientific paper for that. The study was from 1980, mind you. Too bad, because the premise is really interesting and thought provoking. Focusing on the human factor in software development makes a lot of sense.
lifeisstillgood 4y ago
The things that make a team more effective are small things like psychological safety. But at a level of 2 orders of magnitude above the team say (ie team(s) stop being teams at 100 people - so imagine you manage 10,000 people) the psychological safety factors stop mattering as much as just throw bodies at the wall till you break through. not sure how to fix that
m3047 4y ago
Code review works. Ok, yeah I'd say that is the one practice which I've seen consistently tied to high(er) quality work. Article is pretty handwavy about epistemology and just things generally. But I'd say not all code reviews are equal, and that the best ones aren't about the code (in the way that baseball is not about bats and balls) and are instead about people learning to talk to each other, about code. Probably also about how that "campfire" provides a venue to discuss other institutional lore and hence culture.
- majormajor 4y ago
  There's no better example of the sort of problems "you get what you measure" causes than how even code review can go wrong. I've seen people use code review as an excuse not to test their own code, which is one of those things that made my eyes pop out the first couple of times I looked at a request with code that blatantly didn't compile or run. That's not to say "don't do code review" but it's worth making sure the team understands the goals and their individual responsibilities vs just blindly putting a process out there. Like you say, "communication" is handwavy but critical.
- kqr 4y ago
  Code review really does work, yes. Boehm has it at 60 % defects discovered at 15 % additional effort. Apparently "directed" or "scenario-based" code review works even better, uncovering an additional 20 or so percentage points of defects. But I have yet to find out what that means! Does it mean having a list of common problems and looking for one at a time? Does it mean taking a very concrete user story and mentally executing the code involved in satisfying it?
  - m3047 4y ago
    Best code reviews I've participated in were in a CMM 3 shop which was a division of a large corp. There was overarching method to the madness, but as a lowly engineer serving our manufacturing customer it looked more or less like: 1) pair up (from my POV) randomly (not everybody included in this is a coder) 2) (from my POV) flip a coin to decide who picks an item to review 3) pick an item you've touched recently or if there isn't one something you're going to touch in the next week (could be doc, build scripts, other things depending on what your job is; you might adjust your selection depending on your peer's duties and responsibilites, no sense having them review something they will never ever even look at). 4) flip a coin to decide who is going to explain what the code does 5) one person explains what the code does / doc means to the other person; the other person asks clarifying questions Repeat the above steps weekly. There was a sheet that was filled out that recorded defects found in categories such as logic, (failure of) abstraction, requirements, doc. We talked about style but there wasn't really an enforced style guide. It can be uncomfortable at first to talk about somebody else's code or have them explain your code. You learn to listen and have a sense of humor and be patient. Perhaps the most poignant lessons learned had to do with doc in code (generally more is better but watch out for zombie comments) and sometimes the most clever way of doing something isn't worth the potential for misunderstandings. There was usually something which needed more doc! Improving tests was also a common outcome. We did find logic errors occasionally, which would inevitably lead to discussion around improving tests. Requirements issues and abstraction / architecture issues came up infrequently and typically spawned email or a meeting which came later.
evolve2k 4y ago
Great article but I’m left with that it doesn’t actually address the title. Academically what is the epistemology of software quality? I sense it’s also related to practices of software craftsmanship, of which taking care of yourself and your sleep could be interpreted to fit within a broader goal of “improving my quality as a software developer”.
evolve2k 4y ago
I recently added standardb a very comprehensive linter to an old community project. https://github.com/demingfactor/calagator https://github.com/demingfactor/calagator The tools detected an absolute wall of code smells and security bad practices. Surprised that linters don’t have a reported quality impact. I personally was very glad for the issues it auto resolved and the issues it bought to my attention to address.
nunuvit 4y ago
The fundamentals of defect density have been known for a long time [1]. The factors that explain the vast majority of problems fit on a single slide. They're worded generically to apply across domains and it's very simple to evaluate whatever your process is against them. [1] https://www.slideshare.net/AnnMarieNeufelder/the-top-ten-things-that-have-been-proven-to-effect-software-reliability https://www.slideshare.net/AnnMarieNeufelder/the-top-ten-thi...
- majormajor 4y ago
  Somewhat orthoganal list here. That top 10 is almost all about "release frequently and test exhaustively and have well understood requirements." The linked article is all about "sleep and be healthy and work sustainably." You can be healthy happy and productive and ignore tests; or stressed and unproductive and write shit tests against shit requirements.
  - nunuvit 4y ago
    The linked article is about productivity and code quality. This list is about defect density, which is one aspect of code quality. It does not address everything in the article, but it does remove the mystery and common misconceptions of that one aspect.
- marcosdumay 4y ago
  That's very interesting, I just loved how she couldn't discover the impact of Agile because nobody she looked was doing it right.
  - switchbak 4y ago
    That tracks with my experience. Or certainly what I hear others say of their experiences. I've struggled and had what I believe to be a fair amount of success with Agile methods in many ways over the last 20 years. What shocks me is how few folks I talk to have similar stories. Edit: clarification.
    - 4y ago
m3kw9 4y ago
For sure like top tier athletes, they know it isn’t their skill that could hinder them the most, but sleep. Your brain needs to be sharp to beat other world class athletes to the punch. But it’s just a factor, a big factor. Skill matters too but that comes from years of practice
- switchbak 4y ago
  A huge part of keeping your brain sharp is getting lots of exercise too!
thephyber 4y ago
Agree with lots of the criticisms of the piece in other comments. I do instinctively agree with the theme of the piece that “human factors” have more of an effect on software quality than “technical factors”. I just don’t think the physiological factors listed in the articles are the biggest levers. Sure, getting enough sleep, having a reasonable diet and exercise are great for people long term, but I think these factors have a much larger impact: - company culture, - whether a developers feel like they can push back on requirements, - whether deadlines / expectations allow for sustainable pace of product development, - whether product managers / other stakeholders put objectively bad requirements on the project (eg. trading off security for user convenience)
- ChrisMarshallNY 4y ago
  There’s also the very old-fashioned “talent & passion” thing. Some people have it; some don’t. If someone has it, it’s generally attached to a persona with various drivers and flaws, and it takes a good manager to direct that talent and passion. It can get even trickier, when we have teams of these folks. That’s not a popular stance, these says. We’re supposed to come up with process and technology, that lets mediocre managers run teams of mediocre workers, yet produce excellent results. Lemme know how that works out.
  - thephyber 4y ago
    I would agree that there’s probably no match for talent+passion, but the fact is that there aren’t enough people with those attributes to pull an entire economy. Far better to live on the ground (not in the clouds). That said, processes and technologies don’t have to treat each employee as a newbie. Example: I remember Facebook had a deployment process which gave each developer a reputation. Getting a reputation for deploying code that requires hot fixes to revert broken code meant penalties. Penalties required additional pre-merge / pre-deploy reviews and proof of testing. I remember that as a good combination of treating developers as adults but also providing targeted procedures when they are needed.
    - hn_user82179 4y ago

93 points by signa11 4y ago

Poorly titled, I expected an article about how we can know software quality more objectively or something like that, as quality is always somewhat subjective or culturally dependent.

nathias 4y ago

I'm more on the technical solution team, mostly because the problems of the whole process (including your sleepless nights) will be determined by the particularities of your technologies. You can choose your technology well or you can choose poorly and go down the rabbit hole of new and new secondary problems and issues that only exist because of that poor decision. Practical wisdom is very effective. That said of course life/work balance is an important aspect and we should strive to avoid work cultures that value time invested versus output, because they reward mostly the appearance of bussiness instead of useful work, so you get both negatives workers are tired from working hard on appearing to be busy and the production becomes secondary, lower quality and slower.

okaleniuk 4y ago

What baffles me is that everyone talks about software quality but very little organization actually measure their software quality. Not just some made up metrics but exactly how well the software meets its requirements. And not that we don't know how. For any ML model, we have a validation data set and it is imperative that we measure how well a model performs on this data set and not the training data set. We know that without a validation, a machine will overfit its model into the data it has. Programmers... do the same thing. We're very good at passing tests and, unless we have a separate independent validation and verification process, we convince ourselves that green tests = quality. So our tests are always green but our backlogs are always red. And nobody seems to notice the contradiction.

flappyeagle 4y ago
What would you have us do? There is no such validation process.
- _a_a_a_ 4y ago
  Agree. A positive suggestion would be welcome and very interesting to me.
  - okaleniuk 4y ago
    Sorry, I should have been more clear. Independent validation and verification is a thing: https://csrc.nist.gov/glossary/term/independent_verification_and_validation https://csrc.nist.gov/glossary/term/independent_verification.... We still use it in, for instance, NPP automation, avionics, and defense. Which not only makes sense but usually required by law. Interestingly, outside these few domains, we usually omit it as too costly as if doing the first and only independent validation with our own users is not.
    - _a_a_a_ 4y ago
      Useful to know, thanks. Trouble is, it's going to be damn expensive and I know no shortage of bosses who will become angry at having failures pointed out. It really takes money and a good mindset. > we usually omit it as too costly as if doing the first and only independent validation with our own users is not Users will accept absolute shite and that is definitely a cost saving. Unfortunately. End users complain a lot but in the end they'll just work around bugs and this is why I blame a lot of the deficiency in current software development on end users.
- okaleniuk 4y ago
  Literally independent validation and verification. https://csrc.nist.gov/glossary/term/independent_verification_and_validation https://csrc.nist.gov/glossary/term/independent_verification... In some domains, doing IV&V to see how well your software really works is required by regulations, in others... we just guess.

flurgle 4y ago

There was no epistemology in this article.

whitemary 4y ago
Yeah, it gives a brief critique of existing ontologies but does not proceed into epistemology.
actionfromafar 4y ago
I guess we are it then?
deleted 4y ago
[deleted]

_match 4y ago

> Static typing? One study, presented at FSE 2014, found no evidence that static typing is helpful—or harmful And yet the abstract of the linked paper says: > Most notably, it does appear that strong typing is modestly better than weak typing, and among functional languages, static typing is also somewhat better than dynamic typing.

luu 4y ago
If you click through the link in that sentence to https://danluu.com/empirical-pl/ https://danluu.com/empirical-pl/ or read the study itself, you'll see that the paper doesn't support the claims made in the abstract at all. It used automatic classification that's obviously wrong. Table 1 gives a list of "top" projects for each language and many of them are simply misclassified. > ... the "top three" TypeScript projects are bitcoin, litecoin, and qBittorrent). These are C++ projects. So the intermediate result appears to not be that TypeScript is reliable, but that projects mis-identified as TypeScript are reliable. Those projects are reliable because Qt translation files are identified as TypeScript and it turns out that, per line of code, giant dumps of config files from another project don't cause a lot of bugs. It's like saying that a project has few bugs per line of code because it has a giant README. This is the most blatant classification error, but it's far from the only one. > For example, of what they call the "top three" perl projects, one is showdown, a javascript project, and one is rails-dev-box, a shell script and a vagrant file used to launch a Rails dev environment. Without knowing anything about the latter project, one might expect it's not a perl project from its name, rails-dev-box, which correctly indicates that it's a rails related project. There are other major problems with the study, but that one is sufficient to make the results invalid.
- _match 4y ago
  Are you the author of the meta analysis? If so, thanks for your work on that. But I was not commenting on the quality of the studies. I did not think the author of "The Epistemology of Software Quality" was either.

smokel 4y ago

This article is nice to read, but I believe that some of the arguments are quite flawed. I doubt that empirical evidence is helpful in this context. Some references to studies are used to make bold statements. The ones on static vs dynamic typing for example, may look scientific, but scanning GitHub repositories or comparing student assignments does not convince me much about professional applications. It just seems nearly impossible to say something evidence based about this topic. Perhaps rewriting the entire Linux kernel along with all its drivers in Rust might be a nice approach. If it were done in parallel, starting in 1991. Also, applying results from studies in one domain to another is a red flag to me: if productivity drops for construction crews working extra hours, I fail to see how one can conclude that the same holds for software developers. Of course the conclusion is likely, but I don't need a scientific paper for that. The study was from 1980, mind you. Too bad, because the premise is really interesting and thought provoking. Focusing on the human factor in software development makes a lot of sense.

lifeisstillgood 4y ago

The things that make a team more effective are small things like psychological safety. But at a level of 2 orders of magnitude above the team say (ie team(s) stop being teams at 100 people - so imagine you manage 10,000 people) the psychological safety factors stop mattering as much as just throw bodies at the wall till you break through. not sure how to fix that

m3047 4y ago

Code review works. Ok, yeah I'd say that is the one practice which I've seen consistently tied to high(er) quality work. Article is pretty handwavy about epistemology and just things generally. But I'd say not all code reviews are equal, and that the best ones aren't about the code (in the way that baseball is not about bats and balls) and are instead about people learning to talk to each other, about code. Probably also about how that "campfire" provides a venue to discuss other institutional lore and hence culture.

majormajor 4y ago
There's no better example of the sort of problems "you get what you measure" causes than how even code review can go wrong. I've seen people use code review as an excuse not to test their own code, which is one of those things that made my eyes pop out the first couple of times I looked at a request with code that blatantly didn't compile or run. That's not to say "don't do code review" but it's worth making sure the team understands the goals and their individual responsibilities vs just blindly putting a process out there. Like you say, "communication" is handwavy but critical.
kqr 4y ago
Code review really does work, yes. Boehm has it at 60 % defects discovered at 15 % additional effort. Apparently "directed" or "scenario-based" code review works even better, uncovering an additional 20 or so percentage points of defects. But I have yet to find out what that means! Does it mean having a list of common problems and looking for one at a time? Does it mean taking a very concrete user story and mentally executing the code involved in satisfying it?
- m3047 4y ago
  Best code reviews I've participated in were in a CMM 3 shop which was a division of a large corp. There was overarching method to the madness, but as a lowly engineer serving our manufacturing customer it looked more or less like: 1) pair up (from my POV) randomly (not everybody included in this is a coder) 2) (from my POV) flip a coin to decide who picks an item to review 3) pick an item you've touched recently or if there isn't one something you're going to touch in the next week (could be doc, build scripts, other things depending on what your job is; you might adjust your selection depending on your peer's duties and responsibilites, no sense having them review something they will never ever even look at). 4) flip a coin to decide who is going to explain what the code does 5) one person explains what the code does / doc means to the other person; the other person asks clarifying questions Repeat the above steps weekly. There was a sheet that was filled out that recorded defects found in categories such as logic, (failure of) abstraction, requirements, doc. We talked about style but there wasn't really an enforced style guide. It can be uncomfortable at first to talk about somebody else's code or have them explain your code. You learn to listen and have a sense of humor and be patient. Perhaps the most poignant lessons learned had to do with doc in code (generally more is better but watch out for zombie comments) and sometimes the most clever way of doing something isn't worth the potential for misunderstandings. There was usually something which needed more doc! Improving tests was also a common outcome. We did find logic errors occasionally, which would inevitably lead to discussion around improving tests. Requirements issues and abstraction / architecture issues came up infrequently and typically spawned email or a meeting which came later.

evolve2k 4y ago

Great article but I’m left with that it doesn’t actually address the title. Academically what is the epistemology of software quality? I sense it’s also related to practices of software craftsmanship, of which taking care of yourself and your sleep could be interpreted to fit within a broader goal of “improving my quality as a software developer”.

evolve2k 4y ago

I recently added standardb a very comprehensive linter to an old community project. https://github.com/demingfactor/calagator https://github.com/demingfactor/calagator The tools detected an absolute wall of code smells and security bad practices. Surprised that linters don’t have a reported quality impact. I personally was very glad for the issues it auto resolved and the issues it bought to my attention to address.

nunuvit 4y ago

The fundamentals of defect density have been known for a long time [1]. The factors that explain the vast majority of problems fit on a single slide. They're worded generically to apply across domains and it's very simple to evaluate whatever your process is against them. [1] https://www.slideshare.net/AnnMarieNeufelder/the-top-ten-things-that-have-been-proven-to-effect-software-reliability https://www.slideshare.net/AnnMarieNeufelder/the-top-ten-thi...

majormajor 4y ago
Somewhat orthoganal list here. That top 10 is almost all about "release frequently and test exhaustively and have well understood requirements." The linked article is all about "sleep and be healthy and work sustainably." You can be healthy happy and productive and ignore tests; or stressed and unproductive and write shit tests against shit requirements.
- nunuvit 4y ago
  The linked article is about productivity and code quality. This list is about defect density, which is one aspect of code quality. It does not address everything in the article, but it does remove the mystery and common misconceptions of that one aspect.
marcosdumay 4y ago
That's very interesting, I just loved how she couldn't discover the impact of Agile because nobody she looked was doing it right.
- switchbak 4y ago
  That tracks with my experience. Or certainly what I hear others say of their experiences. I've struggled and had what I believe to be a fair amount of success with Agile methods in many ways over the last 20 years. What shocks me is how few folks I talk to have similar stories. Edit: clarification.
  - 4y ago

m3kw9 4y ago

For sure like top tier athletes, they know it isn’t their skill that could hinder them the most, but sleep. Your brain needs to be sharp to beat other world class athletes to the punch. But it’s just a factor, a big factor. Skill matters too but that comes from years of practice

switchbak 4y ago
A huge part of keeping your brain sharp is getting lots of exercise too!

thephyber 4y ago

Agree with lots of the criticisms of the piece in other comments. I do instinctively agree with the theme of the piece that “human factors” have more of an effect on software quality than “technical factors”. I just don’t think the physiological factors listed in the articles are the biggest levers. Sure, getting enough sleep, having a reasonable diet and exercise are great for people long term, but I think these factors have a much larger impact: - company culture, - whether a developers feel like they can push back on requirements, - whether deadlines / expectations allow for sustainable pace of product development, - whether product managers / other stakeholders put objectively bad requirements on the project (eg. trading off security for user convenience)

ChrisMarshallNY 4y ago
There’s also the very old-fashioned “talent & passion” thing. Some people have it; some don’t. If someone has it, it’s generally attached to a persona with various drivers and flaws, and it takes a good manager to direct that talent and passion. It can get even trickier, when we have teams of these folks. That’s not a popular stance, these says. We’re supposed to come up with process and technology, that lets mediocre managers run teams of mediocre workers, yet produce excellent results. Lemme know how that works out.
- thephyber 4y ago
  I would agree that there’s probably no match for talent+passion, but the fact is that there aren’t enough people with those attributes to pull an entire economy. Far better to live on the ground (not in the clouds). That said, processes and technologies don’t have to treat each employee as a newbie. Example: I remember Facebook had a deployment process which gave each developer a reputation. Getting a reputation for deploying code that requires hot fixes to revert broken code meant penalties. Penalties required additional pre-merge / pre-deploy reviews and proof of testing. I remember that as a good combination of treating developers as adults but also providing targeted procedures when they are needed.
  - hn_user82179 4y ago