8 ms·

I work in the defense industry, it’s very much like the aerospace industry in that we deal with human life as a consequence of our work. We have software QA dep

I work in the defense industry, it’s very much like the aerospace industry in that we deal with human life as a consequence of our work. We have software QA departments that operate very much like manufacturing or aerospace QA. Software QA provides nothing of value to software development; having it as a dedicated function works against the overtly stated goals of the function and counterintuitively acts to degrade quality within software by mandating strict top down process and brittle end-to-end testing. Although Software QA is intended to be an independent verification body that provides engineering organizations with tools and resources, in practice they function as a moral crumple zone [1] within the complex socio-technical defense industrial system, being one of the groups that the finger will be pointed to when something goes wrong and absorb shock to the business in the event of a failure. As a result they have a strong incentive to highly systematize their work with specific process steps, to shield them from liability, which can be applied generically to all projects. Good software teams build quality into projects by introducing continuous integration, unit testing, creating feedback, and tightening these feedback loops. This acts to find problems quickly and resolve them quickly. Software QAs need for high level, top down, generic systemization requires them to work against these principles in practice. Bespoke project specific checks, such as unit testing, is not viewed as contributing to the final product and is discouraged by leadership who see it as waste. To give an example of how these dynamics destroy quality in software. I once found a bug in software on a piece of test equipment where a logarithmic search function was not operating on a strictly sorted list. When I pointed this out to my leadership I was told that if we changed any part of code, it would require a new FQT, which would be too expensive to conduct and was not in the budget. Although the bug would have been trivial to solve, and was clearly wrong and would not provide any benefits by remaining in the test equipment software, the process required for changes prevented solving the issue. [1] https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2757236 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2757236

Eisenstein 3y ago
It sounds like they are calling something QA but using it as a liability shield. It makes sense that you are upset about that, but naming something QA and having it do something else doesn't mean that QA as an effort is bad. It means that the people doing that are being deceptive.
- vegetablepotpie 3y ago
  Fair point, you are correct in your inference that there are some bad actors in my workplace. However, I’ll argue that the fundamental dynamics of bifurcating the responsibility of quality from software leads to a steady state where all QA departments end up as a liability shield given enough time. This is driven by Pournelle's iron law of bureaucracy [1], which says that people who promote the bureaucracy rather than the mission of the bureaucracy will get promoted within the organization and come to dominate its decision making. For example, in schools, administrators make more money than teachers. This is despite both groups having similar levels of education and intelligence. The reason for this is that administrators know the laws and regulations of the environment they’re working in and ensure the continuity of the organization. Despite not directly contributing to the organization’s stated mission of education, they are in charge of the organization and take more benefits from it. Software QA has similar dynamics. A QA department may start out making good faith contributions to the organization. Eventually there are product failures, eventually leadership needs a scapegoat to show they’re doing something, and eventually QA takes the blame. People get moved, demoted, or fired. QA realizes its risk, and takes steps to mitigate it. They create a highly systematized workflow and process, adopt or introduce standards. Then assert that following process equates to good outcomes. When bad outcomes occur, they point to their strict adherence to following process as evidence of innocence. If the process does not support the work or mission, that is a cost they are happy to impose on other functions to deal with. This is the final state until a system disruption happens. [1] https://en.m.wikipedia.org/wiki/Jerry_Pournelle https://en.m.wikipedia.org/wiki/Jerry_Pournelle
  - Eisenstein 3y ago
    But what makes 'software QA' fundamentally different than 'non-software QA' to give it the problems you foresee?
    - vegetablepotpie 3y ago
      Fundamentally different problems. Manufacturing process builds identical widgets using standard equipment. Widgets are inspected to confirm they are within spec. Frequentist statistics are used to determine when widgets are consistently out of spec. When this happens, equipment is inspected and repaired as a corrective action. This process is well defined and linear. Software produces bespoke, non-standard widgets to address domain specific needs. At the end of the day, software developers are defining a process for machines to follow. If you want to control quality in the software development , aside from having perfect domain knowledge for a particular project, the only way to do it do it is to define an arbitrary process for developers to follow and track adherence to it. This may have no impact, or be a hindrance. It will never add value because it will never be abstract enough to be appropriate for every domain.
ethbr1 3y ago
> Good software teams build quality into projects by introducing continuous integration, unit testing, creating feedback, and tightening these feedback loops. Agreed, for good software teams. I would content that most software teams at most companies are not good. Which is to ask, with an average to bad software team is it better to have integrated or separate QA?
- vegetablepotpie 3y ago
  Does it make sense to degrade the performance of good software teams because bad software teams exist? Ideally we’d always have good software teams, but in the real world sometimes you have to build software with bad teams. Leaders have options, they can do things like reduce scope, increase budget, increase schedule, or full on abandon or cancel the project. These are all options available to leaders, but they require tradeoffs and decisions to be made on a project by project basis. It is scalable to have a strict process that everyone has to follow, then impose a watchdog to enforce it on a wide scale. It may not be better to have separate QA, but it is easier for those in charge.
  - ethbr1 3y ago
    It makes the most sense to me to match the org structure to the teams you have. If I'm trying to build something with undertrained, demoralized, underpaid engineers... it's not optimal to use methods intended for self-motivated, high-performance teams. And nothing says there must be company-wide mandates. Maybe this area gets a formal, independent QA team, but this other area doesn't. My experience just doesn't bear out that collapsing the QA function into development always leads to better outcomes. I've seen the opposite happen too often, and QA be the sole bulwark between idiocy and customers.
  - bb88 3y ago
    > Ideally we’d always have good software teams, but in the real world sometimes you have to build software with bad teams. Here's a real, perhaps unexpected counterpoint. Say you have a good software team. How do they build good software with bad management?
9659 3y ago
About 2000, Software QA (and almost all traditional QA activities) were changed. The focus was on process over inspection. "Design in quality, do not inspect it into the product" Suppliers (to include software) were expected to manage the quality of the product they provided; the purchaser would focus on how they managed the process, not in the compliance of every part. This had a chance until software process was tossed in the name of "agile".
- bb88 3y ago
  Here's a stupid question: How do you know your process is good unless you inspect it? "Hey Bob I know you're a competent engineer, but don't worry about specifying a certain type of bolt or loctite, the untrained assembly personnel will figure it out. I'm sure they won't let 200 people die in a plane crash."
  - zilti 3y ago
    Make it impossible to use a wrong bolt, and train the assembly workers.
- steve_gh 3y ago
  I recall a bug I was involved with at a telecoms equipment market in the early 2000s. The bug only showed up in our biggest base stations in high load situations. We diagnosed the bug, and there were a couple of parts to it. Sloppy software design in an optional hardware module (no state machine) was one part - and was fixed. But there was another underlying issue in the way message queues were handled. Anyhow, the fix for this was created and written. But we never got to put it into production. The reason: the company didn't have a lab test facility that could put a sufficient load on the software to prove it. Even though we were getting field failures because of this issue that were getting a bad rep, we couldn't fix it because even though the old code was known to be buggy, we couldn't prove the new code. So the process said we couldn't ship it.
  - DougBTX 3y ago
    Another way of looking at that is that within the ability to test, the implementations were indistinguishable, so the process mandated that the older implementation must be used. I wonder if they would have explicitly specified age as a metric if this was considered when designing the process.
amluto 3y ago
> We have software QA departments that operate very much like manufacturing or aerospace QA. I don’t work in this industry, but this seems fairly ridiculous on its face: software is not at all like manufacturing. In manufacturing, there’s a design and a manufacturing process, and a critical function of QA is ensuring that the manufactured produce is manufactured to spec. With software, the software is written, compiled, and then repeatedly copied. And something should verify that it’s copied correctly, but this is straightforward and boring. So software QA ought to be much more like the kind of validation that happens when designing hardware, not like the kind of testing and validation that happens as products are manufactured.
- irrational 3y ago
  Ideally there should be a solid spec written and then qa can test against the spec. Maybe there is somewhere that does write solid specs, including accounting for corner cases, but in my 25 years working professionally in the industry, I’ve never seen it.
  - stickfigure 3y ago
    The only complete and precise specification of software is the code itself. If some other form of specification was complete, we would be able to auto-generate the code.
    - cogwheel 3y ago
      This is beside the point. The code specifies what the product _is_, not what it "should" be. If you ask for a word processor and I deliver a perfectly bug-free and feature-complete calculator would you really believe it lived up to spec?
      - _gabe_ 3y ago
        This is also beside the point. I think both of you are trying to warn against the dangers that lie on both sides of this coin: people can invest too heavily in a specification and waste an enormous amount of time, and people can immediately jump into coding and code something that does not do what it was intended to do. Like with all things in life, there’s a balance between these two extremes that’s correct. You need some level of specification so you know what you’re building, but you have to keep in mind that the final code defines what the behavior truly is. Sometimes, that behavior unintentionally becomes part of the specification because users begin to rely on it. I do like the fact that you both used hyperbole to succinctly illustrate the dangers of veering too far in either direction though :)
bb88 3y ago
> Good software teams build quality into projects by introducing continuous integration, unit testing, creating feedback, and tightening these feedback loops. No. Good software teams are led by competent, technical management. Managers who aren't afraid to get down into the dirty details. Managers who aren't afraid to roll up their sleeves and write code if they need to. The process doesn't matter. The management of what is or is not important does. Agile is just one process out of many. Imagine an accounting team led by someone who never did accounting in their life: "Just make the numbers work out! I don't care how you do it! My bonus is at stake!"
- johnnyworker 3y ago
  But that doesn't contradict the parent, does it? I'd say you both make good points.
  - bb88 3y ago
    Well... here's a thought experiment. Let's say you have a bunch of school children and architects create a skyscraper. I've given both groups the process to design a skyscraper. So in both cases, I should end up with a safe building?
    - johnnyworker 3y ago
      Why not both? Am I missing something? You can have feedback loops and CI and all that, "good craftsmanship" or "good practices" (not "best" practices because those often suck hah), where of course opinions vary on the details of that -- and then someone who is also good at the craft who spends more or most time on helping the rest work together, i.e. manage/lead them.
    - withinboredom 3y ago
      I’d bet the children would come out better simply because they have parents who are likely multi-disciplined as a group. A disparite group will (almost) always come up with better results than a homogeneous one (at least in my experience)
- 3y ago
shermantanktop 3y ago
I’m going to be using “moral crumple zone” in every conversation I possibly can from now on.
- BlueUmarell 3y ago
  "Hey, wanna hang out with a beer this evening?" "Nah, I don't need to slouch anymore in this moral crumple zone!" "We need this new feature in our program!" "If we implement this it means the management fell in a moral crumple zone" "What seat would you have to have for your flight?" "Anywhere, but not in the moral crumple zone, please"
  - shermantanktop 3y ago
    An example from my actual work life, lightly fictionalized— Them: “Please review this design.” Me: “Ok, sure, when do you plan to start coding?” Them: “Oh, it’s already in beta.” Me: “So you can’t do anything with my feedback, but you’ll say I reviewed it?” Them: “Well…” Me: “You’re putting me in a moral crumple zone here!”
__loam 3y ago
High quality comment
jl6 3y ago
I'm not going to argue with the general thrust of your comment, which I think is insightful as to how incentives can compromise objectives. But... > To give an example of how these dynamics destroy quality in software. I once found a bug in software on a piece of test equipment where a logarithmic search function was not operating on a strictly sorted list. When I pointed this out to my leadership I was told that if we changed any part of code, it would require a new FQT, which would be too expensive to conduct and was not in the budget. Although the bug would have been trivial to solve, and was clearly wrong and would not provide any benefits by remaining in the test equipment software, the process required for changes prevented solving the issue. I've seen this happen where it was a bad thing, but also where it was a good thing. It's all about risk. What risk does the software defect pose to the mission? What risk is inherent in making any change to the software? Noting that even trivial changes can be fat-fingered and thus are a source of risk. I've seen it go wrong this way: a seemingly trivial change was made, but the developer accidentally checked an extra file into source control, causing a further defect. And then: what is the cost of mitigating these risks? Maybe the software defect is as trivial as its fix. Maybe an acceptable fix would be to write up a workaround in the documentation. I don't think it's always wrong to say no to fixing issues. I also don't think it's always right that a separate QA department contributes nothing to the organization, even if they act as a handbrake on the software developers (sometimes, precisely because they do that). Human factors are real.
legacynl 3y ago
I think you're sort of misunderstanding the role of QA. You think that QA is a liability shield, but that is only a side effect of the work that they actually do. The task of QA is exactly that: an entity that tries to assure that the quality is up to some standard. Even in favourable conditions mistakes happen, so how do you make sure as a company that not 1 in every 100 product are faulty and tarnishes the good reputation that your company has spent so much time and money on to build? You hire a QA to make sure problems get caught before delivery. But if all humans make mistakes, and QA is human, how do you make sure that the QA doesn't make a mistake? A never ending chain of QAs expecting each other? No of course not. One thing that helps with reducing errors is to have a rigid protocol that is followed to the letter everytime. Pilots, for example, have a preflight checklist that they have to run every time they operate the plane. The rigid protocol of QA teams is therefore an essential part of their jobs. Although from your standpoint as a developer it might seem strange that QA is 'preventing' you from fixing a bug, it is actually very reasonable. Especially since you work in the defence industry, I hope you understand that it is very important that the software that operates radars, planes, missiles, bombs, etc is working exactly as expected. Understandably there is a great deal of effort made to assure that when those things are needed they work exactly to spec. So in your example it is probably very reasonable that any change you make needs to go through some rigorous process. The fact that it 'only' was about test equipment, doesn't matter because test equipment is just as, if not more important as the stuff it tests. The reason why QA has the side-effect of being a 'liability shield' is that it gives companies the ability to argue (and proof) after the fact that the company did their due diligence in making sure that the product was to spec. Especially certification is basically to get an external organisation to approve your QA. In that case if you get sued you can rightfully claim that you did everything that was legally asked of you, and if there is blame, then it is the certifying company using insufficient standards.

Eisenstein 3y ago

It sounds like they are calling something QA but using it as a liability shield. It makes sense that you are upset about that, but naming something QA and having it do something else doesn't mean that QA as an effort is bad. It means that the people doing that are being deceptive.

vegetablepotpie 3y ago
Fair point, you are correct in your inference that there are some bad actors in my workplace. However, I’ll argue that the fundamental dynamics of bifurcating the responsibility of quality from software leads to a steady state where all QA departments end up as a liability shield given enough time. This is driven by Pournelle's iron law of bureaucracy [1], which says that people who promote the bureaucracy rather than the mission of the bureaucracy will get promoted within the organization and come to dominate its decision making. For example, in schools, administrators make more money than teachers. This is despite both groups having similar levels of education and intelligence. The reason for this is that administrators know the laws and regulations of the environment they’re working in and ensure the continuity of the organization. Despite not directly contributing to the organization’s stated mission of education, they are in charge of the organization and take more benefits from it. Software QA has similar dynamics. A QA department may start out making good faith contributions to the organization. Eventually there are product failures, eventually leadership needs a scapegoat to show they’re doing something, and eventually QA takes the blame. People get moved, demoted, or fired. QA realizes its risk, and takes steps to mitigate it. They create a highly systematized workflow and process, adopt or introduce standards. Then assert that following process equates to good outcomes. When bad outcomes occur, they point to their strict adherence to following process as evidence of innocence. If the process does not support the work or mission, that is a cost they are happy to impose on other functions to deal with. This is the final state until a system disruption happens. [1] https://en.m.wikipedia.org/wiki/Jerry_Pournelle https://en.m.wikipedia.org/wiki/Jerry_Pournelle
- Eisenstein 3y ago
  But what makes 'software QA' fundamentally different than 'non-software QA' to give it the problems you foresee?
  - vegetablepotpie 3y ago
    Fundamentally different problems. Manufacturing process builds identical widgets using standard equipment. Widgets are inspected to confirm they are within spec. Frequentist statistics are used to determine when widgets are consistently out of spec. When this happens, equipment is inspected and repaired as a corrective action. This process is well defined and linear. Software produces bespoke, non-standard widgets to address domain specific needs. At the end of the day, software developers are defining a process for machines to follow. If you want to control quality in the software development , aside from having perfect domain knowledge for a particular project, the only way to do it do it is to define an arbitrary process for developers to follow and track adherence to it. This may have no impact, or be a hindrance. It will never add value because it will never be abstract enough to be appropriate for every domain.

ethbr1 3y ago

> Good software teams build quality into projects by introducing continuous integration, unit testing, creating feedback, and tightening these feedback loops. Agreed, for good software teams. I would content that most software teams at most companies are not good. Which is to ask, with an average to bad software team is it better to have integrated or separate QA?

vegetablepotpie 3y ago
Does it make sense to degrade the performance of good software teams because bad software teams exist? Ideally we’d always have good software teams, but in the real world sometimes you have to build software with bad teams. Leaders have options, they can do things like reduce scope, increase budget, increase schedule, or full on abandon or cancel the project. These are all options available to leaders, but they require tradeoffs and decisions to be made on a project by project basis. It is scalable to have a strict process that everyone has to follow, then impose a watchdog to enforce it on a wide scale. It may not be better to have separate QA, but it is easier for those in charge.
- ethbr1 3y ago
  It makes the most sense to me to match the org structure to the teams you have. If I'm trying to build something with undertrained, demoralized, underpaid engineers... it's not optimal to use methods intended for self-motivated, high-performance teams. And nothing says there must be company-wide mandates. Maybe this area gets a formal, independent QA team, but this other area doesn't. My experience just doesn't bear out that collapsing the QA function into development always leads to better outcomes. I've seen the opposite happen too often, and QA be the sole bulwark between idiocy and customers.
- bb88 3y ago
  > Ideally we’d always have good software teams, but in the real world sometimes you have to build software with bad teams. Here's a real, perhaps unexpected counterpoint. Say you have a good software team. How do they build good software with bad management?

9659 3y ago

About 2000, Software QA (and almost all traditional QA activities) were changed. The focus was on process over inspection. "Design in quality, do not inspect it into the product" Suppliers (to include software) were expected to manage the quality of the product they provided; the purchaser would focus on how they managed the process, not in the compliance of every part. This had a chance until software process was tossed in the name of "agile".

bb88 3y ago
Here's a stupid question: How do you know your process is good unless you inspect it? "Hey Bob I know you're a competent engineer, but don't worry about specifying a certain type of bolt or loctite, the untrained assembly personnel will figure it out. I'm sure they won't let 200 people die in a plane crash."
- zilti 3y ago
  Make it impossible to use a wrong bolt, and train the assembly workers.
steve_gh 3y ago
I recall a bug I was involved with at a telecoms equipment market in the early 2000s. The bug only showed up in our biggest base stations in high load situations. We diagnosed the bug, and there were a couple of parts to it. Sloppy software design in an optional hardware module (no state machine) was one part - and was fixed. But there was another underlying issue in the way message queues were handled. Anyhow, the fix for this was created and written. But we never got to put it into production. The reason: the company didn't have a lab test facility that could put a sufficient load on the software to prove it. Even though we were getting field failures because of this issue that were getting a bad rep, we couldn't fix it because even though the old code was known to be buggy, we couldn't prove the new code. So the process said we couldn't ship it.
- DougBTX 3y ago
  Another way of looking at that is that within the ability to test, the implementations were indistinguishable, so the process mandated that the older implementation must be used. I wonder if they would have explicitly specified age as a metric if this was considered when designing the process.

amluto 3y ago

> We have software QA departments that operate very much like manufacturing or aerospace QA. I don’t work in this industry, but this seems fairly ridiculous on its face: software is not at all like manufacturing. In manufacturing, there’s a design and a manufacturing process, and a critical function of QA is ensuring that the manufactured produce is manufactured to spec. With software, the software is written, compiled, and then repeatedly copied. And something should verify that it’s copied correctly, but this is straightforward and boring. So software QA ought to be much more like the kind of validation that happens when designing hardware, not like the kind of testing and validation that happens as products are manufactured.

irrational 3y ago
Ideally there should be a solid spec written and then qa can test against the spec. Maybe there is somewhere that does write solid specs, including accounting for corner cases, but in my 25 years working professionally in the industry, I’ve never seen it.
- stickfigure 3y ago
  The only complete and precise specification of software is the code itself. If some other form of specification was complete, we would be able to auto-generate the code.
  - cogwheel 3y ago
    This is beside the point. The code specifies what the product _is_, not what it "should" be. If you ask for a word processor and I deliver a perfectly bug-free and feature-complete calculator would you really believe it lived up to spec?
    - _gabe_ 3y ago
      This is also beside the point. I think both of you are trying to warn against the dangers that lie on both sides of this coin: people can invest too heavily in a specification and waste an enormous amount of time, and people can immediately jump into coding and code something that does not do what it was intended to do. Like with all things in life, there’s a balance between these two extremes that’s correct. You need some level of specification so you know what you’re building, but you have to keep in mind that the final code defines what the behavior truly is. Sometimes, that behavior unintentionally becomes part of the specification because users begin to rely on it. I do like the fact that you both used hyperbole to succinctly illustrate the dangers of veering too far in either direction though :)

bb88 3y ago

> Good software teams build quality into projects by introducing continuous integration, unit testing, creating feedback, and tightening these feedback loops. No. Good software teams are led by competent, technical management. Managers who aren't afraid to get down into the dirty details. Managers who aren't afraid to roll up their sleeves and write code if they need to. The process doesn't matter. The management of what is or is not important does. Agile is just one process out of many. Imagine an accounting team led by someone who never did accounting in their life: "Just make the numbers work out! I don't care how you do it! My bonus is at stake!"

johnnyworker 3y ago
But that doesn't contradict the parent, does it? I'd say you both make good points.
- bb88 3y ago
  Well... here's a thought experiment. Let's say you have a bunch of school children and architects create a skyscraper. I've given both groups the process to design a skyscraper. So in both cases, I should end up with a safe building?
  - johnnyworker 3y ago
    Why not both? Am I missing something? You can have feedback loops and CI and all that, "good craftsmanship" or "good practices" (not "best" practices because those often suck hah), where of course opinions vary on the details of that -- and then someone who is also good at the craft who spends more or most time on helping the rest work together, i.e. manage/lead them.
  - withinboredom 3y ago
    I’d bet the children would come out better simply because they have parents who are likely multi-disciplined as a group. A disparite group will (almost) always come up with better results than a homogeneous one (at least in my experience)
3y ago

shermantanktop 3y ago

I’m going to be using “moral crumple zone” in every conversation I possibly can from now on.

BlueUmarell 3y ago
"Hey, wanna hang out with a beer this evening?" "Nah, I don't need to slouch anymore in this moral crumple zone!" "We need this new feature in our program!" "If we implement this it means the management fell in a moral crumple zone" "What seat would you have to have for your flight?" "Anywhere, but not in the moral crumple zone, please"
- shermantanktop 3y ago
  An example from my actual work life, lightly fictionalized— Them: “Please review this design.” Me: “Ok, sure, when do you plan to start coding?” Them: “Oh, it’s already in beta.” Me: “So you can’t do anything with my feedback, but you’ll say I reviewed it?” Them: “Well…” Me: “You’re putting me in a moral crumple zone here!”

__loam 3y ago

High quality comment

jl6 3y ago

I'm not going to argue with the general thrust of your comment, which I think is insightful as to how incentives can compromise objectives. But... > To give an example of how these dynamics destroy quality in software. I once found a bug in software on a piece of test equipment where a logarithmic search function was not operating on a strictly sorted list. When I pointed this out to my leadership I was told that if we changed any part of code, it would require a new FQT, which would be too expensive to conduct and was not in the budget. Although the bug would have been trivial to solve, and was clearly wrong and would not provide any benefits by remaining in the test equipment software, the process required for changes prevented solving the issue. I've seen this happen where it was a bad thing, but also where it was a good thing. It's all about risk. What risk does the software defect pose to the mission? What risk is inherent in making any change to the software? Noting that even trivial changes can be fat-fingered and thus are a source of risk. I've seen it go wrong this way: a seemingly trivial change was made, but the developer accidentally checked an extra file into source control, causing a further defect. And then: what is the cost of mitigating these risks? Maybe the software defect is as trivial as its fix. Maybe an acceptable fix would be to write up a workaround in the documentation. I don't think it's always wrong to say no to fixing issues. I also don't think it's always right that a separate QA department contributes nothing to the organization, even if they act as a handbrake on the software developers (sometimes, precisely because they do that). Human factors are real.

legacynl 3y ago

I think you're sort of misunderstanding the role of QA. You think that QA is a liability shield, but that is only a side effect of the work that they actually do. The task of QA is exactly that: an entity that tries to assure that the quality is up to some standard. Even in favourable conditions mistakes happen, so how do you make sure as a company that not 1 in every 100 product are faulty and tarnishes the good reputation that your company has spent so much time and money on to build? You hire a QA to make sure problems get caught before delivery. But if all humans make mistakes, and QA is human, how do you make sure that the QA doesn't make a mistake? A never ending chain of QAs expecting each other? No of course not. One thing that helps with reducing errors is to have a rigid protocol that is followed to the letter everytime. Pilots, for example, have a preflight checklist that they have to run every time they operate the plane. The rigid protocol of QA teams is therefore an essential part of their jobs. Although from your standpoint as a developer it might seem strange that QA is 'preventing' you from fixing a bug, it is actually very reasonable. Especially since you work in the defence industry, I hope you understand that it is very important that the software that operates radars, planes, missiles, bombs, etc is working exactly as expected. Understandably there is a great deal of effort made to assure that when those things are needed they work exactly to spec. So in your example it is probably very reasonable that any change you make needs to go through some rigorous process. The fact that it 'only' was about test equipment, doesn't matter because test equipment is just as, if not more important as the stuff it tests. The reason why QA has the side-effect of being a 'liability shield' is that it gives companies the ability to argue (and proof) after the fact that the company did their due diligence in making sure that the product was to spec. Especially certification is basically to get an external organisation to approve your QA. In that case if you get sued you can rightfully claim that you did everything that was legally asked of you, and if there is blame, then it is the certifying company using insufficient standards.