4 ms·

This article, as is typical around here, addresses a very small part of the industry. Here's a view from a different part; the part where a project has a lifecy

This article, as is typical around here, addresses a very small part of the industry. Here's a view from a different part; the part where a project has a lifecycle of a decade or so. I used to work for a software consultancy that had been acquired by a US defence company (second tier). Over the previous 15 years, the software consultancy had elevated their QA to the point that bugs in the delivered product were literally unknown. Nobody ever saw any in the delivered product, over a decade. Every requirement could be traced to the design that would meet it, to the code that implemented it, to the test that verified it. The test documentation was beautiful, and anyone could (and did) execute a complete set of tests, writing their own personal signature on every confirmation of test success. The signed test specs were scanned before being sealed in an envelope against the customer ever choosing to inspect. QA had (and did) veto over requirements, design, and tests. We were working with two other major, globally known defence names, and the customer had learned through experience that when something went wrong, come to us last. Sometimes, the customer came to us to ask us to work out which of the other companies involved had screwed up (because sometimes, the other companies were themselves unable to work it out). Once, they sued one of the other names for incompetency and gave us their work because we knew how to fix it better than they did, because our interface tests highlighted everything that their software didn't do in accordance with the spec. We could replay any test, at any version, at any time, and compare the results now with the signed results then from the sealed envelope. The time savings we had because we got things right the first time, by doing it properly, was immense. Where this is going is that this was priceless. We had the customer's absolute trust, based on experience. Nobody ever found a bug in our delivered software (I'm not saying there weren't any; I'm saying that our QA was thorough enough that the customer was never going to do anything with it that we hadn't tested already). Code coverage, all warnings on, static analysis, valgrind, cross-platform compilation and testing, we did everything. Customers outright told us that we were worth paying extra because of the peace of mind and the reduced risk, so they paid us more and we spent less doing it. QA is time and money. The better your QA, the more time you will have and the more money you will make.

geoelectric 11y ago
So, this I agree with too, in contrast with my other comment. There are particular industries (engineering, banking, aerospace, defense, for some examples) where it -has- to be correct. These are also the industries where at least the final product should probably be waterfall or spiral, and generally you need formalism all the way down. The problem comes when people bring lessons learned from that to companies that don't need that level of assurance. Documentation that's never read again is a waste. Spending time to make tests replayable if they'll never be replayed again is a waste. And so forth. As for bugs being literally unknown, I'm jealous--it'd be nice to claim I ever guided a project to that state. But keep in mind that's different than no bugs period, just means no bugs in what was tested or used. If you're delivering a product with very specific use cases, flows and a tight scope with strong bumpers around anything unintended, it's pretty straightforward to guarantee all that. But if you're delivering a sandbox product on the internet to a bunch of randoms, much less so. After all, ten checkboxes on a dialog are 1024 testcases. That's not even accounting for free-entry fields. You'll never cover all combinations of all uses of the whole app interface, ever, period, and equivalences and fuzz testing only get you so far. That's a simple (if slightly fantastic) example for UI testing, but the same basic thing applies to most other kinds of testing. At the end of the day you often have to pick your battles. So I really disagree on the better is more. There is absolutely a diminishing return. It just so happens that for what -you- were doing, that point of diminishing return is pretty high.
- deleted 11y ago
  [deleted]
- EliRivers 11y ago
  The problem comes when people bring lessons learned from that to companies that don't need that level of assurance. It goes the other way as well. Bringing agile methods to a situation in which having a requirement changed can take three months. Situations in which your "customer" is a faceless defence ministry (or two) with the level of responsiveness you'd expect. Processes are tools; use the right tools for the job. As for bugs being literally unknown, I'm jealous--it'd be nice to claim I ever guided a project to that state. I certainly can't make that claim. I was only there for five years; I think I saw three staged releases (which were for the purpose of allowing integration testing with other components). Maybe two. :)
pbiggar 11y ago
> QA is time and money. The better your QA, the more time you will have and the more money you will make. Just to caveat again that this is for your part of the industry. That level of detail gets you few benefits if you were Facebook, for example, or a very small startup still doing product validation. Also to say that "the better your QA the more money you'll make" is a very strong position to take, even in your industry.
- EliRivers 11y ago
  Also to say that "the better your QA the more money you'll make" is a very strong position to take, even in your industry. It is a bit strong, isn't it? I felt that to caveat it might detract from the effectiveness of the point. There does come a point where the software is effectively flawless from the customer's point of view, and doing anything to improve it (including more/better QA) from there doesn't make any more money. But if you reach that point, you know :)
  - pbiggar 11y ago
    More than that, better QA or better software doesn't actually make you money. As in, you can't make your software twice as good and make twice as much money. It's a lot more nuanced and much more indirect than that. So certainly you can say "investing X QA hours per month should help reduce customer incidents by 50%, which will reduce churn rate by 30%, increasing MRR by 100% over 1 year, given current churn rates". Or whatever other metrics affect your actual revenue. Overall, you need to draw a much stronger connection between QA and money, esp if you want to claim a continuous function between them.
MichaelCrawford 11y ago
I've done two government contracts, just one for defense. I won't ever take a defense contract again because the primary contractor selected the wrong part, then hired me as a subcontractor based on my expertise with a part they should not have selected. This actually came to the point that I spent six weeks furiously trying to find a workaround while they burned a bunch of money making prototype assemblies that did not work, then delivering them to "The Client" with the expectation that they would integrate my firmware later.
ThatMightBePaul 11y ago
Woah! Thanks for sharing. Love hearing about the other side. All of the projects I've been involved in operate on either a continuous release, or a one to two week release cycle. So, I'm definitely ignorant to the standards at spots with months-long release cycles.

geoelectric 11y ago

So, this I agree with too, in contrast with my other comment. There are particular industries (engineering, banking, aerospace, defense, for some examples) where it -has- to be correct. These are also the industries where at least the final product should probably be waterfall or spiral, and generally you need formalism all the way down. The problem comes when people bring lessons learned from that to companies that don't need that level of assurance. Documentation that's never read again is a waste. Spending time to make tests replayable if they'll never be replayed again is a waste. And so forth. As for bugs being literally unknown, I'm jealous--it'd be nice to claim I ever guided a project to that state. But keep in mind that's different than no bugs period, just means no bugs in what was tested or used. If you're delivering a product with very specific use cases, flows and a tight scope with strong bumpers around anything unintended, it's pretty straightforward to guarantee all that. But if you're delivering a sandbox product on the internet to a bunch of randoms, much less so. After all, ten checkboxes on a dialog are 1024 testcases. That's not even accounting for free-entry fields. You'll never cover all combinations of all uses of the whole app interface, ever, period, and equivalences and fuzz testing only get you so far. That's a simple (if slightly fantastic) example for UI testing, but the same basic thing applies to most other kinds of testing. At the end of the day you often have to pick your battles. So I really disagree on the better is more. There is absolutely a diminishing return. It just so happens that for what -you- were doing, that point of diminishing return is pretty high.

deleted 11y ago
[deleted]
EliRivers 11y ago
The problem comes when people bring lessons learned from that to companies that don't need that level of assurance. It goes the other way as well. Bringing agile methods to a situation in which having a requirement changed can take three months. Situations in which your "customer" is a faceless defence ministry (or two) with the level of responsiveness you'd expect. Processes are tools; use the right tools for the job. As for bugs being literally unknown, I'm jealous--it'd be nice to claim I ever guided a project to that state. I certainly can't make that claim. I was only there for five years; I think I saw three staged releases (which were for the purpose of allowing integration testing with other components). Maybe two. :)

pbiggar 11y ago

> QA is time and money. The better your QA, the more time you will have and the more money you will make. Just to caveat again that this is for your part of the industry. That level of detail gets you few benefits if you were Facebook, for example, or a very small startup still doing product validation. Also to say that "the better your QA the more money you'll make" is a very strong position to take, even in your industry.

EliRivers 11y ago
Also to say that "the better your QA the more money you'll make" is a very strong position to take, even in your industry. It is a bit strong, isn't it? I felt that to caveat it might detract from the effectiveness of the point. There does come a point where the software is effectively flawless from the customer's point of view, and doing anything to improve it (including more/better QA) from there doesn't make any more money. But if you reach that point, you know :)
- pbiggar 11y ago
  More than that, better QA or better software doesn't actually make you money. As in, you can't make your software twice as good and make twice as much money. It's a lot more nuanced and much more indirect than that. So certainly you can say "investing X QA hours per month should help reduce customer incidents by 50%, which will reduce churn rate by 30%, increasing MRR by 100% over 1 year, given current churn rates". Or whatever other metrics affect your actual revenue. Overall, you need to draw a much stronger connection between QA and money, esp if you want to claim a continuous function between them.

MichaelCrawford 11y ago

I've done two government contracts, just one for defense. I won't ever take a defense contract again because the primary contractor selected the wrong part, then hired me as a subcontractor based on my expertise with a part they should not have selected. This actually came to the point that I spent six weeks furiously trying to find a workaround while they burned a bunch of money making prototype assemblies that did not work, then delivering them to "The Client" with the expectation that they would integrate my firmware later.

ThatMightBePaul 11y ago

Woah! Thanks for sharing. Love hearing about the other side. All of the projects I've been involved in operate on either a continuous release, or a one to two week release cycle. So, I'm definitely ignorant to the standards at spots with months-long release cycles.