9 ms·

I worked at two companies 15-20 years ago that invested in top-tier QA teams. They were worth their weight in gold. The products were world class because the

I worked at two companies 15-20 years ago that invested in top-tier QA teams. They were worth their weight in gold. The products were world class because the QA team were fantastic at finding bugs we developers didn't think of looking for because we were too close to the problem. We are too used to looking at the happy path. One key attribute to both companies is that it was dictated from on high that the QA team had final say whether the release went to production or not. These days companies think having the developers write automated tests and spend an inordinate amount of time worrying over code coverage is better. I can't count how many products I've seen with 100% code coverage that objectively, quantifiably doesn't work. I'm not saying automated testing is bad. I'm saying, just as the author does, that doing away with human QA testers is.

supportengineer 3y ago
I've seen QA/QE greatness and it was similar to how you describe. A different chain of command for deciding if releases are certified for production. Different incentive structures as well. Not to mention, at one recent employer, the QE team wrote an enormous amount of code to perform their tests - It was more LOC than the modules being tested/certified.
- eitally 3y ago
  I had a team like that once. It was glorious. And ultimately, I'm convinced it led to overall faster development cycles because the baseline code quality & documentation was so much better than it would have been without such a great QA manager. The QA team, of course, was also technical -- mostly with SWE backgrounds -- and they were primarily colo'd in the same office as the dev team. I still remember the epiphany everyone had one planning cycle when it was mutually understood that by generally agreeing to use TDD, the QA team could participate actively in the real engineering planning and product development process. ... Then I left and my CIO let go the onshore QA team in favor of near term cost savings. Code quality went way down and within a year or two several apps needed to be entirely rewritten. Everything slowed down and people started pointing fingers, and before you knew it, it was time for "cloud native rearchitecting/reengineering" which required an SI to come in with "specialists".
  - MichaelZuo 3y ago
    I've heard this enough times that I'm becoming convinced there's a lot of 'reinventing the wheel' busywork going on in the industry. And not even superior wheels, lower quality, more fragile wheels by clumsier wheel designers.
- esafak 3y ago
  So how was QA incentivized?
- munificent 3y ago
  > It was more LOC than the modules being tested/certified. So much code. I hope they had a QAQA team to test all that.
fizx 3y ago
How often did you release?
- SkyPuncher 3y ago
  That's my question as well. Bug free code is not the goal. Valuable product is.
- Zelphyr 3y ago
  At the first company we released a few times a year. You have to remember that CI/CD wasn't a thing back then. While I worked on a web-based product at this company, it was a time when it was still very common to release software on physical media like CD's. At the second company we integrated Agile so we released every two or weeks (I can't recall which). The first time I heard about CI/CD was at this company from co-workers who came back from a conference all excited about it.
alkonaut 3y ago
I love my really thorough QA’s. Yes it’s an antipattern to let me as a dev lean too much on them catching what I won’t. But where I dread even running the code for a minute, they enjoy it. They take pride in figuring out edge cases far beyond any spec. They are definitely worth their weight in gold. It lets developers have confidence when changing things in the same sense a good type system does. For some classes of very interactive apps (e.g games) having unit and integration tests just doesn’t cover the parameter space.
- ponector 3y ago
  People here are talking about skillful QA worth their weight in gold. Unfortunately people in the industry who have actual power in planning budgets don't think so. An article is right. QA engineers now are viewed as janitors: no one respects then, better to outsource to cheap location.
  - bluGill 3y ago
    Depends on industry. Some care about quality, they respect qa.
PH95VuimJjqBqy 3y ago
The QA culture has to be there, not just dictates that the QA has final say. I've seen companies where that's true and it was still trash because the QA were mostly low-paid contract workers who only did exactly what they were told and no more.
- onlyrealcuzzo 3y ago
  Worked at a company where QA had the final say - and that was by far the most toxic / worst environment I have ever been in. QA also REFUSED to let developers write automation tests, also REFUSED to let us run them ourselves. What a nightmare. YMMV, but just having the final say is not a silver bullet for sure.
  - mikestew 3y ago
    That’s not because “QA had the final say”, it’s because your QA team were ass clowns. Any QA team that discourages dev from writing or running tests needs to be burnt to ground and rebuilt.
    - ponector 3y ago
      Unless dev is switching to write test 100% of time and become a tester it is highly not recommended to let developers write tests. That QA was not a clown, they've seen some shit... Would you let QA to write features in your production code?
      - ahtihn 3y ago
        Test code is just code. If you can write test code you can write production code. If you can write production code you can write tests. If your concern is that devs don't have the right mindset for testing, you can have them collaborate with a QA specialist to define the test cases and review the test implementation.
        ponector 3y ago
ponector 3y ago
I've seen developers writing tests with no assertions, or with assertTrue(true). Always green, with 100% coverage! The same people have been asking: why should I write tests if I can write new features? And then one senior QA comes and destroys everything. Once I found that if I press f5 50 times during a minute then backend will go in outOfMemory while spinning requests to the database.
- kcb 3y ago
  That's why the dev teams tests and the QA teams tests are not mutually exclusive.
  - ponector 3y ago
    Testing is continuous multilayered multistaged process. Testing should start before first developer wrote first line of code for the project. Architecture blueprints, set of requirements should be tested as early as possible. But that is not happening in real life, only in books. In real life pm will bring cheap contractor from India one month before the target release date.
- feoren 3y ago
  > I've seen developers writing tests with no assertions This can be OK if the code executing without throwing exceptions is itself testing something. If you have a lot of assertions written directly into the code, as pre- or post-conditions for instance. But I'm guessing that wasn't the case here.
  - WrongAssumption 3y ago
    Why would it be ok to not test the assertions would be triggered if the conditions are bad? How would you verify that the assertions are correct with just a happy path? If you run a code with pre-post assertions, then remove all the assertions the same test will continue to pass.
    - feoren 3y ago
mikestew 3y ago
I can't count how many products I've seen with 100% code coverage that objectively, quantifiably doesn't work. That’s because code coverage doesn’t find the bugs that result from code you didn’t write, but should have. Code coverage is but one measure, and to treat it as the measure is folly. (But, yes, I have heard a test manager at a large software company we’ve all heard of declare that test team was done because 100% coverage.)
- thaumasiotes 3y ago
  It's more because code coverage isn't measured correctly. What you want is branch coverage. If you have something like this: if condition1: do_something1() if condition2: do_something2() if condition3: do_something3() There are 8 possible paths for the code to follow here, but you can cover 100% of lines of code in one test. If your code coverage measure tells you that testing the case where conditions 1, 2, and 3 are all true achieves 100% coverage, your measure is worthless. Covering this code cannot be done in less than eight tests.
  - alternatex 3y ago
    I think most code coverage analysis tools are smart enough to consider flow control statements. At least when it comes to C# ones.
    - fho 3y ago
      They do, but that's not the point GP made. An example how this could fail is if the first branch sets up something that branch two uses. Two tests are written that call branch 1&2 and not call them. 100% code coverage, even the non-happy path was tested. But in reality, if those branches have any interaction, you would need to write eight test cases for every combination of branches being run and not being run.
      - thaumasiotes 3y ago
        Well, again, eight is a lower bound. If the software uses any values beyond the bare minimum (here) of three separate booleans, you're likely to need more. Take this simplified code: if condition1: do_something() You need one test for code coverage and two for branch coverage, but if there's a bug in the code as written you'll find that you actually need more. Assume that we want to run do_something() when, and only when, condition2 is true. Now we have four cases: - condition1 && condition2: do_something() runs, which is correct. - condition1 && !condition2: do_something() runs, which is incorrect. - !condition1 && !condition2: do_something() does not run, which is correct. - !condition1 && condition2: do_something() does not run, which is incorrect. If you happened to write tests for the first and third of those options, your tests will make it look like your code works, and your (official) coverage metrics will look comprehensive. But because your coverage metrics (in reality) are actually terrible, your code is secretly not working. The big problem we've just run into is that the number of test cases we need depends on the number of bugs in our code. If the goal of writing tests is to prove that we don't have bugs, this is a terrible result; it means our goal is fundamentally impossible to achieve. We don't know whether we have bugs, so we don't know how many tests we need.
bluGill 3y ago
Automated tests take manual testing from 60% to 50% of the time to develop quality software. Valuable, but not a bullet to manual tests.
hedora 3y ago
The best approach I’ve seen is to do all of the above. - Have QA run pessimal versions of real use cases. Trying to sell a word processor to lawyers? Format the entire US legal code and a bajillion contracts in it, then duplicate it 10x and start filing usability/performance bugs. - Have the engineers test everything with randomly generated workloads before committing. Run those tests nightly, and fix all the crashes / failures. - Have Product Management (remember them?) work with marketing and sales to figure out what absolutely has to ship, and when. Make sure it only takes one of the above three groups to stop ship, and also to stop non-essential development tasks.
- Arainach 3y ago
  The devil is in the details of "essential development tasks". An uncountable number of products have died or devolved because "we don't have time to do it that way, put in the quick fix"
turok 3y ago
We try to do this, QA owns deployment to production and has the final say if a feature needs a re-write. The almost adversarial incentives of dev and QA are why it works. Dev wants to close ticket, QA wants the feature to work, Product wants to tick a box, all collaborate so that the closed ticket delivers a working feature and not just one of those three things.

supportengineer 3y ago

I've seen QA/QE greatness and it was similar to how you describe. A different chain of command for deciding if releases are certified for production. Different incentive structures as well. Not to mention, at one recent employer, the QE team wrote an enormous amount of code to perform their tests - It was more LOC than the modules being tested/certified.

eitally 3y ago
I had a team like that once. It was glorious. And ultimately, I'm convinced it led to overall faster development cycles because the baseline code quality & documentation was so much better than it would have been without such a great QA manager. The QA team, of course, was also technical -- mostly with SWE backgrounds -- and they were primarily colo'd in the same office as the dev team. I still remember the epiphany everyone had one planning cycle when it was mutually understood that by generally agreeing to use TDD, the QA team could participate actively in the real engineering planning and product development process. ... Then I left and my CIO let go the onshore QA team in favor of near term cost savings. Code quality went way down and within a year or two several apps needed to be entirely rewritten. Everything slowed down and people started pointing fingers, and before you knew it, it was time for "cloud native rearchitecting/reengineering" which required an SI to come in with "specialists".
- MichaelZuo 3y ago
  I've heard this enough times that I'm becoming convinced there's a lot of 'reinventing the wheel' busywork going on in the industry. And not even superior wheels, lower quality, more fragile wheels by clumsier wheel designers.
esafak 3y ago
So how was QA incentivized?
munificent 3y ago
> It was more LOC than the modules being tested/certified. So much code. I hope they had a QAQA team to test all that.

fizx 3y ago

How often did you release?

SkyPuncher 3y ago
That's my question as well. Bug free code is not the goal. Valuable product is.
Zelphyr 3y ago
At the first company we released a few times a year. You have to remember that CI/CD wasn't a thing back then. While I worked on a web-based product at this company, it was a time when it was still very common to release software on physical media like CD's. At the second company we integrated Agile so we released every two or weeks (I can't recall which). The first time I heard about CI/CD was at this company from co-workers who came back from a conference all excited about it.

alkonaut 3y ago

I love my really thorough QA’s. Yes it’s an antipattern to let me as a dev lean too much on them catching what I won’t. But where I dread even running the code for a minute, they enjoy it. They take pride in figuring out edge cases far beyond any spec. They are definitely worth their weight in gold. It lets developers have confidence when changing things in the same sense a good type system does. For some classes of very interactive apps (e.g games) having unit and integration tests just doesn’t cover the parameter space.

ponector 3y ago
People here are talking about skillful QA worth their weight in gold. Unfortunately people in the industry who have actual power in planning budgets don't think so. An article is right. QA engineers now are viewed as janitors: no one respects then, better to outsource to cheap location.
- bluGill 3y ago
  Depends on industry. Some care about quality, they respect qa.

PH95VuimJjqBqy 3y ago

The QA culture has to be there, not just dictates that the QA has final say. I've seen companies where that's true and it was still trash because the QA were mostly low-paid contract workers who only did exactly what they were told and no more.

onlyrealcuzzo 3y ago
Worked at a company where QA had the final say - and that was by far the most toxic / worst environment I have ever been in. QA also REFUSED to let developers write automation tests, also REFUSED to let us run them ourselves. What a nightmare. YMMV, but just having the final say is not a silver bullet for sure.
- mikestew 3y ago
  That’s not because “QA had the final say”, it’s because your QA team were ass clowns. Any QA team that discourages dev from writing or running tests needs to be burnt to ground and rebuilt.
  - ponector 3y ago
    Unless dev is switching to write test 100% of time and become a tester it is highly not recommended to let developers write tests. That QA was not a clown, they've seen some shit... Would you let QA to write features in your production code?
    - ahtihn 3y ago
      Test code is just code. If you can write test code you can write production code. If you can write production code you can write tests. If your concern is that devs don't have the right mindset for testing, you can have them collaborate with a QA specialist to define the test cases and review the test implementation.
      - ponector 3y ago

ponector 3y ago

I've seen developers writing tests with no assertions, or with assertTrue(true). Always green, with 100% coverage! The same people have been asking: why should I write tests if I can write new features? And then one senior QA comes and destroys everything. Once I found that if I press f5 50 times during a minute then backend will go in outOfMemory while spinning requests to the database.

kcb 3y ago
That's why the dev teams tests and the QA teams tests are not mutually exclusive.
- ponector 3y ago
  Testing is continuous multilayered multistaged process. Testing should start before first developer wrote first line of code for the project. Architecture blueprints, set of requirements should be tested as early as possible. But that is not happening in real life, only in books. In real life pm will bring cheap contractor from India one month before the target release date.
feoren 3y ago
> I've seen developers writing tests with no assertions This can be OK if the code executing without throwing exceptions is itself testing something. If you have a lot of assertions written directly into the code, as pre- or post-conditions for instance. But I'm guessing that wasn't the case here.
- WrongAssumption 3y ago
  Why would it be ok to not test the assertions would be triggered if the conditions are bad? How would you verify that the assertions are correct with just a happy path? If you run a code with pre-post assertions, then remove all the assertions the same test will continue to pass.
  - feoren 3y ago

mikestew 3y ago

I can't count how many products I've seen with 100% code coverage that objectively, quantifiably doesn't work. That’s because code coverage doesn’t find the bugs that result from code you didn’t write, but should have. Code coverage is but one measure, and to treat it as the measure is folly. (But, yes, I have heard a test manager at a large software company we’ve all heard of declare that test team was done because 100% coverage.)

thaumasiotes 3y ago
It's more because code coverage isn't measured correctly. What you want is branch coverage. If you have something like this: if condition1: do_something1() if condition2: do_something2() if condition3: do_something3() There are 8 possible paths for the code to follow here, but you can cover 100% of lines of code in one test. If your code coverage measure tells you that testing the case where conditions 1, 2, and 3 are all true achieves 100% coverage, your measure is worthless. Covering this code cannot be done in less than eight tests.
- alternatex 3y ago
  I think most code coverage analysis tools are smart enough to consider flow control statements. At least when it comes to C# ones.
  - fho 3y ago
    They do, but that's not the point GP made. An example how this could fail is if the first branch sets up something that branch two uses. Two tests are written that call branch 1&2 and not call them. 100% code coverage, even the non-happy path was tested. But in reality, if those branches have any interaction, you would need to write eight test cases for every combination of branches being run and not being run.
    - thaumasiotes 3y ago
      Well, again, eight is a lower bound. If the software uses any values beyond the bare minimum (here) of three separate booleans, you're likely to need more. Take this simplified code: if condition1: do_something() You need one test for code coverage and two for branch coverage, but if there's a bug in the code as written you'll find that you actually need more. Assume that we want to run do_something() when, and only when, condition2 is true. Now we have four cases: - condition1 && condition2: do_something() runs, which is correct. - condition1 && !condition2: do_something() runs, which is incorrect. - !condition1 && !condition2: do_something() does not run, which is correct. - !condition1 && condition2: do_something() does not run, which is incorrect. If you happened to write tests for the first and third of those options, your tests will make it look like your code works, and your (official) coverage metrics will look comprehensive. But because your coverage metrics (in reality) are actually terrible, your code is secretly not working. The big problem we've just run into is that the number of test cases we need depends on the number of bugs in our code. If the goal of writing tests is to prove that we don't have bugs, this is a terrible result; it means our goal is fundamentally impossible to achieve. We don't know whether we have bugs, so we don't know how many tests we need.

bluGill 3y ago

Automated tests take manual testing from 60% to 50% of the time to develop quality software. Valuable, but not a bullet to manual tests.

hedora 3y ago

The best approach I’ve seen is to do all of the above. - Have QA run pessimal versions of real use cases. Trying to sell a word processor to lawyers? Format the entire US legal code and a bajillion contracts in it, then duplicate it 10x and start filing usability/performance bugs. - Have the engineers test everything with randomly generated workloads before committing. Run those tests nightly, and fix all the crashes / failures. - Have Product Management (remember them?) work with marketing and sales to figure out what absolutely has to ship, and when. Make sure it only takes one of the above three groups to stop ship, and also to stop non-essential development tasks.

Arainach 3y ago
The devil is in the details of "essential development tasks". An uncountable number of products have died or devolved because "we don't have time to do it that way, put in the quick fix"

turok 3y ago

We try to do this, QA owns deployment to production and has the final say if a feature needs a re-write. The almost adversarial incentives of dev and QA are why it works. Dev wants to close ticket, QA wants the feature to work, Product wants to tick a box, all collaborate so that the closed ticket delivers a working feature and not just one of those three things.