7 ms·

Surprised to see the negativity here. I have worked in environments with traditional manual QA, and environments where all development is test-driven and nobody

by ef4 11y ago

Surprised to see the negativity here. I have worked in environments with traditional manual QA, and environments where all development is test-driven and nobody is allowed to merge a feature that lacks automated test coverage. Both the productivity and the quality were higher in the places with fully automated testing. Which is not shocking at all: does anybody really think a human can run through 800 test cases better than a computer can? It's not a magic way to save money -- the developers obviously end up spending time writing tests. But the long-term value of those tests is cumulative, whereas the effort spent on manual testing is spent anew every release. Manual review is still good for noticing things that "feel wrong" or for helping think up new corner cases. But those bleed into product owner & design concerns, and aren't really a separate function.

Pizzalover 11y ago
Surprised? Man, this is HN!!
arielweisberg 11y ago
Moving from an environment with a 10:1 dev:qa to 2:1 showed me what happens when dev is not responsible for shipping working software. No thanks. It's a bunch of deflection and diffusion of responsibility coupled with high latency flakey interactions between different teams. Everything that can slip through the cracks does slip through the cracks. I'm sure QA can be done well, but I am convinced that giving your devs a pass to not finish their work is a dead end in several dimensions.
- pc86 11y ago
  The best way I've seen it done is when another developer, on the same team, who is responsible for the same product, reviews your code. Not someone who writes tests for a living, someone who does the exact same job as you on the same product. Any time I've experienced something different it's been exactly as you described.
  - jrumbut 11y ago
    I agree with you that that is a necessary part of the process and a great first step for small teams moving away from "just release it and see if it works". The unfortunate reality though is that people in the same role on the same team often have the same misconceptions and blind spots. Now getting devs to see the product through the users' eyes goes a long way toward solving that, but if you have a process and team of devs that are doing that you're way ahead of the game in a lot of ways.
- deleted 11y ago
  [deleted]
- jboy55 11y ago
  I'm in full agreement. In practise, a manual QA team encourages Devs to throw shit over the wall and expect someone else to do some basic sanity checks they should have already done. By the time those are done, what the QA team theoretically could find gets shipped. Then, when its discovered in Prod, the QA team will get in the way of a speedy fix.
  - DougWebb 11y ago
    That's a management problem, not a problem with having a QA team. The top-level QA manager and the top-level Dev manager should both report to the CTO, and the Dev manager should be judged by how many issues the QA team finds. To keep things fair, the QA manager should not be rewarded nor penalized based on the number of issues found pre-release, and everyone should be rewarded or penalized based on the number of issues found post-release. This keeps the devs incentivized to make sure everything works before the code goes to QA, and it keeps everyone incentivized to eliminate as many bugs as possible before release.
    - fein 11y ago
      Have you ever worked for a fortune 100 company? The dev manager has probably never met the CTO. Im in this right now with my currrent job. I still keep my builds and test them longer than i should. Less broken shit gets through, but i regularly delay my qa ba's because of it.
      - DougWebb 11y ago
        So, not the CTO, but some shared higher-level manager. The point is that the QA lead cannot report into Development, and Development cannot report into QA. Both of those arrangements can lead to a bad actor stopping the other group from doing the right thing. (Eg: if QA reports to Dev, then Dev manager can force bad code to ship, and if Dev reports to QA than QA manager can prevent shipping and force Dev to waste time on excessive/pointless test code development that's not cost-justified.)
        11y ago
- martin1975 11y ago
  10:1? You do realize QA has to have a semblance of life too? Adding more QA isn't designed to screw up the process, but to alleviate the workload. If you want to blame anything or anyone, I'd usually shoot for 1) hiring incompetent QA that isn't SDETs but merely just black-box QA competency, and 2) process - or lack of process, i.e. an agile process of building a little, testing a little, and delivering consistently a working product on pretty much any basis - monthly, weekly even.... That's the ideal of course. If this is in place, adding extra QA can be beneficial. None of this of course is excuse for a dev not producing code that works at least in its happy path plus/minus a few of the most obvious exceptions/error paths...
- Estragon 11y ago
  How is it that in the 2:1 environment developers aren't held accountable when problems turn up in code they're responsible for?
jsprogrammer 11y ago
QA is essential. How else do you know that you have built what you set out to built? Traditional manual QA is highly effective, so much so, that you can get huge gains by automating it. QA occurs at many levels and it makes sense to have a dedicated QA team for each level. Generally, the QA team for a level should be the exact people who requested and/or approved the specific feature subject to QA, as they are in the best position to confirm or deny that the feature meets the specification, as they created and/or approved it.
- pc86 11y ago
  But you're talking about QA as a general process, not QA as is meant in the article: > Software engineers at Yahoo are no longer permitted to hand off their completed code to another team for cross checking. Dev Team A was giving a batch of code to Dev Team B to review. I agree that the business user/product owner/whatever should be reviewing everything in test prior to approval and in production after but I'm not sure that's the article means by QA.
pc86 11y ago
The arguments here such as "oh they just make the devs do it and don't pay them anymore" are ridiculous. Nobody's working 80 hour weeks doing two full-time jobs at once.
- tajen 11y ago
  Please let me tip a name for that: "Volkswagening" - assigning the quality check to the production team.
  - vvanders 11y ago
    Haha, I'm going to start using that term. That's a great description.
    - tajen 11y ago
      To be fair the exact meaning of Volkswagening is to fake the behavior during official tests. Example: http://qz.com/515100/samsung-is-accused-of-volkswagening-its-tvs-to-oversell-their-energy-efficiency/ http://qz.com/515100/samsung-is-accused-of-volkswagening-its... Note that volkswagening.com was registered 20 days after the scandal, but is available for bid, if you want to do something fun with it.
      - smsm42 11y ago
        VW lawyers would probably throw a fit if somebody actually used that domain for anything commercial.
        resoluteteeth 11y ago
        Perhaps it could display different, positive content when accessed from Volkswagen IP addresses, to ensure they wouldn't find out.
  - smsm42 11y ago
    It doesn't even have to be intentional. If you write the code, you always have in mind some picture of how it should be used. That picture means you ignore the ways to use it which you not intended. So when the code is used in that way, it breaks. But you won't ever write a test for it because you would never think such usage is possible - exactly because you know too much about the right way of how it should be, so you start to think it's how it is.
duaneb 11y ago
> Which is not shocking at all: does anybody really think a human can run through 800 test cases better than a computer can? I think this rather misses the point; it's the bug that doesn't have a test case where QA helps.
- philk10 11y ago
  Exactly. And having the computer run through 800 checks means a skilled tester can do what they are good at - finding bugs.
  - IanCal 11y ago
    This to me was always the point of automating tests. One of the best QA people I knew refused to look at the user stories. He was brilliant at finding things that fit the spec but didn't make sense and things that users might do that weren't specified. He found bugs in the specs, gaps in the specs and just plain untested behaviour. QA is a job that requires skill, benefits greatly from technical understanding and requires a lot of domain knowledge.
    - philk10 11y ago
      Yup, that's pretty much my job. The devs do great testing and I try and pick holes in what they miss.
    - lotyrin 11y ago
      This is why specification should happen collaboratively with everyone with a stake (and that includes QA just as much as BA), you can fix them before they're developed against.
    - silverbax88 11y ago
      This is what a great QA tester will do. Good devs don't ship code with known bugs - but they will miss things because they code to the feature/user story/known behavior. That's what their job is. A good QA person's job will be to spend a whole lot of time thinking about how to break the dev's app that the dev missed.
protomyth 11y ago
Did the QA you dealt with not have any automated testing tools? That seems rather foolish to have your QA team manually test every feature every time without an automated smoke test.
serge2k 11y ago
I think it's important to recognize that neither option is perfect by itself. Manual QA can still be an important process. That being said, if you have to pick one or the other then you go with automated tests.
dingaling 11y ago
> does anybody really think a human can run through 800 test cases better than a computer can? Of course not. But automated tests are just the entry ticket to the release process. A computer can't replicate the irrationality, laziness and guile of a human end-user. That's what good QA engineers test.
smsm42 11y ago
It sounds like there is some opposition between automatic tests and QA. But those do not have to be opposed - one can have both. In fact, for many types of software I don't see how you can avoid having both. The only choice you have who is doing your QA - your employees or your end users. Maybe Yahoo can afford the latter. But for some companies it would be a disaster.
otakucode 11y ago
Even more important is that the developers spend time writing tests and their schedules are not expanded to accomodate this, nor is their pay increased. They are simply expected to cut corners elsewhere, or work more hours, or whatever is necessary. That's the real savings.
greggman 11y ago
It really depends on what you're making and if it's possible to automate testing AND if you remembered to test it. Chrome appeared to have broken full screen movie playback for 5% of users once. Something changed, perf got bad, people who previously could play full screen video on a low end machine suddenly got playback too slow to be useful. They just stopped going full screen. No bug reports came in. Nothing in the testing infrastructure caught it. I only notice because my father tried to show me a video on his atom based netbook. Similarly going full screen on a 2 monitor setup broke once. Again, no automated test. The Web Text to Speech API is broken on every browser it's in. It will work with a simple sample but start and stop it a few times and it will break. I suspect because again there is no easy way to test it. There's lots of others. automated testing is not a substitute for QA IMO.

by ef4 11y ago

Pizzalover 11y ago

Surprised? Man, this is HN!!

arielweisberg 11y ago

Moving from an environment with a 10:1 dev:qa to 2:1 showed me what happens when dev is not responsible for shipping working software. No thanks. It's a bunch of deflection and diffusion of responsibility coupled with high latency flakey interactions between different teams. Everything that can slip through the cracks does slip through the cracks. I'm sure QA can be done well, but I am convinced that giving your devs a pass to not finish their work is a dead end in several dimensions.

pc86 11y ago
The best way I've seen it done is when another developer, on the same team, who is responsible for the same product, reviews your code. Not someone who writes tests for a living, someone who does the exact same job as you on the same product. Any time I've experienced something different it's been exactly as you described.
- jrumbut 11y ago
  I agree with you that that is a necessary part of the process and a great first step for small teams moving away from "just release it and see if it works". The unfortunate reality though is that people in the same role on the same team often have the same misconceptions and blind spots. Now getting devs to see the product through the users' eyes goes a long way toward solving that, but if you have a process and team of devs that are doing that you're way ahead of the game in a lot of ways.
deleted 11y ago
[deleted]
jboy55 11y ago
I'm in full agreement. In practise, a manual QA team encourages Devs to throw shit over the wall and expect someone else to do some basic sanity checks they should have already done. By the time those are done, what the QA team theoretically could find gets shipped. Then, when its discovered in Prod, the QA team will get in the way of a speedy fix.
- DougWebb 11y ago
  That's a management problem, not a problem with having a QA team. The top-level QA manager and the top-level Dev manager should both report to the CTO, and the Dev manager should be judged by how many issues the QA team finds. To keep things fair, the QA manager should not be rewarded nor penalized based on the number of issues found pre-release, and everyone should be rewarded or penalized based on the number of issues found post-release. This keeps the devs incentivized to make sure everything works before the code goes to QA, and it keeps everyone incentivized to eliminate as many bugs as possible before release.
  - fein 11y ago
    Have you ever worked for a fortune 100 company? The dev manager has probably never met the CTO. Im in this right now with my currrent job. I still keep my builds and test them longer than i should. Less broken shit gets through, but i regularly delay my qa ba's because of it.
    - DougWebb 11y ago
      So, not the CTO, but some shared higher-level manager. The point is that the QA lead cannot report into Development, and Development cannot report into QA. Both of those arrangements can lead to a bad actor stopping the other group from doing the right thing. (Eg: if QA reports to Dev, then Dev manager can force bad code to ship, and if Dev reports to QA than QA manager can prevent shipping and force Dev to waste time on excessive/pointless test code development that's not cost-justified.)
      - 11y ago
martin1975 11y ago
10:1? You do realize QA has to have a semblance of life too? Adding more QA isn't designed to screw up the process, but to alleviate the workload. If you want to blame anything or anyone, I'd usually shoot for 1) hiring incompetent QA that isn't SDETs but merely just black-box QA competency, and 2) process - or lack of process, i.e. an agile process of building a little, testing a little, and delivering consistently a working product on pretty much any basis - monthly, weekly even.... That's the ideal of course. If this is in place, adding extra QA can be beneficial. None of this of course is excuse for a dev not producing code that works at least in its happy path plus/minus a few of the most obvious exceptions/error paths...
Estragon 11y ago
How is it that in the 2:1 environment developers aren't held accountable when problems turn up in code they're responsible for?

jsprogrammer 11y ago

QA is essential. How else do you know that you have built what you set out to built? Traditional manual QA is highly effective, so much so, that you can get huge gains by automating it. QA occurs at many levels and it makes sense to have a dedicated QA team for each level. Generally, the QA team for a level should be the exact people who requested and/or approved the specific feature subject to QA, as they are in the best position to confirm or deny that the feature meets the specification, as they created and/or approved it.

pc86 11y ago
But you're talking about QA as a general process, not QA as is meant in the article: > Software engineers at Yahoo are no longer permitted to hand off their completed code to another team for cross checking. Dev Team A was giving a batch of code to Dev Team B to review. I agree that the business user/product owner/whatever should be reviewing everything in test prior to approval and in production after but I'm not sure that's the article means by QA.

pc86 11y ago

The arguments here such as "oh they just make the devs do it and don't pay them anymore" are ridiculous. Nobody's working 80 hour weeks doing two full-time jobs at once.

tajen 11y ago
Please let me tip a name for that: "Volkswagening" - assigning the quality check to the production team.
- vvanders 11y ago
  Haha, I'm going to start using that term. That's a great description.
  - tajen 11y ago
    To be fair the exact meaning of Volkswagening is to fake the behavior during official tests. Example: http://qz.com/515100/samsung-is-accused-of-volkswagening-its-tvs-to-oversell-their-energy-efficiency/ http://qz.com/515100/samsung-is-accused-of-volkswagening-its... Note that volkswagening.com was registered 20 days after the scandal, but is available for bid, if you want to do something fun with it.
    - smsm42 11y ago
      VW lawyers would probably throw a fit if somebody actually used that domain for anything commercial.
      - resoluteteeth 11y ago
        Perhaps it could display different, positive content when accessed from Volkswagen IP addresses, to ensure they wouldn't find out.
- smsm42 11y ago
  It doesn't even have to be intentional. If you write the code, you always have in mind some picture of how it should be used. That picture means you ignore the ways to use it which you not intended. So when the code is used in that way, it breaks. But you won't ever write a test for it because you would never think such usage is possible - exactly because you know too much about the right way of how it should be, so you start to think it's how it is.

duaneb 11y ago

> Which is not shocking at all: does anybody really think a human can run through 800 test cases better than a computer can? I think this rather misses the point; it's the bug that doesn't have a test case where QA helps.

philk10 11y ago
Exactly. And having the computer run through 800 checks means a skilled tester can do what they are good at - finding bugs.
- IanCal 11y ago
  This to me was always the point of automating tests. One of the best QA people I knew refused to look at the user stories. He was brilliant at finding things that fit the spec but didn't make sense and things that users might do that weren't specified. He found bugs in the specs, gaps in the specs and just plain untested behaviour. QA is a job that requires skill, benefits greatly from technical understanding and requires a lot of domain knowledge.
  - philk10 11y ago
    Yup, that's pretty much my job. The devs do great testing and I try and pick holes in what they miss.
  - lotyrin 11y ago
    This is why specification should happen collaboratively with everyone with a stake (and that includes QA just as much as BA), you can fix them before they're developed against.
  - silverbax88 11y ago
    This is what a great QA tester will do. Good devs don't ship code with known bugs - but they will miss things because they code to the feature/user story/known behavior. That's what their job is. A good QA person's job will be to spend a whole lot of time thinking about how to break the dev's app that the dev missed.

protomyth 11y ago

Did the QA you dealt with not have any automated testing tools? That seems rather foolish to have your QA team manually test every feature every time without an automated smoke test.

serge2k 11y ago

I think it's important to recognize that neither option is perfect by itself. Manual QA can still be an important process. That being said, if you have to pick one or the other then you go with automated tests.

dingaling 11y ago

> does anybody really think a human can run through 800 test cases better than a computer can? Of course not. But automated tests are just the entry ticket to the release process. A computer can't replicate the irrationality, laziness and guile of a human end-user. That's what good QA engineers test.

smsm42 11y ago

It sounds like there is some opposition between automatic tests and QA. But those do not have to be opposed - one can have both. In fact, for many types of software I don't see how you can avoid having both. The only choice you have who is doing your QA - your employees or your end users. Maybe Yahoo can afford the latter. But for some companies it would be a disaster.

otakucode 11y ago

Even more important is that the developers spend time writing tests and their schedules are not expanded to accomodate this, nor is their pay increased. They are simply expected to cut corners elsewhere, or work more hours, or whatever is necessary. That's the real savings.

greggman 11y ago

It really depends on what you're making and if it's possible to automate testing AND if you remembered to test it. Chrome appeared to have broken full screen movie playback for 5% of users once. Something changed, perf got bad, people who previously could play full screen video on a low end machine suddenly got playback too slow to be useful. They just stopped going full screen. No bug reports came in. Nothing in the testing infrastructure caught it. I only notice because my father tried to show me a video on his atom based netbook. Similarly going full screen on a 2 monitor setup broke once. Again, no automated test. The Web Text to Speech API is broken on every browser it's in. It will work with a simple sample but start and stop it a few times and it will break. I suspect because again there is no easy way to test it. There's lots of others. automated testing is not a substitute for QA IMO.