5 ms·

I've found that the orgs I've been in that expect Engineers + Product to own QA have had a healthier dev process than orgs that have a specific QA process (this

I've found that the orgs I've been in that expect Engineers + Product to own QA have had a healthier dev process than orgs that have a specific QA process (this is all for web platforms). I think it helps create that strong sense of ownership I'm hoping for. This is probably not universal, though - what sides of the fence of org design do people fall into, here?

iEchoic 6y ago
Microsoft used to have separate roles for dev and test/QA (SDE and SDET), and is a good case study here. Sometime around 2013-ish, they merged the SDET role into the SDE role, and made SDEs responsible for automated testing and quality. This was widely recognized as an important decision that improved both product velocity and quality by time I had left a couple years later. I don’t think this is an anomaly. Every engineering team I’ve been on since has given this responsibility to the developers writing the code, and every one has been stronger for it.
- nxc18 6y ago
  Maybe it enhanced velocity, but I can tell you that for sure it did not improve quality for Office, Windows, or Visual Studio (not Code). I work with all of those products, and have worked with them before and after the big layoff. The quality of those offerings took a serious hit when they outsourced QA to their customers. I try to submit feedback and half the time the feedback tool itself fails to work, the other half of the time my bug is closed or ignored so that the team can 'focus on issues with a broad customer impact'. Is there any evidence that Microsoft's product quality has improved recently? The only way I can imagine justifying that point of view is if you divide by the cost to get that quality, which is a useless metric to me as a customer. Maybe I just experience bugs that no one cares about, but I no longer recognize Microsoft Office or Windows as solid software given the massive increase in the volume of bugs I experience.
- Baeocystin 6y ago
  I have friends who have worked for Microsoft for ~20 years. I'm a small-business IT consultant. My MS friends have the feeling that it was a painful transition, yet ultimately a good thing, and that quality has gone up. My experience on the ground, so to speak, is the exact opposite. The May(!) update still has showstopping bugs with a not-small portion of my clients. Last year, one was bitten by the wipe-the-user-folder upgrade bug. The trust in running windows updates is at an all-time low amongst the general users, and I can't blame them. I trust my friends. I also trust my own experiences. I wonder where the disconnect is?
- ThrowawayB7 6y ago
  > "This was widely recognized as an important decision that improved both product velocity and quality by time I had left a couple years later." I'm sure that's what the execs say at the all-hands meetings but ask the former MS SDETs who still use MS products and you'd hear a much different opinion. ;) My standard quip when hitting yet another bug in a MS product, which is an all too frequent occurrence nowadays, is "Gosh, maybe MS should consider hiring some SDETs."
- rorykoehler 6y ago
  Maybe they should apply QA to MS Teams. I have to restart the app many times a day to get it to work.
beardedetim 6y ago
Every org I've worked for that has had a dedicated QA team has had, by far, the shittiest quality of code and the most amount of bugs in the system/outages/issues in production. I am not sure _why_ this is but I think it has something to do with the "Not my problem" mentality where engineers say "QA will figure out if this works or not" instead of "I'm responsible for figuring out of this works". It hasn't become a red flag for me _yet_ but it's definitely _a_ flag.
- stevezsa8 6y ago
  As a QA myself, every project I've worked on except maybe one... has had good shared ownership of quality. Maybe it's because we were embedded in the dev team. Sorry you didn't experience the same thing.
- toast0 6y ago
  Be that as it may, a team that would have shitty code with QA will likely have shittier code if you fire QA. Case Study: Windows 10. That said, an understaffed QA team can be really helpful. Not enough people means you can't ship garbage and expect QA to catch it; they'll catch many things, but not everything. Not enough people also encourages efficiency in testing --- test the most important stuff well, test the other things opportunistically. A QA team also can really help turn customer problem reports into actionable bug reports.
- ZitchDog 6y ago
  The red flag for me is when QA is considered a non-coding team, divorced from product. QA engineers should be very technical, and should be writing code or automating. A non-automated QA process kills the feedback loop of the product.
- hmillison 6y ago
  Agree with this, it's very important for engineers to own the quality and reliability of their own code. The team I was on that had embedded QA on each team had barely any unit tests because developers were used to relying on QA to catch bugs for them. When I've been on teams without formal QA teams, the team was focusing way more on having good automated test practices in place. This also applies from the SDET/SDE separation, which I see someone else in an adjacent thread mention. I've found a lot of value from taking ownership of both the code and the automated browser testing code. I found when those roles are separate, the browser tests end up breaking too often due to developers not being aware of how coupled they are to the current implementation. I also found when a dedicated team is writing browser tests, it becomes easy to go overboard with how much you test.
mathieuh 6y ago
I agree, the best testing process I've experienced still had people whose role was QA, but that was because they were considered experts and would lead the direction of QA and do final acceptance testing before a release. The process we used was the three amigos process, where the ticket dev, BA and a QA would have a 20-30 minute sit down and draw up a mind map of test cases linked to the acceptance criteria. Once the mind map was created, anyone technical could pick up the actual writing of the tests/exploratory testing, whether that's the original dev, a QA, or any other dev. Because devs were all testing each other's tickets instead of just hoofing them over the fence to QA I found that I got a much better understanding of the different areas of the projects. Only downside was it required a lot of meetings to really flesh out the tickets and make the acceptance criteria as detailed as possible.
gwbas1c 6y ago
Honestly, I think it goes down to organizational momentum around writing tests. If you have good engineer-written and maintained tests, then you don't need QA. The problem comes from relying on developers to do manual tests. It works as long as the product is simple and newcomers can learn the entire product. But, developers doing manual testing won't scale once the product becomes too complicated to remember all the corner cases and gotchas. Even worse, newcomers just won't understand the product as well as the people who originally wrote it. So, if you have a complicated product, and a lot of debt in developer-written tests, then you need QA. With regard to the web, it's very easy to write automated tests against server-generated HTML or an HTTP API. In contrast, other platforms are all over the place in how easy / hard it is for a developer to write an automated test.
- dmurray 6y ago
  > With regard to the web, it's very easy to write automated tests against server-generated HTML or an HTTP API. Funny, I find the opposite. Code for the web is hard to write reliable tests for. Typically I want to test that my component actually exists and is visible and does the right thing in a browser, so I need to produce the whole page instead of just the unit under test. This means that bugs - or just changes - in other parts of the page can make my test fail. I'm not a front end expert and my colleagues rarely have been either, so maybe I'm missing something about best practices here, but I've definitely had the opposite experience to what you describe.
tmcneal 6y ago
We sell a product in the QA space and get to talk to a lot companies about this.* The pattern I see is that companies in the 1-10 employee range tend to have a QA process that comes about organically. You start out with zero customers so a bug doesn't really "cost" anything. As you get customers, the cost of a bug grows to a point where it makes sense to do manual test passes - maybe before a feature release, maybe after every deployment. Usually it's in a Google Doc or Google Sheet, and it's either one person running through the whole thing or a bunch of people divide-and-conquering it. Eventually QA ends up being owned by either the Engineering team or a standalone QA team. I've seen both work, and both not work. Generally if it's not working for Engineering, it means they're not automating enough or the tests are flaky and they're just ignoring it. If the QA team approach is not working, they're probably overworked... Likely because the engineering org has deeper quality issues they need to fix, but they're trying to make up for it by throwing bodies at it in the QA org. * We have a no-code test automation product that is used by both QA and Engineers: https://reflect.run https://reflect.run
- ALittleLight 6y ago
  I think "manual test pass" is very seductive and very dangerous. Compared to writing good automation, manual testing once is cheaper, easier, and faster. Manual testing indefinitely is more expensive, harder, and slower. Short term thinking gets you stuck with flawed and burdensome manual testing. In my view, the best approach is rigorous automation and manual exploratory testing which will find the kind of bugs automation misses and inspire new test cases.
  - pydry 6y ago
    There's definitely an inflection point. I think automation is often a waste of time at the beginning of a project too. Bugs don't cost much (if anything). Building functionality that gets tossed away is routine. Tests don't provide value - getting the code in front of customers and seeing if it's what they really want provides value. The problem is that once people get used to not having automated tests is can be difficult to build up the habit. Unit tests are also often hard (if not impossible) to retrofit, although IMO that's a bigger problem with unit tests than it is doing testing at a later stage in the project.
triztian 6y ago
At the company I work for QA is part of the development team; alongside the Engineers and Product Manager, where our products are a set of mobile applications for iOS, Android and tvOS. I think we have a good code-quality/speed-to-release ratio with well known tech debt that is actively being cleaned out, that is also tested after refactored out. Having QA Tester be part of the team makes it really easy to adjust features, adjust the release if an update is needed and it is not passing quite yet quality wise. It also makes it really frictionless specifying what to expect from a ticket/feature and how to baseline test it, the tester also performs some tests of his/her own, the QA tester runs unit, automated UI and performs manual regression tests. I've been part of teams that have QA testers outsourced as a dedicated independent team and it definitively impacted code quality and the release turnaround, mostly because of delayed communication and mis-alignment on what to test or the expectations of the test.
larrik 6y ago
In my experience as a dev and now PM, there's a few issues with devs doing QA: 1) They are expensive. You can hire 3 or 4 QA people for every dev a lot of times. Why waste that money on devs? 2) They aren't that great at. They know how the program works, so most struggle to do things outside of the "happy path". I had a QA (actually my PM) paste in an entire book into a text field and break the site that way. Devs don't think of stuff like that. Another uses the browser's back button constantly, which none of the devs seem to do. That said, it doesn't excuse crappy code. If a code change/addition fails QA, that's on the Dev. However, if that failure hits Prod, that's on QA...
- Jtsummers 6y ago
  A problem that lead to (2): Developers aren't paid to test, the majority of their time is spent making new code because that's what management/customers want. Having one person have to wear 2+ hats doesn't work when they can never take one of the hats off. If they were legitimately given the amount of time to test as they get to code, your project could take twice as long, but it would be more thoroughly tested. And, likely, it wouldn't take twice as long because they'd actually have to think about testability and design earlier. And by "test" I do not mean just unit tests. Designing and implementing integration tests, assembling regression test suites, etc. Devs focused on just unit tests will never catch the errors that a dedicated V&V team can find.
- dntrkv 6y ago
  Once you assign QA to your devs, they will (should) figure out ways to automate as much as possible to reduce the burden that QA has on the team. In my experience, QA teams are usually starved of engineering resources, so a lot of things that can be automated are done manually or the automation is janky and constantly breaks. Once these issues are brought to the forefront, it becomes a priority to fix so that your expensive engineers aren't wasting time chasing bugs or fixing broken automation.
- quadrifoliate 6y ago
  > You can hire 3 or 4 QA people for every dev a lot of times. Why waste that money on devs? Attitudes like this will get you...not great QA. In my experience a sufficiently strong QA person can both cost and be worth more than adding another dev to the team.

iEchoic 6y ago

Microsoft used to have separate roles for dev and test/QA (SDE and SDET), and is a good case study here. Sometime around 2013-ish, they merged the SDET role into the SDE role, and made SDEs responsible for automated testing and quality. This was widely recognized as an important decision that improved both product velocity and quality by time I had left a couple years later. I don’t think this is an anomaly. Every engineering team I’ve been on since has given this responsibility to the developers writing the code, and every one has been stronger for it.

nxc18 6y ago
Maybe it enhanced velocity, but I can tell you that for sure it did not improve quality for Office, Windows, or Visual Studio (not Code). I work with all of those products, and have worked with them before and after the big layoff. The quality of those offerings took a serious hit when they outsourced QA to their customers. I try to submit feedback and half the time the feedback tool itself fails to work, the other half of the time my bug is closed or ignored so that the team can 'focus on issues with a broad customer impact'. Is there any evidence that Microsoft's product quality has improved recently? The only way I can imagine justifying that point of view is if you divide by the cost to get that quality, which is a useless metric to me as a customer. Maybe I just experience bugs that no one cares about, but I no longer recognize Microsoft Office or Windows as solid software given the massive increase in the volume of bugs I experience.
Baeocystin 6y ago
I have friends who have worked for Microsoft for ~20 years. I'm a small-business IT consultant. My MS friends have the feeling that it was a painful transition, yet ultimately a good thing, and that quality has gone up. My experience on the ground, so to speak, is the exact opposite. The May(!) update still has showstopping bugs with a not-small portion of my clients. Last year, one was bitten by the wipe-the-user-folder upgrade bug. The trust in running windows updates is at an all-time low amongst the general users, and I can't blame them. I trust my friends. I also trust my own experiences. I wonder where the disconnect is?
ThrowawayB7 6y ago
> "This was widely recognized as an important decision that improved both product velocity and quality by time I had left a couple years later." I'm sure that's what the execs say at the all-hands meetings but ask the former MS SDETs who still use MS products and you'd hear a much different opinion. ;) My standard quip when hitting yet another bug in a MS product, which is an all too frequent occurrence nowadays, is "Gosh, maybe MS should consider hiring some SDETs."
rorykoehler 6y ago
Maybe they should apply QA to MS Teams. I have to restart the app many times a day to get it to work.

beardedetim 6y ago

Every org I've worked for that has had a dedicated QA team has had, by far, the shittiest quality of code and the most amount of bugs in the system/outages/issues in production. I am not sure _why_ this is but I think it has something to do with the "Not my problem" mentality where engineers say "QA will figure out if this works or not" instead of "I'm responsible for figuring out of this works". It hasn't become a red flag for me _yet_ but it's definitely _a_ flag.

stevezsa8 6y ago
As a QA myself, every project I've worked on except maybe one... has had good shared ownership of quality. Maybe it's because we were embedded in the dev team. Sorry you didn't experience the same thing.
toast0 6y ago
Be that as it may, a team that would have shitty code with QA will likely have shittier code if you fire QA. Case Study: Windows 10. That said, an understaffed QA team can be really helpful. Not enough people means you can't ship garbage and expect QA to catch it; they'll catch many things, but not everything. Not enough people also encourages efficiency in testing --- test the most important stuff well, test the other things opportunistically. A QA team also can really help turn customer problem reports into actionable bug reports.
ZitchDog 6y ago
The red flag for me is when QA is considered a non-coding team, divorced from product. QA engineers should be very technical, and should be writing code or automating. A non-automated QA process kills the feedback loop of the product.
hmillison 6y ago
Agree with this, it's very important for engineers to own the quality and reliability of their own code. The team I was on that had embedded QA on each team had barely any unit tests because developers were used to relying on QA to catch bugs for them. When I've been on teams without formal QA teams, the team was focusing way more on having good automated test practices in place. This also applies from the SDET/SDE separation, which I see someone else in an adjacent thread mention. I've found a lot of value from taking ownership of both the code and the automated browser testing code. I found when those roles are separate, the browser tests end up breaking too often due to developers not being aware of how coupled they are to the current implementation. I also found when a dedicated team is writing browser tests, it becomes easy to go overboard with how much you test.

mathieuh 6y ago

I agree, the best testing process I've experienced still had people whose role was QA, but that was because they were considered experts and would lead the direction of QA and do final acceptance testing before a release. The process we used was the three amigos process, where the ticket dev, BA and a QA would have a 20-30 minute sit down and draw up a mind map of test cases linked to the acceptance criteria. Once the mind map was created, anyone technical could pick up the actual writing of the tests/exploratory testing, whether that's the original dev, a QA, or any other dev. Because devs were all testing each other's tickets instead of just hoofing them over the fence to QA I found that I got a much better understanding of the different areas of the projects. Only downside was it required a lot of meetings to really flesh out the tickets and make the acceptance criteria as detailed as possible.

gwbas1c 6y ago

Honestly, I think it goes down to organizational momentum around writing tests. If you have good engineer-written and maintained tests, then you don't need QA. The problem comes from relying on developers to do manual tests. It works as long as the product is simple and newcomers can learn the entire product. But, developers doing manual testing won't scale once the product becomes too complicated to remember all the corner cases and gotchas. Even worse, newcomers just won't understand the product as well as the people who originally wrote it. So, if you have a complicated product, and a lot of debt in developer-written tests, then you need QA. With regard to the web, it's very easy to write automated tests against server-generated HTML or an HTTP API. In contrast, other platforms are all over the place in how easy / hard it is for a developer to write an automated test.

dmurray 6y ago
> With regard to the web, it's very easy to write automated tests against server-generated HTML or an HTTP API. Funny, I find the opposite. Code for the web is hard to write reliable tests for. Typically I want to test that my component actually exists and is visible and does the right thing in a browser, so I need to produce the whole page instead of just the unit under test. This means that bugs - or just changes - in other parts of the page can make my test fail. I'm not a front end expert and my colleagues rarely have been either, so maybe I'm missing something about best practices here, but I've definitely had the opposite experience to what you describe.

tmcneal 6y ago

We sell a product in the QA space and get to talk to a lot companies about this.* The pattern I see is that companies in the 1-10 employee range tend to have a QA process that comes about organically. You start out with zero customers so a bug doesn't really "cost" anything. As you get customers, the cost of a bug grows to a point where it makes sense to do manual test passes - maybe before a feature release, maybe after every deployment. Usually it's in a Google Doc or Google Sheet, and it's either one person running through the whole thing or a bunch of people divide-and-conquering it. Eventually QA ends up being owned by either the Engineering team or a standalone QA team. I've seen both work, and both not work. Generally if it's not working for Engineering, it means they're not automating enough or the tests are flaky and they're just ignoring it. If the QA team approach is not working, they're probably overworked... Likely because the engineering org has deeper quality issues they need to fix, but they're trying to make up for it by throwing bodies at it in the QA org. * We have a no-code test automation product that is used by both QA and Engineers: https://reflect.run https://reflect.run

ALittleLight 6y ago
I think "manual test pass" is very seductive and very dangerous. Compared to writing good automation, manual testing once is cheaper, easier, and faster. Manual testing indefinitely is more expensive, harder, and slower. Short term thinking gets you stuck with flawed and burdensome manual testing. In my view, the best approach is rigorous automation and manual exploratory testing which will find the kind of bugs automation misses and inspire new test cases.
- pydry 6y ago
  There's definitely an inflection point. I think automation is often a waste of time at the beginning of a project too. Bugs don't cost much (if anything). Building functionality that gets tossed away is routine. Tests don't provide value - getting the code in front of customers and seeing if it's what they really want provides value. The problem is that once people get used to not having automated tests is can be difficult to build up the habit. Unit tests are also often hard (if not impossible) to retrofit, although IMO that's a bigger problem with unit tests than it is doing testing at a later stage in the project.

triztian 6y ago

At the company I work for QA is part of the development team; alongside the Engineers and Product Manager, where our products are a set of mobile applications for iOS, Android and tvOS. I think we have a good code-quality/speed-to-release ratio with well known tech debt that is actively being cleaned out, that is also tested after refactored out. Having QA Tester be part of the team makes it really easy to adjust features, adjust the release if an update is needed and it is not passing quite yet quality wise. It also makes it really frictionless specifying what to expect from a ticket/feature and how to baseline test it, the tester also performs some tests of his/her own, the QA tester runs unit, automated UI and performs manual regression tests. I've been part of teams that have QA testers outsourced as a dedicated independent team and it definitively impacted code quality and the release turnaround, mostly because of delayed communication and mis-alignment on what to test or the expectations of the test.

larrik 6y ago

In my experience as a dev and now PM, there's a few issues with devs doing QA: 1) They are expensive. You can hire 3 or 4 QA people for every dev a lot of times. Why waste that money on devs? 2) They aren't that great at. They know how the program works, so most struggle to do things outside of the "happy path". I had a QA (actually my PM) paste in an entire book into a text field and break the site that way. Devs don't think of stuff like that. Another uses the browser's back button constantly, which none of the devs seem to do. That said, it doesn't excuse crappy code. If a code change/addition fails QA, that's on the Dev. However, if that failure hits Prod, that's on QA...

Jtsummers 6y ago
A problem that lead to (2): Developers aren't paid to test, the majority of their time is spent making new code because that's what management/customers want. Having one person have to wear 2+ hats doesn't work when they can never take one of the hats off. If they were legitimately given the amount of time to test as they get to code, your project could take twice as long, but it would be more thoroughly tested. And, likely, it wouldn't take twice as long because they'd actually have to think about testability and design earlier. And by "test" I do not mean just unit tests. Designing and implementing integration tests, assembling regression test suites, etc. Devs focused on just unit tests will never catch the errors that a dedicated V&V team can find.
dntrkv 6y ago
Once you assign QA to your devs, they will (should) figure out ways to automate as much as possible to reduce the burden that QA has on the team. In my experience, QA teams are usually starved of engineering resources, so a lot of things that can be automated are done manually or the automation is janky and constantly breaks. Once these issues are brought to the forefront, it becomes a priority to fix so that your expensive engineers aren't wasting time chasing bugs or fixing broken automation.
quadrifoliate 6y ago
> You can hire 3 or 4 QA people for every dev a lot of times. Why waste that money on devs? Attitudes like this will get you...not great QA. In my experience a sufficiently strong QA person can both cost and be worth more than adding another dev to the team.