6 ms·

Asking developers to own QA is broken because developers are naturally biased towards the happy path. If you want to build a bar, you need someone to order -1 b

Asking developers to own QA is broken because developers are naturally biased towards the happy path. If you want to build a bar, you need someone to order -1 beers[1]. Handing off QA to an external team is broken because those people don't have the necessary experience with the product, nor can they quickly and easily engage with development to get to the heart of a problem (and a fix.) Having QA rely exclusively on automation brings quality down as the application's complexity goes up. Writing tests to cover every possible edge case before the product ships isn't feasible. The best solution I've seen in the decades I've been in software development has been to have QA and Dev as part of a team. Automation covers as much of the product as possible, and continually grows as issues are identified that can be covered by CI from here on out. QA become essential at the boundary of new features, and require a deep understanding of the product in order to know what knobs to turn that might break the application in ways the developer never thought about. Once those sharp edges are also automated, QA pushes the boundary forward. [1]: https://twitter.com/brenankeller/status/1068615953989087232 https://twitter.com/brenankeller/status/1068615953989087232

notmyfuture 5y ago
I've also arrived at this approach and don't think it's that uncommon - IMO the article is presenting a false dichotomy. There are still gotchas to look out for in the team-embedded QA approach. In typical team sizes, you often end up with only one QA per team - you need to make sure they have cover (everyone needs a break), and they need support in their discipline (do something to share QA knowledge across teams).
- vlovich123 5y ago
  QA also works best for features that the user can meaningfully interact with. The more esoteric, interconnected, or complex the failure is, the more it gets muddled with other reports and weird conditions (it breaks on 3pm when the moon is in equinox and I have my mouse pointed north). That’s an over exaggeration but the sentiment is accurate. QA will frequently lack a deeper understanding of how the software works and is connected. That ignorance can be valuable (finds blind spots) but has trade offs (assumptions are made based on imperfect observation of cause and effect where human biases lead you down dark hallways). Speaking from experience with lots of manual QA which is actually a rarity these days. The other thing to consider is, if you’re careful, user feedback can be obtained more quickly. If you can keep things stable, then you won’t upset your users too badly and you’ll avoid the need for QA to act as a stand in.
  - notmyfuture 5y ago
    I agree, this all makes sense. Although I think the team-embedded QA is generally the right thing, I wouldn't use it blindly in all cases. Some teams I manage only produce HTTP API's, these are ideal candidates for automated testing (incl. end-to-end integrated tests) and the developers are happy to own this without a QA on the team.
  - chx 5y ago
    > (it breaks on 3pm when the moon is in equinox and I have my mouse pointed north). That’s an over exaggeration it's not. OpenOffice won't print on Tuesdays https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161/comments/28 https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161... etc
- Agentlien 5y ago
  What size team do you consider typical? Most projects I've worked on have been games where the studio was split into teams of about 6-10 people with one dedicated in-house QA member for some but not all teams, a few in-house QA workers not assigned to a specific team, plus a much larger external QA team (from the parent company/ publisher). That has worked great. I've also been on projects with only external QA. It works, but the lack of internal QA has been a frequent annoyance.
  - notmyfuture 5y ago
    Yeah - in my experience 6-10 people per team is typical, often with one of them being dedicated QA. Having this, plus a separate central QA team is one way to address the pitfalls of embedded QA I was pointing out - they can cover for time away for team QA, or act as bench capacity.
    - ethbr0 5y ago
      IMHO, the reason embedded QA works better is because QA is fundamentally a negative task. In that you are telling people they made a mistake. That's difficult to hear, repeatedly, from the same person. And I've seen a lot of developers not react well to the underlying discomfort. When it's an external team, at arm's length, that relationship can get pretty bad. At least when it's someone on your team, then you have non-QA moments and interactions to help balance out the unfortunate bug truth. And the end result is faster, more thorough, more productive bidirectional communication between QA and coders.
- varenc 5y ago
  Agreed it's presenting a false dichotomy. The article ends with a pitch for Rainforest's "no-code" QA platform so it makes sense they want to present their product as the clear solution. I could take the article more seriously if it at least mentioned a more integrated approach.
- cwkoss 5y ago
  Absolutely. Disappointing to see this sort of shallow sales pitch blogspam making it to the frontpage. I'm surprised more HN readers don't see through this. The entire purpose of this article is to self-servingly attempt to convince the reader that their product is the only solution to QA problems. It correctly identifies some challenges with QA, but this solution is certainly not the only way to have effective QA. That Rainforest is resorting to such a disingenuous presentation of solutions to QA issues makes me think they probably don't actually solve QA problems very well. "We have researched what makes QA successful and X, Y, and Z are what we found. Here's how we believe we're solving Z for our customers" would be a much more honest pitch.
  - fredsters_s 5y ago
    I totally see your point here - it's frustrating that how I presented it undermined the impact of the core point, which is that siloed ownership of quality is an anti-pattern, and an anti-pattern which is often a response to the limitations and design principles of quality tooling. We are to my knowledge the only product built for this kind of cross-functional ownership, that's why I don't recommend other products. While I agree the post is too marketing-y - to be honest I didn't expect it to appeal so much to the HN audience - it was written in good faith. We built the product because after 7 years of building a product for one of those silos, we became convinced that the only long-term durable solution was to empower everyone to own quality. Clearly I need to figure out how to strike the balance that you outline in your last sentence, which was the goal!
  - Shacklz 5y ago
    Oh c'mon. Of course a blog on the website of a company is ultimately trying to pitch their product, one way or another. That does not invalidate the points they bring up per se. It obviously needs a bit of critical thinking when reading it and taking everything with a grain of salt, but that's something I really hope we can expect people on HN to be capable of?
foolfoolz 5y ago
i don’t agree with this at all. i’ve seen qa automation team members work solely as a resourcing issue: that if you have X developers you always have X feature developers and never enough people to automate. it’s not that devs aren’t capabale. i fundamentally disagree with rainforests mission that you need “other people” to achieve success in testing. your top engineers have no problem succeeding here
fredsters_s 5y ago
[author here] You are absolutely right: the optimal setup is a cross-functional team of domain experts collaborating on solving the customer's problem. The article is meant to point out that siloing QA to either just dev or just QA is an anti-pattern that is unstable and will not work, long term, without significant pain. I think the challenge is that, while we agree on the optimal setup, it's almost never done like this. There's lots of reasons why, but the main one (imo) is organizational scaling. The logic of scaling teams tends towards specialization, especially because developer time is extremely expensive. There are many ways to address the problem of QA. From our perspective, if they don't broaden the ownership of QA from just developers or just QA engineers (which is where all the current products are targeted), they will exacerbate this specialization problem.
- Guthur 5y ago
  I think that to say developers should not be interest or responsible for quality is most certainly a cop out, there can and should be a level of pride of workmanship that is focused on the customer. The main challenge with software is the possibility of unexpected regression in areas that are non-obvious when implementing a given feature. One of the more scalable ways of catching these regressions is via some sort of automated testing. I think that a QA support team can provide the tools and support that allow for teams to easily add end to end testing, which is vitally important when build large distributed software. If the QA team is responsible for testing every feature from many teams there is definite risk of bus factor and finger pointing. In the super ideal world the testing is specified via something like Cucumber and the orginal author could be the product owner with feedback from the engineer as he works to that specification. But I have never seen this though have heard it can happen, rarely.
  - fredsters_s 5y ago
    > to say developers should not be interest or responsible for quality is most certainly a cop out Read the piece - it says no such thing. It's talking about 'ownership', which is distinct from participation and most certainly from interest.
- 5y ago
developeron29 5y ago
Agreed 100%. Communication is key. QA and Devs should always be part of the same team and have no communication gaps
accountofme 5y ago
I would agree with this. In a past job doing UI work, I had a tester sitting across from me. As soon as I got things working reasonably, I would give her a build to test. It was great, because we found a lot of bugs super early, very shortly after the code was written...
ozim 5y ago
Problem comes from BigCo approach to people. Such corporations would like to have a QA team that should be utilized to a 100%. So that if they don't have much work on project X you just drop them into project Y. In reality we all know that is just not possible, because people have to have domain knowledge and keep up with changes. It is not like if you come back to project after 2 months you will be proficient with it as the world changed. This way if project X has slow times, dedicated QA will have less to do and BigCo has to eat that "loss". In small startup it is easier to have QA incorporated into dev/product team because as software is constantly changed there is work to be done all the time.
tjpnz 5y ago
>Asking developers to own QA is broken because developers are naturally biased towards the happy path. I've been arguing the same thing for a while now. There's a clear mismatch in incentives. If you hand off the QA to developers you might initially get less bug reports but that doesn't mean your software isn't garbage. Some of the best QA engineers I've worked with were developers in a previous life but they've since spent years cultivating a very different way of thinking about software. Believing you can get your engineers to adopt that mindset overnight is pure fantasy.
jjav 5y ago
For the first ~15 years of my career, multiple teams, the model was always that engineering was responsible for all the test coverage of all areas. It was wonderful. A separate dedicated QA team will never, almost no matter how good they are, have the same deep insight into the code both big picture and small picture. So they won't be able to exercise it via tests as well as the engineers who wrote it. I always thought, this is the only way that makes sense. To my surprise in subsequent roles, I've seen many development teams, even good ones, who don't have that deep instinct to break their code via tests. Which was very alien to me. I've seen enough cases now to realize it's somewhat common. For such teams, separate QA engineers are a must. I still feel having quality responsibility in the engineering team ultimately can offer the best quality outcome, but only if you can build a team with the right attitude about testing. If not, having a separate QA team is vital.
wildmanx 5y ago
> Asking developers to own QA is broken because developers are naturally biased towards the happy path This overlooks that "no customer calls with product issues that dev needs to get fixed asap on a Saturday evening" is part of the happy path to optimize for. If those things happen, the dev has deviated from the happy path, including the embarrassment in front of fellow devs "but that's really a case we should have tested". This is another of those details the article is lacking.

notmyfuture 5y ago

I've also arrived at this approach and don't think it's that uncommon - IMO the article is presenting a false dichotomy. There are still gotchas to look out for in the team-embedded QA approach. In typical team sizes, you often end up with only one QA per team - you need to make sure they have cover (everyone needs a break), and they need support in their discipline (do something to share QA knowledge across teams).

vlovich123 5y ago
QA also works best for features that the user can meaningfully interact with. The more esoteric, interconnected, or complex the failure is, the more it gets muddled with other reports and weird conditions (it breaks on 3pm when the moon is in equinox and I have my mouse pointed north). That’s an over exaggeration but the sentiment is accurate. QA will frequently lack a deeper understanding of how the software works and is connected. That ignorance can be valuable (finds blind spots) but has trade offs (assumptions are made based on imperfect observation of cause and effect where human biases lead you down dark hallways). Speaking from experience with lots of manual QA which is actually a rarity these days. The other thing to consider is, if you’re careful, user feedback can be obtained more quickly. If you can keep things stable, then you won’t upset your users too badly and you’ll avoid the need for QA to act as a stand in.
- notmyfuture 5y ago
  I agree, this all makes sense. Although I think the team-embedded QA is generally the right thing, I wouldn't use it blindly in all cases. Some teams I manage only produce HTTP API's, these are ideal candidates for automated testing (incl. end-to-end integrated tests) and the developers are happy to own this without a QA on the team.
- chx 5y ago
  > (it breaks on 3pm when the moon is in equinox and I have my mouse pointed north). That’s an over exaggeration it's not. OpenOffice won't print on Tuesdays https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161/comments/28 https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161... etc
Agentlien 5y ago
What size team do you consider typical? Most projects I've worked on have been games where the studio was split into teams of about 6-10 people with one dedicated in-house QA member for some but not all teams, a few in-house QA workers not assigned to a specific team, plus a much larger external QA team (from the parent company/ publisher). That has worked great. I've also been on projects with only external QA. It works, but the lack of internal QA has been a frequent annoyance.
- notmyfuture 5y ago
  Yeah - in my experience 6-10 people per team is typical, often with one of them being dedicated QA. Having this, plus a separate central QA team is one way to address the pitfalls of embedded QA I was pointing out - they can cover for time away for team QA, or act as bench capacity.
  - ethbr0 5y ago
    IMHO, the reason embedded QA works better is because QA is fundamentally a negative task. In that you are telling people they made a mistake. That's difficult to hear, repeatedly, from the same person. And I've seen a lot of developers not react well to the underlying discomfort. When it's an external team, at arm's length, that relationship can get pretty bad. At least when it's someone on your team, then you have non-QA moments and interactions to help balance out the unfortunate bug truth. And the end result is faster, more thorough, more productive bidirectional communication between QA and coders.
varenc 5y ago
Agreed it's presenting a false dichotomy. The article ends with a pitch for Rainforest's "no-code" QA platform so it makes sense they want to present their product as the clear solution. I could take the article more seriously if it at least mentioned a more integrated approach.
cwkoss 5y ago
Absolutely. Disappointing to see this sort of shallow sales pitch blogspam making it to the frontpage. I'm surprised more HN readers don't see through this. The entire purpose of this article is to self-servingly attempt to convince the reader that their product is the only solution to QA problems. It correctly identifies some challenges with QA, but this solution is certainly not the only way to have effective QA. That Rainforest is resorting to such a disingenuous presentation of solutions to QA issues makes me think they probably don't actually solve QA problems very well. "We have researched what makes QA successful and X, Y, and Z are what we found. Here's how we believe we're solving Z for our customers" would be a much more honest pitch.
- fredsters_s 5y ago
  I totally see your point here - it's frustrating that how I presented it undermined the impact of the core point, which is that siloed ownership of quality is an anti-pattern, and an anti-pattern which is often a response to the limitations and design principles of quality tooling. We are to my knowledge the only product built for this kind of cross-functional ownership, that's why I don't recommend other products. While I agree the post is too marketing-y - to be honest I didn't expect it to appeal so much to the HN audience - it was written in good faith. We built the product because after 7 years of building a product for one of those silos, we became convinced that the only long-term durable solution was to empower everyone to own quality. Clearly I need to figure out how to strike the balance that you outline in your last sentence, which was the goal!
- Shacklz 5y ago
  Oh c'mon. Of course a blog on the website of a company is ultimately trying to pitch their product, one way or another. That does not invalidate the points they bring up per se. It obviously needs a bit of critical thinking when reading it and taking everything with a grain of salt, but that's something I really hope we can expect people on HN to be capable of?

foolfoolz 5y ago

i don’t agree with this at all. i’ve seen qa automation team members work solely as a resourcing issue: that if you have X developers you always have X feature developers and never enough people to automate. it’s not that devs aren’t capabale. i fundamentally disagree with rainforests mission that you need “other people” to achieve success in testing. your top engineers have no problem succeeding here

fredsters_s 5y ago

[author here] You are absolutely right: the optimal setup is a cross-functional team of domain experts collaborating on solving the customer's problem. The article is meant to point out that siloing QA to either just dev or just QA is an anti-pattern that is unstable and will not work, long term, without significant pain. I think the challenge is that, while we agree on the optimal setup, it's almost never done like this. There's lots of reasons why, but the main one (imo) is organizational scaling. The logic of scaling teams tends towards specialization, especially because developer time is extremely expensive. There are many ways to address the problem of QA. From our perspective, if they don't broaden the ownership of QA from just developers or just QA engineers (which is where all the current products are targeted), they will exacerbate this specialization problem.

Guthur 5y ago
I think that to say developers should not be interest or responsible for quality is most certainly a cop out, there can and should be a level of pride of workmanship that is focused on the customer. The main challenge with software is the possibility of unexpected regression in areas that are non-obvious when implementing a given feature. One of the more scalable ways of catching these regressions is via some sort of automated testing. I think that a QA support team can provide the tools and support that allow for teams to easily add end to end testing, which is vitally important when build large distributed software. If the QA team is responsible for testing every feature from many teams there is definite risk of bus factor and finger pointing. In the super ideal world the testing is specified via something like Cucumber and the orginal author could be the product owner with feedback from the engineer as he works to that specification. But I have never seen this though have heard it can happen, rarely.
- fredsters_s 5y ago
  > to say developers should not be interest or responsible for quality is most certainly a cop out Read the piece - it says no such thing. It's talking about 'ownership', which is distinct from participation and most certainly from interest.
5y ago

developeron29 5y ago

Agreed 100%. Communication is key. QA and Devs should always be part of the same team and have no communication gaps

accountofme 5y ago

I would agree with this. In a past job doing UI work, I had a tester sitting across from me. As soon as I got things working reasonably, I would give her a build to test. It was great, because we found a lot of bugs super early, very shortly after the code was written...

ozim 5y ago

Problem comes from BigCo approach to people. Such corporations would like to have a QA team that should be utilized to a 100%. So that if they don't have much work on project X you just drop them into project Y. In reality we all know that is just not possible, because people have to have domain knowledge and keep up with changes. It is not like if you come back to project after 2 months you will be proficient with it as the world changed. This way if project X has slow times, dedicated QA will have less to do and BigCo has to eat that "loss". In small startup it is easier to have QA incorporated into dev/product team because as software is constantly changed there is work to be done all the time.

tjpnz 5y ago

>Asking developers to own QA is broken because developers are naturally biased towards the happy path. I've been arguing the same thing for a while now. There's a clear mismatch in incentives. If you hand off the QA to developers you might initially get less bug reports but that doesn't mean your software isn't garbage. Some of the best QA engineers I've worked with were developers in a previous life but they've since spent years cultivating a very different way of thinking about software. Believing you can get your engineers to adopt that mindset overnight is pure fantasy.

jjav 5y ago

For the first ~15 years of my career, multiple teams, the model was always that engineering was responsible for all the test coverage of all areas. It was wonderful. A separate dedicated QA team will never, almost no matter how good they are, have the same deep insight into the code both big picture and small picture. So they won't be able to exercise it via tests as well as the engineers who wrote it. I always thought, this is the only way that makes sense. To my surprise in subsequent roles, I've seen many development teams, even good ones, who don't have that deep instinct to break their code via tests. Which was very alien to me. I've seen enough cases now to realize it's somewhat common. For such teams, separate QA engineers are a must. I still feel having quality responsibility in the engineering team ultimately can offer the best quality outcome, but only if you can build a team with the right attitude about testing. If not, having a separate QA team is vital.

wildmanx 5y ago

> Asking developers to own QA is broken because developers are naturally biased towards the happy path This overlooks that "no customer calls with product issues that dev needs to get fixed asap on a Saturday evening" is part of the happy path to optimize for. If those things happen, the dev has deviated from the happy path, including the embarrassment in front of fellow devs "but that's really a case we should have tested". This is another of those details the article is lacking.