4 ms·

Yeah, this is the only thing that practically worked, in a complex product I used to work on. - If you ask the devs to do QA, you'll get no bugs other than the

Yeah, this is the only thing that practically worked, in a complex product I used to work on. - If you ask the devs to do QA, you'll get no bugs other than the ones they already caught during testing and deployment. - If you have a mostly independent QA team, they will find somewhat silly/trivial bugs like the login page not working in an extreme edge case scenario. - However, when you ask your Product team to own QA, you get the real good stuff - "Why does this feature not actually work well with this other feature when you combine the configuration" etc. It's great!

franciscassel 5y ago
When I was a PM, I was lucky to have a big, talented QA team, but I still knew I'd have to do a "smoke test" myself after every major feature release. I cared the most, and I knew the most about the intricacies of the product.
- tpmx 5y ago
  I really recognize that feeling. Also: Bliss is when you as a product owner get to such a place that you trust the QA team that you've worked with so closely so much that you only have have to some basic tests a few hours after the release.
travisjungroth 5y ago
On independent QA teams finding trivial bugs, I think this is a social problem. Specifically an alignment one. If a bug goes out, whose fault is it? Eng for making it or QA for not finding it? The answers are different at different orgs, and they have more to do with power dynamics than anything else. QA is pretty easy to fake (for a while). The last thing you want to do is quality check your QA team. So the further the incentives of the QA team are from the success of the product, the worse things get. It's a spectrum from the other cofounder doing QA to someone in another country charging by the hour. There are opposing forces to this. It's inherently difficult for people to check themselves. Also, QA is its own skill. It's one more thing to ask devs to be great at. Maybe it's worth having some specific experts around. It's kind of a specific mindset that not all devs have. This mindset issue is where I'm not entirely sold on what looks like a new strategy from Rainforest QA (where I used to work) of strongly targeting product teams. I haven't seen any QA tool so good that it removes the complete need for skill like indoor heating means you don't need to know anything about how to tend a fire. The best ones are more like how a modern gas range helps a chef. So I question how great the results will be if you have a PM or CSM doing it. Still, I wish them the best of luck.
- tpmx 5y ago
  We had a very large QA silo in a previous company. ~150 people all working under a former QA (non-developer) person turned into a "QA VP". It was a disaster in so many ways. Empire-building tendencies, large numbers of bad hires, etc. The solution was to break up this silo and getting these people into the product teams where more technical people could handle management and hiring.
- ukd1 5y ago
  Thanks for the luck Travis! Def something we've been edging around for a while. From what we've been seeing, a lot of PMs and no/low code folks already do this kind of thing manually, and don't have tooling for it. RF automation is now way better / easier to use than when you were with us (imho, of course).
- deleted 5y ago
  [deleted]
mschuster91 5y ago
> However, when you ask your Product team to own QA, you get the real good stuff - "Why does this feature not actually work well with this other feature when you combine the configuration" etc. It's great! That would require a skilled product management that is on equal footing with sales and has a veto right before stuff gets sold to customers. All too often however, the only thing that matters is customer wishes for new features, with no one taking care that the tugs the customers are pulling on don't rip the product apart.
- tpmx 5y ago
  Having previously spent a decade in that role for a $1B company: I never got a veto right before stuff got sold to customers. I really, really wanted that in the beginning. In the end what I ended up doing was spending a lot of time packaging the product as well as educating the sales force. If you don't have well-packaged products, the sales force will sell anything, even if it doesn't exist. And they'll get backed up by a typically slightly disconnected exec team, because they also want revenue really, really badly. This strategy worked surprisingly well. I think both the sales team and the ever slightly disconnected exec team benefited from the artificial structure that I spent so much effort making up. I guess people instinctively just like following clear structures over chaos. (This was 5-10 years ago, and not in Silicon Valley.)
sumedh 5y ago
> when you ask your Product team to own QA, you get the real good stuff A good QA who is part of the team can do that as well.
salawat 5y ago
>However, when you ask your Product team to own QA, you get the real good stuff - "Why does this feature not actually work well with this other feature when you combine the configuration" etc. It's great! This works right up until you hit a complexity point product can't handle, or worse, you find out that product is building something completely different from the requirements SME's are giving. Your QA group (or misguided product group) can do jack squat about badly translated reqs or the right questions not being asked, which in the presence of the best devs and QA's degenerates into building the wrong thing perfectly, which has to get redone over and over again. A good QA that's been around the block is usually pretty good at sussing out major communication defects, but it can be tough to pull out of. It's a surreal class of miscommunication to behold, and as close a death spiral as you can end up in, because everybody day in day out is working hard and getting frustrated but no one's getting closer to the end goal.
somerandomqaguy 5y ago
> - If you have a mostly independent QA team, they will find somewhat silly/trivial bugs like the login page not working in an extreme edge case scenario. Fair but you have to keep in mind that part of QA's job is to explore the silly and extreme edge cases too. One of the nastiest bugs I ever found started as a joke of a test. What happens if you enter a massive long text string (I think it was something like 600 kilobytes) into the password field and enter it? I expected the UI to barf and a bit of mischief. What I got was the entire server backend crashing and loosing every user session with it when it restarted. The joke test revealed a uncovered a serious denial of service attack vector that was trivial to execute, trivial to automate, difficult to detect, and incredibly effective. I agree it would be silly if QA insisted that trivial or edge cases be fixed; every bug has to be judged on the cost of fixing vs the risk of it happening in the field. But finding those silly edge case bugs is still very well worth the time to explore. If not because chasing the edge can reveal serious problems, but also because it also helps define where the limits of sanity really are instead of were it is imagined to be.

franciscassel 5y ago

When I was a PM, I was lucky to have a big, talented QA team, but I still knew I'd have to do a "smoke test" myself after every major feature release. I cared the most, and I knew the most about the intricacies of the product.

tpmx 5y ago
I really recognize that feeling. Also: Bliss is when you as a product owner get to such a place that you trust the QA team that you've worked with so closely so much that you only have have to some basic tests a few hours after the release.

travisjungroth 5y ago

On independent QA teams finding trivial bugs, I think this is a social problem. Specifically an alignment one. If a bug goes out, whose fault is it? Eng for making it or QA for not finding it? The answers are different at different orgs, and they have more to do with power dynamics than anything else. QA is pretty easy to fake (for a while). The last thing you want to do is quality check your QA team. So the further the incentives of the QA team are from the success of the product, the worse things get. It's a spectrum from the other cofounder doing QA to someone in another country charging by the hour. There are opposing forces to this. It's inherently difficult for people to check themselves. Also, QA is its own skill. It's one more thing to ask devs to be great at. Maybe it's worth having some specific experts around. It's kind of a specific mindset that not all devs have. This mindset issue is where I'm not entirely sold on what looks like a new strategy from Rainforest QA (where I used to work) of strongly targeting product teams. I haven't seen any QA tool so good that it removes the complete need for skill like indoor heating means you don't need to know anything about how to tend a fire. The best ones are more like how a modern gas range helps a chef. So I question how great the results will be if you have a PM or CSM doing it. Still, I wish them the best of luck.

tpmx 5y ago
We had a very large QA silo in a previous company. ~150 people all working under a former QA (non-developer) person turned into a "QA VP". It was a disaster in so many ways. Empire-building tendencies, large numbers of bad hires, etc. The solution was to break up this silo and getting these people into the product teams where more technical people could handle management and hiring.
ukd1 5y ago
Thanks for the luck Travis! Def something we've been edging around for a while. From what we've been seeing, a lot of PMs and no/low code folks already do this kind of thing manually, and don't have tooling for it. RF automation is now way better / easier to use than when you were with us (imho, of course).
deleted 5y ago
[deleted]

mschuster91 5y ago

> However, when you ask your Product team to own QA, you get the real good stuff - "Why does this feature not actually work well with this other feature when you combine the configuration" etc. It's great! That would require a skilled product management that is on equal footing with sales and has a veto right before stuff gets sold to customers. All too often however, the only thing that matters is customer wishes for new features, with no one taking care that the tugs the customers are pulling on don't rip the product apart.

tpmx 5y ago
Having previously spent a decade in that role for a $1B company: I never got a veto right before stuff got sold to customers. I really, really wanted that in the beginning. In the end what I ended up doing was spending a lot of time packaging the product as well as educating the sales force. If you don't have well-packaged products, the sales force will sell anything, even if it doesn't exist. And they'll get backed up by a typically slightly disconnected exec team, because they also want revenue really, really badly. This strategy worked surprisingly well. I think both the sales team and the ever slightly disconnected exec team benefited from the artificial structure that I spent so much effort making up. I guess people instinctively just like following clear structures over chaos. (This was 5-10 years ago, and not in Silicon Valley.)

sumedh 5y ago

> when you ask your Product team to own QA, you get the real good stuff A good QA who is part of the team can do that as well.

salawat 5y ago

>However, when you ask your Product team to own QA, you get the real good stuff - "Why does this feature not actually work well with this other feature when you combine the configuration" etc. It's great! This works right up until you hit a complexity point product can't handle, or worse, you find out that product is building something completely different from the requirements SME's are giving. Your QA group (or misguided product group) can do jack squat about badly translated reqs or the right questions not being asked, which in the presence of the best devs and QA's degenerates into building the wrong thing perfectly, which has to get redone over and over again. A good QA that's been around the block is usually pretty good at sussing out major communication defects, but it can be tough to pull out of. It's a surreal class of miscommunication to behold, and as close a death spiral as you can end up in, because everybody day in day out is working hard and getting frustrated but no one's getting closer to the end goal.

somerandomqaguy 5y ago

> - If you have a mostly independent QA team, they will find somewhat silly/trivial bugs like the login page not working in an extreme edge case scenario. Fair but you have to keep in mind that part of QA's job is to explore the silly and extreme edge cases too. One of the nastiest bugs I ever found started as a joke of a test. What happens if you enter a massive long text string (I think it was something like 600 kilobytes) into the password field and enter it? I expected the UI to barf and a bit of mischief. What I got was the entire server backend crashing and loosing every user session with it when it restarted. The joke test revealed a uncovered a serious denial of service attack vector that was trivial to execute, trivial to automate, difficult to detect, and incredibly effective. I agree it would be silly if QA insisted that trivial or edge cases be fixed; every bug has to be judged on the cost of fixing vs the risk of it happening in the field. But finding those silly edge case bugs is still very well worth the time to explore. If not because chasing the edge can reveal serious problems, but also because it also helps define where the limits of sanity really are instead of were it is imagined to be.