3 ms·

[author here] You are absolutely right: the optimal setup is a cross-functional team of domain experts collaborating on solving the customer's problem. The arti

[author here] You are absolutely right: the optimal setup is a cross-functional team of domain experts collaborating on solving the customer's problem. The article is meant to point out that siloing QA to either just dev or just QA is an anti-pattern that is unstable and will not work, long term, without significant pain. I think the challenge is that, while we agree on the optimal setup, it's almost never done like this. There's lots of reasons why, but the main one (imo) is organizational scaling. The logic of scaling teams tends towards specialization, especially because developer time is extremely expensive. There are many ways to address the problem of QA. From our perspective, if they don't broaden the ownership of QA from just developers or just QA engineers (which is where all the current products are targeted), they will exacerbate this specialization problem.

Guthur 5y ago
I think that to say developers should not be interest or responsible for quality is most certainly a cop out, there can and should be a level of pride of workmanship that is focused on the customer. The main challenge with software is the possibility of unexpected regression in areas that are non-obvious when implementing a given feature. One of the more scalable ways of catching these regressions is via some sort of automated testing. I think that a QA support team can provide the tools and support that allow for teams to easily add end to end testing, which is vitally important when build large distributed software. If the QA team is responsible for testing every feature from many teams there is definite risk of bus factor and finger pointing. In the super ideal world the testing is specified via something like Cucumber and the orginal author could be the product owner with feedback from the engineer as he works to that specification. But I have never seen this though have heard it can happen, rarely.
- fredsters_s 5y ago
  > to say developers should not be interest or responsible for quality is most certainly a cop out Read the piece - it says no such thing. It's talking about 'ownership', which is distinct from participation and most certainly from interest.
newmnhn 5y ago
Hi author, I found parts of this downright wrong, especially on the Cypress side, which is where I have the most familiarity: >Compare this to code-based automation solutions like Selenium or Cypress, where a coder would need to manually update the code across all the relevant tests any time the product changes. ... Well, no. Those tools can have reusable chunks and references that work just like your steps and are organized in the same manner, so that things used multiple places are stored once and reused. That's a basic programming good practice. When a change is needed to, say, a login flow, you don't change 100 UI tests, you change the login function that those tests are all calling. > Tests created in code-based automation solutions like Selenium and Cypress evaluate what’s going on in the code behind the scenes of your product’s UI. That is, these solutions test what the computer sees, not what your users see. and > For example, if the HTML ID attribute of the signup button in the example above changes, a Selenium or Cypress coded test that identifies that button by its ID would likely break. This is all very disingenuous in my book. In these tools the tests interact with the rendered web page and _can_ be made to use IDs, classes, and other brittle stuff as selectors, but that's a naive approach that a professional wouldn't typically use, because the brittleness surfaces early and there are better patterns that avoid it. One of those is to use selectors based on something that itself should be present - eg, click a button based on its label being "Log In". If that test fails because the button label was changed or removed, that's a reasonable failure, because labels are important in the UI. In the case where we are selecting/making assertions about DOM elements that don't have meaningful user-facing labels, dedicated test-id attributes solve the brittleness problem. But by and large, if something is interactive to the user, it should have a label, so the fallback to test attrs for selectors is rare, and used for the case where we want to measure some specific effect took place that is not itself an interactive element. It's unfair to suggest that these tools inherently create brittle tests. > Rainforest QA tests don’t break when there are minor, behind-the-scenes code changes that don’t change the UI. but > It works by pixel matching and using AI to find and interact with (click, fill, select, etc.) the correct elements. Look, maybe your tool is GREAT, I haven't tried it. But a functional UI test based on label contents may be more robust than something that breaks when the visual structure of a page changes, which happens itself pretty frequently. Maybe the AI is good enough to know that a bunch of stuff moved around but the actual flow is the same and can find its way through. With tests based on labels and dedicated selectors, devs can make massive structural changes to the DOM and not break any tests if all the workflows are the same. I would sure love if this post contained more info on how the visual AI system of selecting elements is better than using labels, test attributes etc as is conventional in automated testing done by professionals. Likewise it would be good to compare with an actual competitor like Ghost Inspector, where non-programmers have been able to generate step-based tests like this for years. The main gripe I had with Ghost Inspector was that it creates brittle selectors and so tests need to be changed a lot in response to DOM changes, unless a developer gets in there and manually reviews/picks better selectors. If what you have a is a tool that _makes more robust tests than Ghost Inspector_ but is _as easy for a PM to use_, then that is interesting. I actually support this underlying idea completely - I love systems where non-developers can create and maintain tests that reflect what they care about in the UI. I even love it when tools like this create low-quality tests that are still expressive enough that a dev can quickly come in and fix up selectors to make them more suitable. Cypress Studio is a currently-experimental feature that looks promising for this too, allowing folks to edit and create tests by clicking in the UI, not editing code files. It's a good direction for automated test frameworks to explore. I'm just really uncomfortable after reading this post. It strays beyond typical marketing hyperbole to be actually deceptive about the nature of these other tools and the practices of teams using them. Instead of highlighting the actual uniqueness of your product, you exaggerate the benefits of this approach vs other solution. Come on, you can do tests in parallel with many tools, and you can avoid captcha for automated testing in various ways. Some of the process points you make are fair but also, again, kind of weak, because sure "QA at the end with no consultation during planning and development" is bad, but that's inherently bad for well known reasons. What you've said is basically "our tool is better than every other idea about QA, if those ideas are deliberately implemented in the worst possible way, and our tool is implemented in the best way". Well sure. But also, of course it is. Sorry if this seems harsh. To give you the benefit of the doubt: Marketing copy walks a fine line when trying to be persuasive and you might not have expected to create this impression. It's also possible that you did some research into weaknesses of automated test frameworks and just don't understand that those things are compensated for quite easily and routinely, because maybe you don't have the background. I don't know, but I hope future materials are a little more grounded in reality.

Guthur 5y ago

I think that to say developers should not be interest or responsible for quality is most certainly a cop out, there can and should be a level of pride of workmanship that is focused on the customer. The main challenge with software is the possibility of unexpected regression in areas that are non-obvious when implementing a given feature. One of the more scalable ways of catching these regressions is via some sort of automated testing. I think that a QA support team can provide the tools and support that allow for teams to easily add end to end testing, which is vitally important when build large distributed software. If the QA team is responsible for testing every feature from many teams there is definite risk of bus factor and finger pointing. In the super ideal world the testing is specified via something like Cucumber and the orginal author could be the product owner with feedback from the engineer as he works to that specification. But I have never seen this though have heard it can happen, rarely.

fredsters_s 5y ago
> to say developers should not be interest or responsible for quality is most certainly a cop out Read the piece - it says no such thing. It's talking about 'ownership', which is distinct from participation and most certainly from interest.

newmnhn 5y ago

Hi author, I found parts of this downright wrong, especially on the Cypress side, which is where I have the most familiarity: >Compare this to code-based automation solutions like Selenium or Cypress, where a coder would need to manually update the code across all the relevant tests any time the product changes. ... Well, no. Those tools can have reusable chunks and references that work just like your steps and are organized in the same manner, so that things used multiple places are stored once and reused. That's a basic programming good practice. When a change is needed to, say, a login flow, you don't change 100 UI tests, you change the login function that those tests are all calling. > Tests created in code-based automation solutions like Selenium and Cypress evaluate what’s going on in the code behind the scenes of your product’s UI. That is, these solutions test what the computer sees, not what your users see. and > For example, if the HTML ID attribute of the signup button in the example above changes, a Selenium or Cypress coded test that identifies that button by its ID would likely break. This is all very disingenuous in my book. In these tools the tests interact with the rendered web page and _can_ be made to use IDs, classes, and other brittle stuff as selectors, but that's a naive approach that a professional wouldn't typically use, because the brittleness surfaces early and there are better patterns that avoid it. One of those is to use selectors based on something that itself should be present - eg, click a button based on its label being "Log In". If that test fails because the button label was changed or removed, that's a reasonable failure, because labels are important in the UI. In the case where we are selecting/making assertions about DOM elements that don't have meaningful user-facing labels, dedicated test-id attributes solve the brittleness problem. But by and large, if something is interactive to the user, it should have a label, so the fallback to test attrs for selectors is rare, and used for the case where we want to measure some specific effect took place that is not itself an interactive element. It's unfair to suggest that these tools inherently create brittle tests. > Rainforest QA tests don’t break when there are minor, behind-the-scenes code changes that don’t change the UI. but > It works by pixel matching and using AI to find and interact with (click, fill, select, etc.) the correct elements. Look, maybe your tool is GREAT, I haven't tried it. But a functional UI test based on label contents may be more robust than something that breaks when the visual structure of a page changes, which happens itself pretty frequently. Maybe the AI is good enough to know that a bunch of stuff moved around but the actual flow is the same and can find its way through. With tests based on labels and dedicated selectors, devs can make massive structural changes to the DOM and not break any tests if all the workflows are the same. I would sure love if this post contained more info on how the visual AI system of selecting elements is better than using labels, test attributes etc as is conventional in automated testing done by professionals. Likewise it would be good to compare with an actual competitor like Ghost Inspector, where non-programmers have been able to generate step-based tests like this for years. The main gripe I had with Ghost Inspector was that it creates brittle selectors and so tests need to be changed a lot in response to DOM changes, unless a developer gets in there and manually reviews/picks better selectors. If what you have a is a tool that _makes more robust tests than Ghost Inspector_ but is _as easy for a PM to use_, then that is interesting. I actually support this underlying idea completely - I love systems where non-developers can create and maintain tests that reflect what they care about in the UI. I even love it when tools like this create low-quality tests that are still expressive enough that a dev can quickly come in and fix up selectors to make them more suitable. Cypress Studio is a currently-experimental feature that looks promising for this too, allowing folks to edit and create tests by clicking in the UI, not editing code files. It's a good direction for automated test frameworks to explore. I'm just really uncomfortable after reading this post. It strays beyond typical marketing hyperbole to be actually deceptive about the nature of these other tools and the practices of teams using them. Instead of highlighting the actual uniqueness of your product, you exaggerate the benefits of this approach vs other solution. Come on, you can do tests in parallel with many tools, and you can avoid captcha for automated testing in various ways. Some of the process points you make are fair but also, again, kind of weak, because sure "QA at the end with no consultation during planning and development" is bad, but that's inherently bad for well known reasons. What you've said is basically "our tool is better than every other idea about QA, if those ideas are deliberately implemented in the worst possible way, and our tool is implemented in the best way". Well sure. But also, of course it is. Sorry if this seems harsh. To give you the benefit of the doubt: Marketing copy walks a fine line when trying to be persuasive and you might not have expected to create this impression. It's also possible that you did some research into weaknesses of automated test frameworks and just don't understand that those things are compensated for quite easily and routinely, because maybe you don't have the background. I don't know, but I hope future materials are a little more grounded in reality.