3 ms·

Frameworks like playright can record as code user actions and you can replay them in a test. So you can make your QA teams create plenty of tests if you give t

Frameworks like playright can record as code user actions and you can replay them in a test. So you can make your QA teams create plenty of tests if you give them the right tools.

chopin 3y ago
In my experience such tests are brittle as hell.
- Osmose 3y ago
  You're not wrong, but a good, well resourced QA org can both help write or develop more flexible tests, and also help fix brittle tests when they do break. The idea of brittle tests that break often being a blocker is predicated on practices like running every type of test on every commit that exist to deal with a lack of QA effort in the first place. Maybe recorded integration tests are run on every release instead of every commit? Maybe the QA team uses them less to pass/fail work and more to easily note which parts of the product have changed and need attention for the next release? There's lots of possibilities.
  - Kinrany 3y ago
    > Maybe recorded integration tests are run on every release instead of every commit? That would limit the frequency of releases.
kredd 3y ago
Absolutely hellish to maintain once you get into territories of different localization, A/B testing, engineering teams that constantly change small things in UI (for valid reasons). I think it might work on a product that changes once a year, but not sure about maintainability past that. Every time I worked with dedicated QA teams, we (core engineering) ended up leading the effort of writing maintainable automated tests. At some point, it was much easier to just do it ourselves.
- solardev 3y ago
  Well, that's kinda the point, isn't it? Have the tests run after each PR, and it flags all the changes for you. For the intentional ones, that means the tests have to be updated (often just changing a line or two). But it also catches all your unintentional changes that you didn't mean to break. This is especially important in complex UIs where everything from localization to state management to A/B testing can accidentally break things you didn't even mean to touch. Isn't it better for the devs to spend a few minutes looking at a broken test (which they can easily fix themselves) than for a user to discover it and have to go through triage months later, when it's out of everyone's memories already?
  - kredd 3y ago
    Realistically the build will fail, but since QAs used some action capturing thing, devs will have to either recapture the same flow or delegate it back to QA. Now you have the pipeline blocked, QAs most likely working on some other stuff so they need to reprioritize, devs are also annoyed as their build is not going through. I completely understand there are pros and cons to every set up, but if higher velocity is more important than complete correctness, such set up might be detrimental.
    - solardev 3y ago
      In our setup, we had the ability to either fix the test ourselves, comment out those tests as useless, or just force a build and bypass any hooks if we needed to force a new deploy and we were absolutely certain the tests aren't important. But at least they would be caught, even if they served as a soft warning rather than a forcible barrier. > but if higher velocity is more important than complete correctness, such set up might be detrimental. I guess it's a cultural value thing: move fast and break things vs polish before shipping. As a user I always prefer the latter, but hell, I don't run these multibillion dollar companies that are always shipping buggy things, lol
      - kredd 3y ago
        Totally agreed, it really depends on the case. I’ve been user of both types of products where I value extreme correctness (think of anything financial, or core software that I want to never crash like a browser), but also really don’t care if something like Spotify or Twitter doesn’t work 5% of the time I use it. Obviously that barrier is different for everyone, but that’s where the product development planning and prioritization comes into play.

Frameworks like playright can record as code user actions and you can replay them in a test. So you can make your QA teams create plenty of tests if you give them the right tools.

chopin 3y ago

In my experience such tests are brittle as hell.

Osmose 3y ago
You're not wrong, but a good, well resourced QA org can both help write or develop more flexible tests, and also help fix brittle tests when they do break. The idea of brittle tests that break often being a blocker is predicated on practices like running every type of test on every commit that exist to deal with a lack of QA effort in the first place. Maybe recorded integration tests are run on every release instead of every commit? Maybe the QA team uses them less to pass/fail work and more to easily note which parts of the product have changed and need attention for the next release? There's lots of possibilities.
- Kinrany 3y ago
  > Maybe recorded integration tests are run on every release instead of every commit? That would limit the frequency of releases.

kredd 3y ago

Absolutely hellish to maintain once you get into territories of different localization, A/B testing, engineering teams that constantly change small things in UI (for valid reasons). I think it might work on a product that changes once a year, but not sure about maintainability past that. Every time I worked with dedicated QA teams, we (core engineering) ended up leading the effort of writing maintainable automated tests. At some point, it was much easier to just do it ourselves.

solardev 3y ago
Well, that's kinda the point, isn't it? Have the tests run after each PR, and it flags all the changes for you. For the intentional ones, that means the tests have to be updated (often just changing a line or two). But it also catches all your unintentional changes that you didn't mean to break. This is especially important in complex UIs where everything from localization to state management to A/B testing can accidentally break things you didn't even mean to touch. Isn't it better for the devs to spend a few minutes looking at a broken test (which they can easily fix themselves) than for a user to discover it and have to go through triage months later, when it's out of everyone's memories already?
- kredd 3y ago
  Realistically the build will fail, but since QAs used some action capturing thing, devs will have to either recapture the same flow or delegate it back to QA. Now you have the pipeline blocked, QAs most likely working on some other stuff so they need to reprioritize, devs are also annoyed as their build is not going through. I completely understand there are pros and cons to every set up, but if higher velocity is more important than complete correctness, such set up might be detrimental.
  - solardev 3y ago
    In our setup, we had the ability to either fix the test ourselves, comment out those tests as useless, or just force a build and bypass any hooks if we needed to force a new deploy and we were absolutely certain the tests aren't important. But at least they would be caught, even if they served as a soft warning rather than a forcible barrier. > but if higher velocity is more important than complete correctness, such set up might be detrimental. I guess it's a cultural value thing: move fast and break things vs polish before shipping. As a user I always prefer the latter, but hell, I don't run these multibillion dollar companies that are always shipping buggy things, lol
    - kredd 3y ago
      Totally agreed, it really depends on the case. I’ve been user of both types of products where I value extreme correctness (think of anything financial, or core software that I want to never crash like a browser), but also really don’t care if something like Spotify or Twitter doesn’t work 5% of the time I use it. Obviously that barrier is different for everyone, but that’s where the product development planning and prioritization comes into play.