4 ms·

Absolutely hellish to maintain once you get into territories of different localization, A/B testing, engineering teams that constantly change small things in UI

by kredd 3y ago

Absolutely hellish to maintain once you get into territories of different localization, A/B testing, engineering teams that constantly change small things in UI (for valid reasons). I think it might work on a product that changes once a year, but not sure about maintainability past that. Every time I worked with dedicated QA teams, we (core engineering) ended up leading the effort of writing maintainable automated tests. At some point, it was much easier to just do it ourselves.

solardev 3y ago
Well, that's kinda the point, isn't it? Have the tests run after each PR, and it flags all the changes for you. For the intentional ones, that means the tests have to be updated (often just changing a line or two). But it also catches all your unintentional changes that you didn't mean to break. This is especially important in complex UIs where everything from localization to state management to A/B testing can accidentally break things you didn't even mean to touch. Isn't it better for the devs to spend a few minutes looking at a broken test (which they can easily fix themselves) than for a user to discover it and have to go through triage months later, when it's out of everyone's memories already?
- kredd 3y ago
  Realistically the build will fail, but since QAs used some action capturing thing, devs will have to either recapture the same flow or delegate it back to QA. Now you have the pipeline blocked, QAs most likely working on some other stuff so they need to reprioritize, devs are also annoyed as their build is not going through. I completely understand there are pros and cons to every set up, but if higher velocity is more important than complete correctness, such set up might be detrimental.
  - solardev 3y ago
    In our setup, we had the ability to either fix the test ourselves, comment out those tests as useless, or just force a build and bypass any hooks if we needed to force a new deploy and we were absolutely certain the tests aren't important. But at least they would be caught, even if they served as a soft warning rather than a forcible barrier. > but if higher velocity is more important than complete correctness, such set up might be detrimental. I guess it's a cultural value thing: move fast and break things vs polish before shipping. As a user I always prefer the latter, but hell, I don't run these multibillion dollar companies that are always shipping buggy things, lol
    - kredd 3y ago
      Totally agreed, it really depends on the case. I’ve been user of both types of products where I value extreme correctness (think of anything financial, or core software that I want to never crash like a browser), but also really don’t care if something like Spotify or Twitter doesn’t work 5% of the time I use it. Obviously that barrier is different for everyone, but that’s where the product development planning and prioritization comes into play.

by kredd 3y ago

solardev 3y ago

Well, that's kinda the point, isn't it? Have the tests run after each PR, and it flags all the changes for you. For the intentional ones, that means the tests have to be updated (often just changing a line or two). But it also catches all your unintentional changes that you didn't mean to break. This is especially important in complex UIs where everything from localization to state management to A/B testing can accidentally break things you didn't even mean to touch. Isn't it better for the devs to spend a few minutes looking at a broken test (which they can easily fix themselves) than for a user to discover it and have to go through triage months later, when it's out of everyone's memories already?

kredd 3y ago
Realistically the build will fail, but since QAs used some action capturing thing, devs will have to either recapture the same flow or delegate it back to QA. Now you have the pipeline blocked, QAs most likely working on some other stuff so they need to reprioritize, devs are also annoyed as their build is not going through. I completely understand there are pros and cons to every set up, but if higher velocity is more important than complete correctness, such set up might be detrimental.
- solardev 3y ago
  In our setup, we had the ability to either fix the test ourselves, comment out those tests as useless, or just force a build and bypass any hooks if we needed to force a new deploy and we were absolutely certain the tests aren't important. But at least they would be caught, even if they served as a soft warning rather than a forcible barrier. > but if higher velocity is more important than complete correctness, such set up might be detrimental. I guess it's a cultural value thing: move fast and break things vs polish before shipping. As a user I always prefer the latter, but hell, I don't run these multibillion dollar companies that are always shipping buggy things, lol
  - kredd 3y ago
    Totally agreed, it really depends on the case. I’ve been user of both types of products where I value extreme correctness (think of anything financial, or core software that I want to never crash like a browser), but also really don’t care if something like Spotify or Twitter doesn’t work 5% of the time I use it. Obviously that barrier is different for everyone, but that’s where the product development planning and prioritization comes into play.