4 ms·

Sigh. Releasing more often is not an ideal situation. Yes, being able to release every single change on its own, fully tested, is a nice to have tool. But the r

Sigh. Releasing more often is not an ideal situation. Yes, being able to release every single change on its own, fully tested, is a nice to have tool. But the real world works in office days, weeks, and quarters. And whatever amount of work you put into your test suite and automation, sometimes you cannot test some things. Sometimes you have to wait for a real day (or God forbid a real quarter) to pass. Yes, testing, especially unit and integration tests, are a sine-qua-non for quality work. But so are automatic proofs of absence of certain errors (vulgo type checks) and non-functional tests. But all this has been at least partially resolved. What we are still sorely lacking is simulation. Everyone who ever worked with a staging environment knows how limited this usually is. What I would like to work with before I retire is a tool that lets me standup a complete copy of the production environment, isolated from the real world of course, with all external services automatically mocked. Now that would help to improve the quality.

smokel 3y ago
I've worked on quite a few, admittedly large, projects where we simulate nearly all external interfaces. It is mostly the human that is most expensive to replace with a simulator, apart from simple end-to-end or stress tests. Is anyone working on a reinforcement learning agent for proper monkey testing? :)
MoreQARespect 3y ago
>And whatever amount of work you put into your test suite and automation, sometimes you cannot test some things. Sometimes you have to wait for a real day (or God forbid a real quarter) to pass. I honestly can't imagine any software product with test automation that would require more than a real day to pass, never mind a quarter. Not unless some person or bureaucratic process was deliberately slowing it down. >But all this has been at least partially resolved. What we are still sorely lacking is simulation. Everyone who ever worked with a staging environment knows how limited this usually is. What I would like to work with before I retire is a tool that lets me standup a complete copy of the production environment, isolated from the real world of course, with all external services automatically mocked. Now that would help to improve the quality. There are plenty of tools that do this. I run simulators to catch sent mails, pretend to be browsers, mock REST APIs, etc. This is the bread and butter of test automation. The actual scenario writing I find to be the easiest part. The real problem is that this stuff is usually done really badly by cheap, poorly skilled people who will never earn as much as a dev and who almost never do this part of the job properly. The actual solution is in my username.
- rhdunn 3y ago
  Some bugs/issues appear over longer time spans. Things like issues around daylight savings time. Or memory leaks (or file size limits) that build up over a long time. Or even just accruing a lot of data in a system to the point where it starts exhibiting performance issues in certain workloads.
  - MoreQARespect 3y ago
    Ive created tests that mocked all of these scenarios before. They're all hard. They require engineering skill. They all fit into the category of things that a good dev should be able to do but "wont have time for" but a QA on 40-80% of their salary won't ever manage. The weird thing is that this really isnt a technical problem. It's an issue with social class.
    - Silhouette 3y ago
      It's not a skill issue when your application depends on some external API and the provider doesn't have a sandbox system where your integration tests can spin up a realistic "fake" API environment on demand. IME almost no APIs actually provide that level of sandbox support so that immediately means you can't run comprehensive, fully automated tests of those integrations. Of course you can mock those endpoints instead. But then you're not really testing the integration at all and conveniently ignoring the place where most of the real risk is often found.
      - MoreQARespect 3y ago
        I've faced had this exact scenario before. It was an SMS API which a major bank forced us to use. I set up a test against a mock version of the API request/response which I constructed manually from their crappy docs. It didn't work exactly right of course because the docs weren't exactly right. Ultimately all we could do was try to fit to the docs, manage expectations, release into production, see the errors, log as much as we could, try to cope with the fallout and update the mocks accordingly afterwards. For each problem/error we ran into, this took under a day. A longer test/release cycle would have made this problem far worse not better.
karmakaze 3y ago
This idyllic view sounds theoretical to me. What I've seen working best is a fast CI/CD pipeline with unit tests, key journey tests, and integration tests with blue/green deployments. This in my experience is far superior to staging environments for both startup and very large scales. In addition the development environment itself should be as close to prod as is practical so most checks can be done there.

smokel 3y ago

I've worked on quite a few, admittedly large, projects where we simulate nearly all external interfaces. It is mostly the human that is most expensive to replace with a simulator, apart from simple end-to-end or stress tests. Is anyone working on a reinforcement learning agent for proper monkey testing? :)

MoreQARespect 3y ago

>And whatever amount of work you put into your test suite and automation, sometimes you cannot test some things. Sometimes you have to wait for a real day (or God forbid a real quarter) to pass. I honestly can't imagine any software product with test automation that would require more than a real day to pass, never mind a quarter. Not unless some person or bureaucratic process was deliberately slowing it down. >But all this has been at least partially resolved. What we are still sorely lacking is simulation. Everyone who ever worked with a staging environment knows how limited this usually is. What I would like to work with before I retire is a tool that lets me standup a complete copy of the production environment, isolated from the real world of course, with all external services automatically mocked. Now that would help to improve the quality. There are plenty of tools that do this. I run simulators to catch sent mails, pretend to be browsers, mock REST APIs, etc. This is the bread and butter of test automation. The actual scenario writing I find to be the easiest part. The real problem is that this stuff is usually done really badly by cheap, poorly skilled people who will never earn as much as a dev and who almost never do this part of the job properly. The actual solution is in my username.

rhdunn 3y ago
Some bugs/issues appear over longer time spans. Things like issues around daylight savings time. Or memory leaks (or file size limits) that build up over a long time. Or even just accruing a lot of data in a system to the point where it starts exhibiting performance issues in certain workloads.
- MoreQARespect 3y ago
  Ive created tests that mocked all of these scenarios before. They're all hard. They require engineering skill. They all fit into the category of things that a good dev should be able to do but "wont have time for" but a QA on 40-80% of their salary won't ever manage. The weird thing is that this really isnt a technical problem. It's an issue with social class.
  - Silhouette 3y ago
    It's not a skill issue when your application depends on some external API and the provider doesn't have a sandbox system where your integration tests can spin up a realistic "fake" API environment on demand. IME almost no APIs actually provide that level of sandbox support so that immediately means you can't run comprehensive, fully automated tests of those integrations. Of course you can mock those endpoints instead. But then you're not really testing the integration at all and conveniently ignoring the place where most of the real risk is often found.
    - MoreQARespect 3y ago
      I've faced had this exact scenario before. It was an SMS API which a major bank forced us to use. I set up a test against a mock version of the API request/response which I constructed manually from their crappy docs. It didn't work exactly right of course because the docs weren't exactly right. Ultimately all we could do was try to fit to the docs, manage expectations, release into production, see the errors, log as much as we could, try to cope with the fallout and update the mocks accordingly afterwards. For each problem/error we ran into, this took under a day. A longer test/release cycle would have made this problem far worse not better.

karmakaze 3y ago

This idyllic view sounds theoretical to me. What I've seen working best is a fast CI/CD pipeline with unit tests, key journey tests, and integration tests with blue/green deployments. This in my experience is far superior to staging environments for both startup and very large scales. In addition the development environment itself should be as close to prod as is practical so most checks can be done there.