4 ms·

Ive created tests that mocked all of these scenarios before. They're all hard. They require engineering skill. They all fit into the category of things that a g

Ive created tests that mocked all of these scenarios before. They're all hard. They require engineering skill. They all fit into the category of things that a good dev should be able to do but "wont have time for" but a QA on 40-80% of their salary won't ever manage. The weird thing is that this really isnt a technical problem. It's an issue with social class.

Silhouette 3y ago
It's not a skill issue when your application depends on some external API and the provider doesn't have a sandbox system where your integration tests can spin up a realistic "fake" API environment on demand. IME almost no APIs actually provide that level of sandbox support so that immediately means you can't run comprehensive, fully automated tests of those integrations. Of course you can mock those endpoints instead. But then you're not really testing the integration at all and conveniently ignoring the place where most of the real risk is often found.
- MoreQARespect 3y ago
  I've faced had this exact scenario before. It was an SMS API which a major bank forced us to use. I set up a test against a mock version of the API request/response which I constructed manually from their crappy docs. It didn't work exactly right of course because the docs weren't exactly right. Ultimately all we could do was try to fit to the docs, manage expectations, release into production, see the errors, log as much as we could, try to cope with the fallout and update the mocks accordingly afterwards. For each problem/error we ran into, this took under a day. A longer test/release cycle would have made this problem far worse not better.
  - Silhouette 3y ago
    It didn't work exactly right of course because the docs weren't exactly right. Indeed. That approach also won't pick up changes to the remote API that break your integration. An alternative strategy in this situation if you have access to some kind of remote sandbox - but maybe just a single stateful one that must be shared by your whole org - is to factor out your integration with that service into its own component/service and deploy controlled updates to that service less frequently and after running through a comprehensive integration test using the real sandbox. This probably involves a much lower release frequency but it also keeps your integration as stable as possible for your internal clients and tested against the real external service. If you're talking about remote services that do things you can't trivially revert - like physically transporting some object or causing money to change hands - then the strategy you described of trying to manage expectations while testing in production is probably not an acceptable alternative anyway.
twic 3y ago
To be able to test them, you need to be able enumerate them. Part of the problem here is that you don't always know what regularly occurring external thing is going to cause the problem.

Silhouette 3y ago

It's not a skill issue when your application depends on some external API and the provider doesn't have a sandbox system where your integration tests can spin up a realistic "fake" API environment on demand. IME almost no APIs actually provide that level of sandbox support so that immediately means you can't run comprehensive, fully automated tests of those integrations. Of course you can mock those endpoints instead. But then you're not really testing the integration at all and conveniently ignoring the place where most of the real risk is often found.

MoreQARespect 3y ago
I've faced had this exact scenario before. It was an SMS API which a major bank forced us to use. I set up a test against a mock version of the API request/response which I constructed manually from their crappy docs. It didn't work exactly right of course because the docs weren't exactly right. Ultimately all we could do was try to fit to the docs, manage expectations, release into production, see the errors, log as much as we could, try to cope with the fallout and update the mocks accordingly afterwards. For each problem/error we ran into, this took under a day. A longer test/release cycle would have made this problem far worse not better.
- Silhouette 3y ago
  It didn't work exactly right of course because the docs weren't exactly right. Indeed. That approach also won't pick up changes to the remote API that break your integration. An alternative strategy in this situation if you have access to some kind of remote sandbox - but maybe just a single stateful one that must be shared by your whole org - is to factor out your integration with that service into its own component/service and deploy controlled updates to that service less frequently and after running through a comprehensive integration test using the real sandbox. This probably involves a much lower release frequency but it also keeps your integration as stable as possible for your internal clients and tested against the real external service. If you're talking about remote services that do things you can't trivially revert - like physically transporting some object or causing money to change hands - then the strategy you described of trying to manage expectations while testing in production is probably not an acceptable alternative anyway.

twic 3y ago

To be able to test them, you need to be able enumerate them. Part of the problem here is that you don't always know what regularly occurring external thing is going to cause the problem.