4 ms·
> If you're writing a custom frontend for GitHub using the GitHub API and don't bother writing a decent set of mocks for how you expect the GitHub API to behave
by whack 2y ago
> If you're writing a custom frontend for GitHub using the GitHub API and don't bother writing a decent set of mocks for how you expect the GitHub API to behave, your app will quickly require either full manual QA at best or become untestable at worst. Some APIs are very stable, and testing against the API itself can hit rate limiting, bans, and other anti-abuse mechanisms that introduce all kinds of instability to your test suite.
I've been doing E2E testing using 3rd-party APIs for a decade now, and this has yet to be a significant problem. The majority of my APIs had a dedicated sandbox environment to avoid "rate limiting, bans, and other anti-abuse mechanisms". The remainder were simple enough that the provider didn't care about users exploring on the live API, and were usually read-only as well.
Did I run into the occasional flaky failure, or API stability issues? Sure. But it was very rare and easy to workaround. It never devolved into becoming "untestable" or "full manual QA"
My other teams that relied on mocks suffered from far worse problems - a ton of time being spent on manual-QA, and bugs that leaked into production, because of mock-reality mismatches.
- chii 2y ago> because of mock-reality mismatches. you also need to test the mocks against the real thing separately.
- switchbak 2y agoI would prefer that we all try to use this language consistently: https://www.martinfowler.com/articles/mocksArentStubs.html https://www.martinfowler.com/articles/mocksArentStubs.html What you're describing sounds like a fake to me.
- drewcoo 2y agoCan't tell from the context that's here. It's important to have a contract testing layer in place to make sure your test doubles are still behaving like the real thing, though.
- ffsm8 2y agoContract testing is still a thing? I thought interested mostly fizzled out over the years after the initial hype died down
- Too 2y agoGood luck with that. Any org I’ve seen can’t even consistently agree what is a unit test, component test, black box test vs integration test.
- simonw 2y agoOutside of the payments industry I haven't encountered many sandbox APIs that don't have rate-limits, what are some good ones you've seen of those?
- jph00 2y agoI have a custom frontend for GitHub using the GitHub API (https://github.com/fastai/ghapi/ https://github.com/fastai/ghapi/) and don't use mocks - I test using the real API. I've had very occasional, but not enough to ever cause any real issues. I don't find mocks for this kind of thing very helpful, because what you're really testing for are things like changes to how an API changes over time -- you need real API calls to see this.
- hot_gril 2y agoYeah, even if there's no sandbox mode, a separate sandbox account will usually do. Sometimes this catches misuse that would've caused rate-limiting in prod. And if a service makes this hard, maybe you shouldn't use it in prod either.
- ljm 2y agoThere are plenty of libraries out there, like VCR, that can set up a test and then save the response for future test runs. You don't really have to renew them that often either. That was always the go-to for me when testing against 3rd party services, especially because the tests would then survive the offboarding of the engineer who set them up with their personal credentials. If your test suite relies on live Github PATs or user-specific OAuth access tokens, then you can either figure out how to manage some kind of service account with a 'bot' user, or live with things breaking every time someone leaves the org. Services that incur a per-request charge, or consume account credits, are another problem. Especially if they don't have sandboxes.
- KingOfCoders 2y agoSame here, I think Customer.IO had the nices sandbox experience yet.