There are two things that can be tested here, not one: whether the upstream service conforms to the contract / API promise, and whether your code behaves correc

There are two things that can be tested here, not one: whether the upstream service conforms to the contract / API promise, and whether your code behaves correctly with respect to what the API promises. So that gives you a number of options for testing the second one of those. Recording sample traffic and replaying it in the test suite is one approach. Actually running an instance of the service (if it's open-source - there's still value in paying someone to competently run an OSS service) in your test suite is another, as is running some clone of the service (e.g., if you're talking to S3, there are probably a hundred S3 API-compatible clones that are good enough to run in your test suite, even if, again, you are happy to pay Amazon to competently run production). You also want to pay attention to the first one of those, but that's not a job for your test suite. That's the job for some balance between their test suite, your monitoring or production logging, and your business relationship with them.