4 ms·

I’m a bit confused as to why a “unit test” would ever be hitting a real, external API. I would call that an integration test, E2E test, etc. that probably shoul

by cmckn 5y ago

I’m a bit confused as to why a “unit test” would ever be hitting a real, external API. I would call that an integration test, E2E test, etc. that probably shouldn’t be run frequently (as part of a normal build, for example). This API sounds like garbage, don’t get me wrong; and because of that, mocking it’s behavior is going to be almost impossible.

holstvoogd 5y ago
This. Unit Test should not connect to external service imo. And I do not get the gripe with a production rate limit tbh.
- bestcoder69 5y ago
  Same. I’d be more worried about an API with no rate limit. Say if another customer ships a bug that DoS’s the API on accident.
  - clukic 5y ago
    I would think a user token level limit would prevent this, although I could imagine a case where a bug simultaneous affected all user tokens, but I'd imagine you'd set that app level limit pretty high because otherwise you'd be making life very difficult for legitimate use cases.
- clukic 5y ago
  Rate limits are of course fine. I think token level limits make more sense, because application level limits force the consumer to either to track a rate window across asynchronous processes or make the calls synchronously. But, I mean, that's fine too. Think it's just like OAuth1a, fine in itself but add enough of these things on top of one another, and you've created a technical hurdle that's just too difficult to leap.
clukic 5y ago
I don't think I realized this was a thing. Are other developers mocking API endpoints, to test the code their as they develop and maintain it? It seems like a tremendous amount of work - recreating a service that will make an async call back to your endpoint as a means of testing. I've thought about it, but the fact that it seemed like so much additional work and that I'd be mocking a moving target just made me thing that would be a visit crazy town.
- tpxl 5y ago
  We did this at my previous company. What our app did was get data from a queue, enrich it with the API, then send it off to another queue. They provided test data and a test environment, but it was horrible, changed unpredictably and data frequently went missing. We recorded all endpoints hit during our integration tests, saved them to files and mocked their API (mostly GETs with no state so backed by a simple rest service with a hashmap). The API team was an internal team to the company and there was absolutely no communication possible with them, so we kinda had no choice.
- simonw 5y ago
  I avoid writing automated tests that hit an external API - especially one out of my control - because I don't want my CI runs to ever fail because someone else's service wasn't responding. I want CI to be a completely closed box, such that any failures mean there's a bug in my code. If I want to test external APIs I'll do that in a separate set of integration tests which are run as part of a separate system, not as part of my CI for every code commit to my repo. I mostly use Python, and the APIs I talk to are mostly accessed via the requests or httpx libraries - both of which have excellent libraries for productive mocking: - https://requests-mock.readthedocs.io/en/latest/pytest.html https://requests-mock.readthedocs.io/en/latest/pytest.html - https://github.com/Colin-b/pytest_httpx https://github.com/Colin-b/pytest_httpx
- cmckn 5y ago
  It totally depends on the API and how critical of a code path you're writing tests for. For simple GET/POST calls, it's usually pretty easy to do this and I think you get a lot of value out of it. I've used an in-process HTTP server to mock such calls for a unit test, or you can add a layer to the nginx docker image to have a composable mock of the service.
haroldl 5y ago
One reason for tests to hit a real external API is if you're using a "record and replay" test framework to capture the interactions so that you can run the tests against the recorded data quickly later. But because the API calls you make change (and the external implementation changes) you need to re-record from time to time. This strikes a balance where 99% of the time you are making calls that never leave the process for fast testing, but can validate against the real implementation as needed.
ryansouza 5y ago
The unit under testing could be the API interaction. If I wrap an external API in a client library I would test it’s behavior before then mocking that out when testing other code that uses the library. I think situations like external API hit a fuzzy line between unit/integration. For testing an API unit I would reach for some way to save, re-play, and re-record those interactions just for sanity sake. To me an integration test would be multi-step behavior, not just testing a specific request does a specific thing but a chain of requests, or verifying side-effects.
ramchip 5y ago
I've seen quite a few people that just call all automated tests "unit tests". I wouldn't read too much into it.

by cmckn 5y ago

holstvoogd 5y ago

This. Unit Test should not connect to external service imo. And I do not get the gripe with a production rate limit tbh.

bestcoder69 5y ago
Same. I’d be more worried about an API with no rate limit. Say if another customer ships a bug that DoS’s the API on accident.
- clukic 5y ago
  I would think a user token level limit would prevent this, although I could imagine a case where a bug simultaneous affected all user tokens, but I'd imagine you'd set that app level limit pretty high because otherwise you'd be making life very difficult for legitimate use cases.
clukic 5y ago
Rate limits are of course fine. I think token level limits make more sense, because application level limits force the consumer to either to track a rate window across asynchronous processes or make the calls synchronously. But, I mean, that's fine too. Think it's just like OAuth1a, fine in itself but add enough of these things on top of one another, and you've created a technical hurdle that's just too difficult to leap.

clukic 5y ago

I don't think I realized this was a thing. Are other developers mocking API endpoints, to test the code their as they develop and maintain it? It seems like a tremendous amount of work - recreating a service that will make an async call back to your endpoint as a means of testing. I've thought about it, but the fact that it seemed like so much additional work and that I'd be mocking a moving target just made me thing that would be a visit crazy town.

tpxl 5y ago
We did this at my previous company. What our app did was get data from a queue, enrich it with the API, then send it off to another queue. They provided test data and a test environment, but it was horrible, changed unpredictably and data frequently went missing. We recorded all endpoints hit during our integration tests, saved them to files and mocked their API (mostly GETs with no state so backed by a simple rest service with a hashmap). The API team was an internal team to the company and there was absolutely no communication possible with them, so we kinda had no choice.
simonw 5y ago
I avoid writing automated tests that hit an external API - especially one out of my control - because I don't want my CI runs to ever fail because someone else's service wasn't responding. I want CI to be a completely closed box, such that any failures mean there's a bug in my code. If I want to test external APIs I'll do that in a separate set of integration tests which are run as part of a separate system, not as part of my CI for every code commit to my repo. I mostly use Python, and the APIs I talk to are mostly accessed via the requests or httpx libraries - both of which have excellent libraries for productive mocking: - https://requests-mock.readthedocs.io/en/latest/pytest.html https://requests-mock.readthedocs.io/en/latest/pytest.html - https://github.com/Colin-b/pytest_httpx https://github.com/Colin-b/pytest_httpx
cmckn 5y ago
It totally depends on the API and how critical of a code path you're writing tests for. For simple GET/POST calls, it's usually pretty easy to do this and I think you get a lot of value out of it. I've used an in-process HTTP server to mock such calls for a unit test, or you can add a layer to the nginx docker image to have a composable mock of the service.

haroldl 5y ago

One reason for tests to hit a real external API is if you're using a "record and replay" test framework to capture the interactions so that you can run the tests against the recorded data quickly later. But because the API calls you make change (and the external implementation changes) you need to re-record from time to time. This strikes a balance where 99% of the time you are making calls that never leave the process for fast testing, but can validate against the real implementation as needed.

ryansouza 5y ago

The unit under testing could be the API interaction. If I wrap an external API in a client library I would test it’s behavior before then mocking that out when testing other code that uses the library. I think situations like external API hit a fuzzy line between unit/integration. For testing an API unit I would reach for some way to save, re-play, and re-record those interactions just for sanity sake. To me an integration test would be multi-step behavior, not just testing a specific request does a specific thing but a chain of requests, or verifying side-effects.

ramchip 5y ago

I've seen quite a few people that just call all automated tests "unit tests". I wouldn't read too much into it.