6 ms·

To point out the obvious: generally API providers don’t particularly want you to pararelize your request (they even implement rate limiting to make it harder on

To point out the obvious: generally API providers don’t particularly want you to pararelize your request (they even implement rate limiting to make it harder on purpose). If they wanted to make it easy to get all the results, they would allow you to access the data without pagination - just download all the data in one go.

sb8244 6y ago
> If they wanted to make it easy to get all the results Speaking from experience...we want to make it easy but also want to keep it performant. Getting the data all in one go is generally not performant and is easy to abuse as an API consumer. For example, always asking for all of the data rather than maintaining a cursor and secondary index (which is so much more performant for everyone involved).
- alexchamberlain 6y ago
  We provide (internal) access to data where we provide interactive access via GraphQL-based APIs and bulk access via CSV or RDF dumps - I feel like dump files are grossly undervalued these days.
  - sb8244 6y ago
    I agree. I am going to reflect on this and see if there's a way to support dump files long term in our app. We sorta support it today but it's ad hoc implementation since an export can range from a few hundred of a thing to tens of millions of a thing. Is there any good literature or patterns on supporting dumps in the tens of millions or larger? I wrote a sheets plug-in that uses our cursor API to provide a full dump into a spreadsheet. Our professional services team is in love with it, so I bet they'd love generic data export capability.
    - jerf 6y ago
      "Is there any good literature or patterns on supporting dumps in the tens of millions or larger?" The two main things you need are: 1. HTTP is a streaming protocol. You don't need to fully manifest a response in memory before you send it. If your framework forces that, bypass it for this particular call. (If you can't bypass it... your framework choice is now a problem for you.) 2. You presumably have some sort of JSON encoder in your language. As long as it doesn't have some sort of hard-coded "close the stream once we send this JSON" behavior (and if so, file a bug because a JSON encoder has no business doing that), all you have to do is ensure that the right bytes go out on the wire. You, again, don't have to fully manifest the reply in memory before you encode it. Something like: stream.Write("[") needComma = False for item in toBeSerialized: if needComma: stream.Write(",") json.Write(stream, item) needComma = True stream.Write("]") A lot of times when you're emitting gigabytes of JSON it's still in lots of little chunks where each individual chunk isn't a memory problem on its own, so doing something like this can be very memory-efficient, especially if "toBeSerialized" is itself something like a cursor coming out of a DB where it itself is not manifesting in memory. (Newlines are also a good idea if your JSON encoder isn't naturally doing it already; helps debugging a lot for very little cost.) JSON objects can be more annoying; you may need to manually deserialize one and that's more annoying. Protip: Whenever possible, use the JSON encoder in your language; there is no shame or anything in using the JSON encoder to emit strings corresponding to the keys of your object. Much like HTML, you need to be very careful writing things directly to the stream; it really always should go through the encoder. I even send constant strings through the encoder just to make the code look right. The last little tidbit is that the HTTP software stack will tend to fight you on the matter of keeping long-lived connections open. There can be a lot of places that have timeouts you may want to extend. If this gets too big you may need to do something other than HTTP. You may also need to consider detecting failures (hashing or something) and the ability to restart. (Although don't underestimate modern bandwidth and the speed you can iterate through SELECT -type queries; definitely check into the virtues of just retrying. 10GB/year of extra bandwidth and processing power is still cheaper than a developer even designing* a solution to that problem, let alone implementing and testing it.) Oh, and if you can use HTTP, be sure you're gzip'ing. It's dirt cheap nowadays on the CPU; only in extreme situations of bandwidth abundance and CPU shortage can it be worth skipping. My rule-of-thumb on JSON shrinking is about 15:1. CSVs don't quite shrink that much but they still shrink down pretty well.
- tshaddox 6y ago
  That’s the point. Running multiple paginated queries in parallel is essentially circumventing the API provider’s intent to limit the number of items requested at one time.
  - eyelidlessness 6y ago
    If your API doesn’t support a single client making ~6 concurrent requests, let me tell you about browsers since the last millennium.
    - tshaddox 6y ago
      It’s not about what the API provider can physically support, it’s just about being a polite client. They probably can support 6 concurrent requests without it causing a problem. The point is that if they use offset or page number pagination with a maximum limit of 100, that’s probably a clue that you shouldn’t preemptively grab the first 100 pages in parallel.
      - gkop 6y ago
        It’s not hard to imagine scenarios where an API provider might reasonably prefer n concurrent requests of m batch size over a single request of n * m batch size.
        eyelidlessness 6y ago
        I can more than imagine it, I design for it. Concurrent all the things. The vast majority of GET workloads are either embarrassingly parallel or so badly designed that they fall over with even the gentlest touch (which I’m sad to say I’ve encountered; services falling over from sequential requests!). Edit to add: this isn’t some wild idea, recent network and web standards have made increasing concurrency both automatic (HTTP/2 and beyond) and tunable (various prefetch APIs).
      - eyelidlessness 6y ago
        Right, that’s my point. Don’t be overzealous obviously. But some concurrency should be expected (and in my opinion designed for and encouraged). There’s a huge gulf between grabbing 100 pages and grabbing the number of resources a browser grabs by default.
eyelidlessness 6y ago
A certain level of parallelism is generally within the realm of good API citizenship. Even naive rate limiting schemes tend to permit a certain number of concurrent requests (as they well should, since even browsers may perform concurrent requests without any developer intervention). Rate limiting and pagination aren’t (necessarily) about making full data consumption more difficult. They’re more often about optimizing common use cases and general quality of service. Edit to add: in certain circles (eg those of us who take REST and HATEOAS as baseline HTTP API principles), parallelism is often not just expected but often encouraged. A service can provide efficient, limited subsets of a full representation and allow clients to retrieve as little or as much of the full representation as they see fit.
- corty 6y ago
  One thing that frequently bugs me is APIs limiting number of items per page for reasons of efficiency. I can perfectly understand low limits for other reasons, like not helping people scrape your data. But limiting for efficiency is usually done in a way that I would call a cargo cult: First, the number of items per "page" is usually a number one would pick per displayed page, in the range of 10 to 20. This is inefficient for the general case, the amount of data transmitted is usually just the same size as the request plus response headers. So if the API isn't strictly for display purposes, pick a number of items per page that gives a useful balance between not transmitting too much useless data, but keeping query and response overhead low. Paginate in chunks of 100kB or more. In terms of computation and backend load, pagination can be as expensive for a 1-page-query as for a full query. Usually this occurs when the query doesn't directly hit an index or similar data structure where a full sweep over all the data cannot be avoided. So think and benchmark before you paginate, and maybe add an index here and there.
  - cunac 6y ago
    my intent with pagination is always to prevent problems with open ended size of data set. 1000 at once is usually not a problem but 10x , 100x etc.. is a big problem for transferring over the wire.
  - jerf 6y ago

sb8244 6y ago

> If they wanted to make it easy to get all the results Speaking from experience...we want to make it easy but also want to keep it performant. Getting the data all in one go is generally not performant and is easy to abuse as an API consumer. For example, always asking for all of the data rather than maintaining a cursor and secondary index (which is so much more performant for everyone involved).

alexchamberlain 6y ago
We provide (internal) access to data where we provide interactive access via GraphQL-based APIs and bulk access via CSV or RDF dumps - I feel like dump files are grossly undervalued these days.
- sb8244 6y ago
  I agree. I am going to reflect on this and see if there's a way to support dump files long term in our app. We sorta support it today but it's ad hoc implementation since an export can range from a few hundred of a thing to tens of millions of a thing. Is there any good literature or patterns on supporting dumps in the tens of millions or larger? I wrote a sheets plug-in that uses our cursor API to provide a full dump into a spreadsheet. Our professional services team is in love with it, so I bet they'd love generic data export capability.
  - jerf 6y ago
    "Is there any good literature or patterns on supporting dumps in the tens of millions or larger?" The two main things you need are: 1. HTTP is a streaming protocol. You don't need to fully manifest a response in memory before you send it. If your framework forces that, bypass it for this particular call. (If you can't bypass it... your framework choice is now a problem for you.) 2. You presumably have some sort of JSON encoder in your language. As long as it doesn't have some sort of hard-coded "close the stream once we send this JSON" behavior (and if so, file a bug because a JSON encoder has no business doing that), all you have to do is ensure that the right bytes go out on the wire. You, again, don't have to fully manifest the reply in memory before you encode it. Something like: stream.Write("[") needComma = False for item in toBeSerialized: if needComma: stream.Write(",") json.Write(stream, item) needComma = True stream.Write("]") A lot of times when you're emitting gigabytes of JSON it's still in lots of little chunks where each individual chunk isn't a memory problem on its own, so doing something like this can be very memory-efficient, especially if "toBeSerialized" is itself something like a cursor coming out of a DB where it itself is not manifesting in memory. (Newlines are also a good idea if your JSON encoder isn't naturally doing it already; helps debugging a lot for very little cost.) JSON objects can be more annoying; you may need to manually deserialize one and that's more annoying. Protip: Whenever possible, use the JSON encoder in your language; there is no shame or anything in using the JSON encoder to emit strings corresponding to the keys of your object. Much like HTML, you need to be very careful writing things directly to the stream; it really always should go through the encoder. I even send constant strings through the encoder just to make the code look right. The last little tidbit is that the HTTP software stack will tend to fight you on the matter of keeping long-lived connections open. There can be a lot of places that have timeouts you may want to extend. If this gets too big you may need to do something other than HTTP. You may also need to consider detecting failures (hashing or something) and the ability to restart. (Although don't underestimate modern bandwidth and the speed you can iterate through SELECT -type queries; definitely check into the virtues of just retrying. 10GB/year of extra bandwidth and processing power is still cheaper than a developer even designing* a solution to that problem, let alone implementing and testing it.) Oh, and if you can use HTTP, be sure you're gzip'ing. It's dirt cheap nowadays on the CPU; only in extreme situations of bandwidth abundance and CPU shortage can it be worth skipping. My rule-of-thumb on JSON shrinking is about 15:1. CSVs don't quite shrink that much but they still shrink down pretty well.
tshaddox 6y ago
That’s the point. Running multiple paginated queries in parallel is essentially circumventing the API provider’s intent to limit the number of items requested at one time.
- eyelidlessness 6y ago
  If your API doesn’t support a single client making ~6 concurrent requests, let me tell you about browsers since the last millennium.
  - tshaddox 6y ago
    It’s not about what the API provider can physically support, it’s just about being a polite client. They probably can support 6 concurrent requests without it causing a problem. The point is that if they use offset or page number pagination with a maximum limit of 100, that’s probably a clue that you shouldn’t preemptively grab the first 100 pages in parallel.
    - gkop 6y ago
      It’s not hard to imagine scenarios where an API provider might reasonably prefer n concurrent requests of m batch size over a single request of n * m batch size.
      - eyelidlessness 6y ago
        I can more than imagine it, I design for it. Concurrent all the things. The vast majority of GET workloads are either embarrassingly parallel or so badly designed that they fall over with even the gentlest touch (which I’m sad to say I’ve encountered; services falling over from sequential requests!). Edit to add: this isn’t some wild idea, recent network and web standards have made increasing concurrency both automatic (HTTP/2 and beyond) and tunable (various prefetch APIs).
    - eyelidlessness 6y ago
      Right, that’s my point. Don’t be overzealous obviously. But some concurrency should be expected (and in my opinion designed for and encouraged). There’s a huge gulf between grabbing 100 pages and grabbing the number of resources a browser grabs by default.

eyelidlessness 6y ago

A certain level of parallelism is generally within the realm of good API citizenship. Even naive rate limiting schemes tend to permit a certain number of concurrent requests (as they well should, since even browsers may perform concurrent requests without any developer intervention). Rate limiting and pagination aren’t (necessarily) about making full data consumption more difficult. They’re more often about optimizing common use cases and general quality of service. Edit to add: in certain circles (eg those of us who take REST and HATEOAS as baseline HTTP API principles), parallelism is often not just expected but often encouraged. A service can provide efficient, limited subsets of a full representation and allow clients to retrieve as little or as much of the full representation as they see fit.

corty 6y ago
One thing that frequently bugs me is APIs limiting number of items per page for reasons of efficiency. I can perfectly understand low limits for other reasons, like not helping people scrape your data. But limiting for efficiency is usually done in a way that I would call a cargo cult: First, the number of items per "page" is usually a number one would pick per displayed page, in the range of 10 to 20. This is inefficient for the general case, the amount of data transmitted is usually just the same size as the request plus response headers. So if the API isn't strictly for display purposes, pick a number of items per page that gives a useful balance between not transmitting too much useless data, but keeping query and response overhead low. Paginate in chunks of 100kB or more. In terms of computation and backend load, pagination can be as expensive for a 1-page-query as for a full query. Usually this occurs when the query doesn't directly hit an index or similar data structure where a full sweep over all the data cannot be avoided. So think and benchmark before you paginate, and maybe add an index here and there.
- cunac 6y ago
  my intent with pagination is always to prevent problems with open ended size of data set. 1000 at once is usually not a problem but 10x , 100x etc.. is a big problem for transferring over the wire.
- jerf 6y ago