5 ms·
Notes on streaming large API responses
- aliasEli 5y agoI don't think you use a large response when there are other alternatives. One problem is see is with tests. You can expect that there are some problems that might arise with very large data sets, but it may take some computer time to set-up and break-down such tests.
- simonw 5y agoI've found writing tests against these endpoints to be pretty easy - I populate the associated database table with one record more than the default page size and then run a test that retrieves the full result and checks that the expected records are present. No need to run tests that generate 100MB responses every time - though I will do a few manual tests of large responses to check that RAM usage is as-expected. Here's my test in Datasette for the streaming CSV response - I already have a table with 1001 items in it setup for another test: https://github.com/simonw/datasette/blob/0.57.1/tests/test_csv.py#L155-L161 https://github.com/simonw/datasette/blob/0.57.1/tests/test_c...
- snicker7 5y agoI used to work at a financial data provider. Our clients were institutional investors (e.g. banks, mutual funds). We would FTP our data directly to our client's machine (or SFTP for our more savvy clients).
- debo_ 5y agoIndeed, FTP makes the fintech world go round. Banks do a lot of things in batch (e.g. clearing, settlement) and FTP matches that nicely. At least in North America, this is starting to be disrupted a bit as faster payment instruments and transfer options arise.
- etaioinshrdlu 5y agoThis is fun, just recently I had to implement a large data export feature. So I looked into generating zip file streams with this package: https://github.com/arjan-s/python-zipstream https://github.com/arjan-s/python-zipstream You can generate a zip stream and then stream it over http.
- klysm 5y agoIs that better than just using GZIP though?
- etaioinshrdlu 5y agoYou can send the user multiple files on one download this way.
- simonw 5y agoThat's really cool, I hadn't seen that library.
- rzzzt 5y agoIn a multipart message, a missing (final) boundary delimiter is a clear indication of partial content. Returning one might might work for CSV files.
- collaborative 5y agoThis is how I do it: store the large data files on GCS. Then, offer them via an F1 Node.js endpoint but make the process 'smart' and offer them in 10MB chunks. The client sees a chunk progress (3/10 etc) and a chunk % progress (0 - 100 for each chunk). The client also sequentially persists the large text responses as they arrive This has saved both my very limited F1 server from sad OOM exceptions and my iOS vm'd client from crashing while processing too much text I do the same for uploads
- foota 5y agoDo you just read the response linearly? What made you choose GCS instead of something like memcache With some kind of chunking?
- collaborative 5y agoSo the great thing about using an F1 instance is that it completely falls under the free tier (28hrs/day) There is no memcache in node.js F1 but what I do is I use the GCS container itself as stream memory instead of the usual node.js memory multer (yes, this is possible!) This only adds a few ms to the read op which is entirely fine
- arthurcolle 5y ago28 hrs/ day...?
- collaborative 5y ago1 instance running all day = 24 hrs Times as many instances as you have. So 28 hrs = 1 instance running all day + an extra 4 hours of a second instance It auto scales
- arthurcolle 5y agoThanks, thought I had slipped into an alternative dimension. Appreciate you helping me take the edge off.
- bickeringyokel 5y ago> If you’re streaming a response, you start with an HTTP 200 code... but then what happens if an error occurs half Maybe a 202 Accepted code makes more sense
- masklinn 5y agoIt does not. 202 means the request has been accepted and will be processed somewhere somehow. Here the response is being returned immediately, just streamed.
- thibauts 5y ago200 looks perfectly valid. Any HTTP response can be interrupted at any time by any error down the stack (app crashing, server, VM, network etc). 200 does not a guarantee complete delivery.
- 153957 5y agoFor a project where I worked on we offered streaming TSV responses and always finished the response with a last row containing a comment: `# Finished downloading.`. Since we also managed one of the client packages to access the endpoints, we check for that when the download is finished and raise an error if it is not there.
- biermic 5y agoI followed the exporting to S3 approach. Unfortunately it takes a while until those files become available, even though the AWS API says they are. We had many issues with this approach in the past. Better host the response yourself....
- robjan 5y agoAs of just over a year ago they now provide strong read after write consistency which should solve this problem.
- biermic 5y agoJust ran into the problem again last week. Even though we verify the files availability using the boto3 client...
- loevborg 5y agoOnce the initial S3 PUT request succeeds, the object is guaranteed to be available immediately AFAIK.
- cle 5y agoAs long as you haven’t tried to GET the object before the PUT succeeded. Otherwise there can be some eventual consistency for the negative cache to be overwritten. This can be quite annoying if you have e.g. a separate process polling for an S3 object. The object can appear in a HEAD and then return a 404 on the subsequent GET.
- loevborg 5y agoYou're right, I encountered this issue as well a few years ago. However, this has changed recently. S3 now offers strong consistency https://aws.amazon.com/s3/consistency/ https://aws.amazon.com/s3/consistency/
- throwaway858 5y agoAs mentioned in the sibling comment, this has been fixed by AWS a few months ago and now all S3 operations are always strongly consistent (not eventually consistent)
- foreigner 5y agoI tried this once. The problem I ran in to was that you really only see the benefit of the streaming if all of your components do it properly, including the database, server, client, and application logic that consumes the data. Streaming is really easy to screw up and hard to debug if it's not working. For example at one point my problem was the client's JSON parser. I had carefully chosen a streaming parser, but it turned out that the streaming had to be explicitly enabled - by default it slurped all the data.
- k__ 5y agoWouldn't it help with slow clients? If I can send a bunch of small chunks to the client and the first few can already be processed, it doesn't matter that I had to load all of the chunks at once from my database.
- adreamingsoul 5y agoThis is indeed a fun and challenging type of API endpoint to implement. I built a streaming statistics server that ingests data from edge nodes and aggregates the data into a memory database. The server has an API interface and one of the endpoints will stream all the data in either a compact or pretty JSON format. example of the datasets: millions of unique unordered keys with each key containing a list of time series data. The memory overhead to stream Gigabytes of data is limited to a couple of MB (buffers for encoding and writing). I ended up having to write my own JSON encoder to achieve the small memory overhead, JSON formatting, and write performance.
- Traubenfuchs 5y agoI‘d like to mention that streaming large results is now trivial in the Java world as well, thanks to reactive spring (spring webflux). It even works with SQL thanks to r2dbc (reactive SQL driver API). It‘s quite refreshing to develop with reactive spring.
- humbleMouse 5y agoDon’t mention reactive spring on hacker news, everyone here writes in GO and javascript. They’d rather re-invent the wheel every 5 minutes than use any out of the box java solution.
- loevborg 5y agoHow about returning large API results as NDJSON (newline-delimited JSON)? Modern browsers can read the response of a network request in a streaming fashion See e.g. https://canjs.com/doc/can-ndjson-stream.html https://canjs.com/doc/can-ndjson-stream.html
- lmilcin 5y agoWe are using ndjson. It is pretty good for our case of processing couple of terabytes of 1kB-20kB documents daily. Some of the calls return millions of objects. In such cases we opt to return very small amount of information (ideally just ID) and have the client batch responses and direct batches of IDs to another endpoint to fetch full documents, with response also streamed using ndjson. But there is nothing wrong with returning entire 20GB of data except that the client gets no other option than wait for entire set being returned through a single connection.
- karpour 5y agothat was my first thought too. ndjson makes a lot more sense there. You don't want to keep the entire streamed API response in memory anyway.
- simonw 5y agoI mentioned that here - I agree, it's a really simple and useful format for streaming: > Some export formats are friendlier for streaming than others. CSV and TSV are pretty easy to stream, as is newline-delimited JSON. It does have the same minor disadvantage as CSV in that it can be truncated (by an error or a dropped connection) in a way that can look like a successful download.
- lmilcin 5y agoI am switching all our endpoints from paging to streaming. Currently for the paging endpoints some of the clients are opening hundreds of connections to slurp tens of gigabytes of data as fast as possible. Switching these to streaming directly from database improved the performance of a single request about 100 times as clients can use 10 times less connections and still fetch whatever they need 10 times faster than before. As to drawbacks, the biggest one is additional effort needed to educate people on proper streaming, design of APIs, implementation and correct use of Reactor (ReactiveX implementation for Java). There are couple of technical drawbacks. Resuming a streaming response is a little bit more involved. My ideal solution, whenever I can, is to produce the report in order of some database index and if the call fails, have the client submit the ID of last received response when retrying the call. Achieving optimum performance requires a bit of thought and planning. A lot of internal processes require individual things to be batched together (we try to never direct calls to database for single items, opting to process everything in batches of at least 100 or 1000). When the endpoint has functionality that needs to be executed multiple times concurrently to get needed performance, sometimes an internal function also has its own concurrency because it needs to run something else multiple times in parallel to achieve its target internally. This can quickly cause the developer to loose track of how much stuff is running concurrently. The solution to this is that rather than components returning Flux<Something> typically they return Flux<Flux<Something>>. This can be thought as function saying "my result can be fetched from a collection of multiple substreams of results, and you can decide however many you are comfortable running concurrently". This allows to give the calling function the control of concurrency at the cost of little bit of mental yoga.
- cle 5y agoWhat about the other drawbacks mentioned in the article, like connection draining and error messaging?
- lmilcin 5y agoWell, connection draining is no harder than with normal connections. I think of this problem following way: you already have to design your system to be able to handle network faults and node crashes. So just kill the node and rely on existing solution to handle the error. This also gives the opportunity to test your fault handling logic. Error messaging is a little bit more complicated. At first sight, in normal request-response you get either successful response or an error. But if you look a little bit closer, in fact you get 200 response header followed by response data, which may itself fail to be transmitted. So technically you already need to be able to resolve an error that happens after you have received 200 header. In case of streaming response the difference is not in how the response is being sent to the client but how the client is processing it. The client has option to wait for entire response (header followed by entire stream of data) and only start processing the response after entire response has been received. This is no different from a normal, non-streamed request response. The client also has option to start processing the response while it is being received. The response does not need to be streamed from the server nor it needs any kind of streaming protocol. I have been stream-processing XML responses two decades ago with no special streaming support. One solution I use is that is that the client architecture treats this data as speculative meaning that an error can undo entire batch received. Another solution is that the design is such that the data makes sense even partially received, the processing of response is idempotent and the client will be asking, ad infinitum, to receive the rest of the response until the call succeeds. This of course has its own reliability problems (what if the error is permanent?)
- barrkel 5y agoEither do efficient pagination, or convert into an already fairly well solved problem, like large file downloads, via batch exports. Batch exports just move the problem in house though. Either way, don't rely on a continuous long running connection. You need to be able to restart or retry without starting over. That's table stakes, and it makes a resumable pure streaming API look a lot like a paginated one. The hard bit of pagination is efficient resuming, not limiting to a page size. Efficient pagination means having enough information on each subsequent request to pick up reliably and efficiently from where you left off, e.g. using a sequentially allocated indexed ID on the root table (to which other data is joined). Sometimes the database needs convincing to use the right index of you have joins, as DBs are generally tuned for extracting a handful of rows at a time. I've had more reliable success (over a multi-year maintenance timeline) by doing a single paginated query over a root table and doing separate queries to get the joined data, using a page of root IDs at a time for the joins. Doing the join in code also gives leeway for parallelization of the data load across cores (MySQL queries are single threaded), or even across replicas.
- simonw 5y agoI described an implementation using keyset pagination in the article, which sounds like what you're advocating for here.
- barrkel 5y agoIndeed.
- jon_richards 5y agoThis really annoys me in PostgreSQL. Django encourages you to use joins across one-to-one relationships, but the query planner doesn’t know how to apply the limit before the join. It’s relatively simple to change select_related to prefetch_related on all paginated queries, but it still feels like the query planner should do it, especially for left joins (inner joins are slightly more complex to check).
- dap 5y agoThere are at least two other reasons I prefer small (paginated) requests. First, it’s much simpler to define meaningful latency SLOs. If requests take roughly the same time, I can say that requests (or even requests “of this type”) should complete with p99 under 300ms. If they’re very variable, it’s harder to know when the service is degraded. (You can use time-to-first-byte, which is usually more uniform, but you can miss degradation that affects the data stream. Then it’s tempting to look at throughput, but that’s impacted by client speed, which you usually want to ignore here.) The other is that bulk requests can make it much cheaper to DoS the service. If all requests take about the same amount of work on the server, then forcing the server to do work requires proportional client resources to make the requests. If you can make one small request that causes the server to go off and do a bunch of work while the client does almost no work, that’s a cheap way to DoS a server.