3 ms·

How about returning large API results as NDJSON (newline-delimited JSON)? Modern browsers can read the response of a network request in a streaming fashion Se

• lmilcin 5y ago
  We are using ndjson. It is pretty good for our case of processing couple of terabytes of 1kB-20kB documents daily. Some of the calls return millions of objects. In such cases we opt to return very small amount of information (ideally just ID) and have the client batch responses and direct batches of IDs to another endpoint to fetch full documents, with response also streamed using ndjson. But there is nothing wrong with returning entire 20GB of data except that the client gets no other option than wait for entire set being returned through a single connection.
• karpour 5y ago
  that was my first thought too. ndjson makes a lot more sense there. You don't want to keep the entire streamed API response in memory anyway.
• simonw 5y ago
  I mentioned that here - I agree, it's a really simple and useful format for streaming: > Some export formats are friendlier for streaming than others. CSV and TSV are pretty easy to stream, as is newline-delimited JSON. It does have the same minor disadvantage as CSV in that it can be truncated (by an error or a dropped connection) in a way that can look like a successful download.