4 ms·
This is how I do it: store the large data files on GCS. Then, offer them via an F1 Node.js endpoint but make the process 'smart' and offer them in 10MB chunks.
by collaborative 5y ago
This is how I do it: store the large data files on GCS. Then, offer them via an F1 Node.js endpoint but make the process 'smart' and offer them in 10MB chunks. The client sees a chunk progress (3/10 etc) and a chunk % progress (0 - 100 for each chunk). The client also sequentially persists the large text responses as they arrive
This has saved both my very limited F1 server from sad OOM exceptions and my iOS vm'd client from crashing while processing too much text
I do the same for uploads
- foota 5y agoDo you just read the response linearly? What made you choose GCS instead of something like memcache With some kind of chunking?
- collaborative 5y agoSo the great thing about using an F1 instance is that it completely falls under the free tier (28hrs/day) There is no memcache in node.js F1 but what I do is I use the GCS container itself as stream memory instead of the usual node.js memory multer (yes, this is possible!) This only adds a few ms to the read op which is entirely fine
- arthurcolle 5y ago28 hrs/ day...?
- collaborative 5y ago1 instance running all day = 24 hrs Times as many instances as you have. So 28 hrs = 1 instance running all day + an extra 4 hours of a second instance It auto scales
- arthurcolle 5y agoThanks, thought I had slipped into an alternative dimension. Appreciate you helping me take the edge off.
- collaborative 5y agoNo probs mate
- orbz 5y agoIf you want to get real fancy you can have your F1 server generate a self-signed GCS url and have it returned as a 307 redirect. Even less load!
- collaborative 5y agoI need to serve it from an endpoint because the response includes multiple text files from GCS bundled together (as many as can fit a 10MB chunk). But yeah otherwise that would work (for downloads)