4 ms·
I have an application where users upload hundreds of small (~10kB) files every day. We quickly realized that S3 was not an option, since you get charged per req
by cshimmin 10y ago
I have an application where users upload hundreds of small (~10kB) files every day. We quickly realized that S3 was not an option, since you get charged per request. As a workaround we batched the file uploads by archiving uploads each hour, and uploading a single .tar.gz to S3 at the end of the hour. The aggregated data gets processed offline later.
We have N upload endpoints (as part of an autoscaling group), it's a bit of a pain to reaggregate the data since each server uploads its own .tgz to S3. I'm very happy to have EFS now! It makes scaling out our upload endpoints much simpler, as they can just dump their data into a common directory (which can still get archived and persisted to S3 later).