4 ms·
I built this tool to create tar archives on Amazon S3 without the need to download the data. It uses existing Amazon S3 APIs to ‘copy’ the data into a new Multi
by yanko7 4y ago
I built this tool to create tar archives on Amazon S3 without the need to download the data. It uses existing Amazon S3 APIs to ‘copy’ the data into a new Multipart object in the backend. There is no minimum size for the objects as long as the total archive is greater than 5MB, the maximum size for an object in the archive can be 5GB and the total archive can be up to 5TB. This tool helps solve the problem when users have too many objects by combining them into one and then moving it to glacier. Reducing the cost of glacier requests and minimum object size requirements.
Tars generated with this tool can also be extracted on Amazon S3 without having to download the file. This is possible because a table of contents csv file is prepended for the tool to know how to extract the data.
- QuinnyPig 4y agoI like this very much. I dream of one day having native zip/brotli functionality but that’s almost certainly a terrible idea for a variety of reasons.
- yanko7 4y agoThanks :) Yes to QuinnyPig's point, for other readers, it's worth noting that because this tool is just copying data from one object to another via API calls, there is no compute involved that the data goes through. So the tars created are not compressed.