5 ms·

I've been looking for something like this for a while, to the point where I started writing it yesterday. Thanks for saving my time! :D I've been running it fo

I've been looking for something like this for a while, to the point where I started writing it yesterday. Thanks for saving my time! :D I've been running it for the past few hours to put backups of my photos to Glacier. I have zips of each album which vary from a few megabytes to a few gigabytes - in total it's around 40GB. My internet connection isn't that reliable, so around half the uploads have failed with 'Connection Interrupted'. It would be great if they were automatically requeued (maybe at the end of the list?) and were able to be resumed without starting from scratch (I assume behind the scenes uploads are multipart?).

sebcode 11y ago
You're welcome :) Yes, Freeze uses multipart uploads with 16 MB chunks. Resuming multipart uploads works chunk-wise, for example when you've uploaded 20 MB and then the connection drops, the upload then continues at chunk number 2. When you see that an upload restarts from scratch, then the connection was probably interrupted while trying to upload the first 16 MB. About automatic retries: Freeze automatically retries 3 times and then you have retry manually. For unreliable and slow internet connection I recommend setting max. parallel uploads to 1 and disabling the upload speed limit. I plan make the chunk size configurable for a future update. I makes sense to use a smaller upload chunk size for unreliable internet connections.
- kzim 11y ago
  Yesterday I uploaded 12GB to Glacier and big surprise for me was price per upload request: $0.055 per 1,000 requests Command line tool[1] split my 12GB into 1mb chunks and generate around 12k requests: $0.055 per 1,000 requests 12,694 Requests $0.70 So bigger chunks are cheaper, but probably generate more retries. [1] - https://github.com/uskudnik/amazon-glacier-cmd-interface https://github.com/uskudnik/amazon-glacier-cmd-interface

sebcode 11y ago

You're welcome :) Yes, Freeze uses multipart uploads with 16 MB chunks. Resuming multipart uploads works chunk-wise, for example when you've uploaded 20 MB and then the connection drops, the upload then continues at chunk number 2. When you see that an upload restarts from scratch, then the connection was probably interrupted while trying to upload the first 16 MB. About automatic retries: Freeze automatically retries 3 times and then you have retry manually. For unreliable and slow internet connection I recommend setting max. parallel uploads to 1 and disabling the upload speed limit. I plan make the chunk size configurable for a future update. I makes sense to use a smaller upload chunk size for unreliable internet connections.

kzim 11y ago
Yesterday I uploaded 12GB to Glacier and big surprise for me was price per upload request: $0.055 per 1,000 requests Command line tool[1] split my 12GB into 1mb chunks and generate around 12k requests: $0.055 per 1,000 requests 12,694 Requests $0.70 So bigger chunks are cheaper, but probably generate more retries. [1] - https://github.com/uskudnik/amazon-glacier-cmd-interface https://github.com/uskudnik/amazon-glacier-cmd-interface