5 ms·

Glacier Costlier Than S3 for Small Files

47 points by drue 11y ago

jakozaur 11y ago
AWS is more honest with S3 Infrequent Access: https://aws.amazon.com/s3/storage-classes/ https://aws.amazon.com/s3/storage-classes/ No additional metadata, but you get charged at least for 128 KB . Luckily lifecycle transition doesn't move files smaller than 128 KB. However, even using lifecycle transition you pay $0.01 per 1000 transitions. It doesn't seem much but for smaller items it can decrease savings a lot. E.g. if you average file is 6 MB, than you will loose 4 days of savings on S3 IA comparing to standard class.
aps-sids 11y ago
Unfortunately, I learned it the hard way :(
raverbashing 11y ago
The real question is why are people glaciering small files? Tar everything and send it up. And then do an incremental backup Amazon charges per operation, reduce the number of operations
- lucaspiller 11y ago
  My guess is they don't realise. AWS let's you setup lifecycle rules to archive S3 objects to Glacier automatically. It doesn't explicitly say it is cheaper (it says "this may reduce your costs"), but they probably just saw the headline prices for Glacier and moved everything.
  - smackfu 11y ago
    Also, I would guess a lot of Glacier users are probably just slotting it in place of some other system that doesn't have the same pricing model.
- drue 11y ago
  If you write a AWS-native service that uses S3 as a back end (a very reasonable thing to do), it may create many small files in S3. It would be unreasonable to incur the costs to retrieve them and tar them up, and it also does not make cents to "click the box" to have them archived to glacier.
TazeTSchnitzel 11y ago
> For consistency and precision, the following units are used throughout this article. > KB: 1,024 bytes, expressed as 2^10 Why not just use KiB? Unlike KB, it's unambiguously binary.
- drue 11y ago
  I primarily wanted to note that I used 2^X notation in the math, in case it was confusing for folks. I used KB/MB/GB to be consistent with the language AWS uses on their S3 pricing page (https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/).
tghw 11y ago
Personally, I'm looking forward to Backblaze's cloud storage, which is being advertised at $0.005/GB/month. Cheaper than glacier, without all of the transaction fees. https://www.backblaze.com/b2/cloud-storage.html https://www.backblaze.com/b2/cloud-storage.html
- Veratyr 11y ago
  And none of the 3 hour retrieval time. Might drag me off GCS.
- tw04 11y ago
  The whole custom API instead of S3 or Swift compatibility is really, REALLY annoying.
  - toomuchtodo 11y ago
    Someone from Backblaze previously mentioned on HN they're working on an S3-compatible API.
bound008 11y ago
AWS has a wizard for bucket storage policy where they tell you this information explicitly with a big warning sign.
boulos 11y ago
This is one of the things I love about GCS Nearline, it really is a penny per byte and then the whole retrieval charges (which for my lazy rsync backup is never). Disclaimer: I work at Google on Compute Engine, but not on GCS.
nine_k 11y ago
Glacier has a very clear use case, to my mind. It is useful for keeping archives of massive data you're unlikely to ever need, but legally obliged to keep around, or just want to have available for a very improbable later examination. Think some huge transaction logs of two years back. For a case like this, you don't need fast retrieval, and mostly you don't need retrieval at all. You plan ahead to only ever retrieve a small percent of these data. The rest will be silently discarded when retention period has expired. If your use case is not like that, Glacier probably makes little sense for you. This is totally not a backup which you likely keep in order to restore the entire state from as soon as possible.
- drue 11y ago
  Right, but you missed the salient point: it also does not make sense for files smaller than about 200KB.

47 points by drue 11y ago

AWS is more honest with S3 Infrequent Access: https://aws.amazon.com/s3/storage-classes/ https://aws.amazon.com/s3/storage-classes/ No additional metadata, but you get charged at least for 128 KB . Luckily lifecycle transition doesn't move files smaller than 128 KB. However, even using lifecycle transition you pay $0.01 per 1000 transitions. It doesn't seem much but for smaller items it can decrease savings a lot. E.g. if you average file is 6 MB, than you will loose 4 days of savings on S3 IA comparing to standard class.

aps-sids 11y ago

Unfortunately, I learned it the hard way :(

raverbashing 11y ago

The real question is why are people glaciering small files? Tar everything and send it up. And then do an incremental backup Amazon charges per operation, reduce the number of operations

lucaspiller 11y ago
My guess is they don't realise. AWS let's you setup lifecycle rules to archive S3 objects to Glacier automatically. It doesn't explicitly say it is cheaper (it says "this may reduce your costs"), but they probably just saw the headline prices for Glacier and moved everything.
- smackfu 11y ago
  Also, I would guess a lot of Glacier users are probably just slotting it in place of some other system that doesn't have the same pricing model.
drue 11y ago
If you write a AWS-native service that uses S3 as a back end (a very reasonable thing to do), it may create many small files in S3. It would be unreasonable to incur the costs to retrieve them and tar them up, and it also does not make cents to "click the box" to have them archived to glacier.

TazeTSchnitzel 11y ago

> For consistency and precision, the following units are used throughout this article. > KB: 1,024 bytes, expressed as 2^10 Why not just use KiB? Unlike KB, it's unambiguously binary.

drue 11y ago
I primarily wanted to note that I used 2^X notation in the math, in case it was confusing for folks. I used KB/MB/GB to be consistent with the language AWS uses on their S3 pricing page (https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/).

tghw 11y ago

Personally, I'm looking forward to Backblaze's cloud storage, which is being advertised at $0.005/GB/month. Cheaper than glacier, without all of the transaction fees. https://www.backblaze.com/b2/cloud-storage.html https://www.backblaze.com/b2/cloud-storage.html

Veratyr 11y ago
And none of the 3 hour retrieval time. Might drag me off GCS.
tw04 11y ago
The whole custom API instead of S3 or Swift compatibility is really, REALLY annoying.
- toomuchtodo 11y ago
  Someone from Backblaze previously mentioned on HN they're working on an S3-compatible API.

bound008 11y ago

AWS has a wizard for bucket storage policy where they tell you this information explicitly with a big warning sign.

boulos 11y ago

This is one of the things I love about GCS Nearline, it really is a penny per byte and then the whole retrieval charges (which for my lazy rsync backup is never). Disclaimer: I work at Google on Compute Engine, but not on GCS.

nine_k 11y ago

Glacier has a very clear use case, to my mind. It is useful for keeping archives of massive data you're unlikely to ever need, but legally obliged to keep around, or just want to have available for a very improbable later examination. Think some huge transaction logs of two years back. For a case like this, you don't need fast retrieval, and mostly you don't need retrieval at all. You plan ahead to only ever retrieve a small percent of these data. The rest will be silently discarded when retention period has expired. If your use case is not like that, Glacier probably makes little sense for you. This is totally not a backup which you likely keep in order to restore the entire state from as soon as possible.

drue 11y ago
Right, but you missed the salient point: it also does not make sense for files smaller than about 200KB.