9 ms·

Glacier pricing is surprisingly complicated, and the actual cost can be much higher than $0.01 per GB-month if you don't read the fine print. The biggest gotch

by panarky 12y ago

Glacier pricing is surprisingly complicated, and the actual cost can be much higher than $0.01 per GB-month if you don't read the fine print. The biggest gotcha is that you can only access 0.17% of the data you've stored in any given day without extra charges. So if you've stored 1000 GB, you can only access 1.7 GB per day for free. The cost for going over your daily "retrieval allowance" can be large, because cost is driven by "peak retrieval rate". They find the highest hourly retrieval rate over the entire month, and charge the whole month's retrieval at the peak rate. This can get expensive fast. Again, with 1000 GB stored, if you retrieve 200 GB over the course of 4 hours, your hourly retrieval rate is 200 GB / 4 hours = 50 GB per hour. Your free retrieval allowance is 1.7 GB per day / 24 hours = 0.07 GB / hour. So your excess retrieval is 49.93 GB per hour. Amazingly, that 49.93 GB per hour for 4 hours is charged for all 720 hours in the month, so that's 49.93 * 720 * 0.01 = $359.49. That's an astonishing $1.79 per GB just to retrieve one-fifth of the data from Glacier storage. So Glacier only makes sense for true cold storage, where you are very unlikely to touch more than 0.00007 of it in any given hour. Google's Nearline is much simpler and faster for the same storage cost -- and far cheaper if you need to access more than a tiny fraction of the data. Source: http://aws.amazon.com/glacier/faqs/#How_much_data_can_I_retrieve_for_free http://aws.amazon.com/glacier/faqs/#How_much_data_can_I_retr...

AceJohnny2 12y ago
Thanks for the detailed breakdown, that's really interesting! Sounds like you were bitten by this? :)
- Siliticxx 12y ago
  Ok. I have to this to this. i'm sorry. I guess he would say he was... sunglasses FROSTBITTEN YEAH!
  - AceJohnny2 12y ago
    I understand, but please go stand in the corner there and think about what you've done.
- ak217 12y ago
  Don't know about the OP, but we were bitten by this - badly - after our logic to throttle Glacier retrieval failed. AWS has now added the ability to set a spending limit to avoid runaway retrievals. Nonetheless, from my point of view the ultra-complicated pricing scheme is nothing short of a disaster for Glacier as a product. I think it will continue to seriously impact its uptake, and Google is smart to exploit the opportunity to offer a simple, straightforward pricing scheme.
  - NeutronBoy 12y ago
    The use case is people who have a requirement to store data and probably never need to access it, either for reg requirements, or legal requirements, etc. In which case paying a couple of hundred bucks for retrieval is fine, especially if they can access at a high level of granularity edit: Also as others have pointed out, Nearline is limited in retrieval time as well, so the cost difference isn't nearly as large.
oconnore 12y ago
Why does Amazon charge like this? It seems like on the rare occasion that you need to send some person to go grab the tape/disk from storage and bring it online, Amazon would want you to get all the data you need and put it back in storage. Incentivizing users to bring the data online once a day to trickle it out seems bad for all involved.
- tveita 12y ago
  I think it makes most sense to think of it as paying for access to the tape robot. If Amazon store your data in a tape archive (I don't know if they do, but they at least seem to have similar constraints), they can only access a small portion of the stored data at a time, so they need to control how often people request data. They could just rate limit everyone, but this way allows people to pay for priority in an emergency while still discouraging everyday read requests. The pricing makes more sense if you're a large user with data spanning several tapes than if you're in the single terabyte range, but the low limit still discourages you from making requests causally, which helps them keep their SLA. If they predict that you'll trickle out your small file, they can just read out everything on the first access and cache it online, so there's no extra trips to the archive for them.
  - discodave 12y ago
    You assume Glacier uses tape storage... But the same principle still applies, they're trying to prevent people clogging up the network so when somebody does need to do an emergency restore there is lots of spare network capacity.
  - breakingcups 12y ago
    Note: The following is speculation best I know. I'm recalling from memory what I've read on the internet written by someone who did not have a direct source. Glacier uses low-speed (5400 RPM) consumer drives, which they then clock down lower to save energy. Any given 'rack' only has enough power to power a few drives on that rack, the rest is powered down. To prevent multiple customers from all trying to pull their data out they needed to introduce a rate limiting system, which they did with this exorbitant pricing.

by panarky 12y ago

AceJohnny2 12y ago

Thanks for the detailed breakdown, that's really interesting! Sounds like you were bitten by this? :)

Siliticxx 12y ago
Ok. I have to this to this. i'm sorry. I guess he would say he was... sunglasses FROSTBITTEN YEAH!
- AceJohnny2 12y ago
  I understand, but please go stand in the corner there and think about what you've done.
ak217 12y ago
Don't know about the OP, but we were bitten by this - badly - after our logic to throttle Glacier retrieval failed. AWS has now added the ability to set a spending limit to avoid runaway retrievals. Nonetheless, from my point of view the ultra-complicated pricing scheme is nothing short of a disaster for Glacier as a product. I think it will continue to seriously impact its uptake, and Google is smart to exploit the opportunity to offer a simple, straightforward pricing scheme.
- NeutronBoy 12y ago
  The use case is people who have a requirement to store data and probably never need to access it, either for reg requirements, or legal requirements, etc. In which case paying a couple of hundred bucks for retrieval is fine, especially if they can access at a high level of granularity edit: Also as others have pointed out, Nearline is limited in retrieval time as well, so the cost difference isn't nearly as large.

oconnore 12y ago

Why does Amazon charge like this? It seems like on the rare occasion that you need to send some person to go grab the tape/disk from storage and bring it online, Amazon would want you to get all the data you need and put it back in storage. Incentivizing users to bring the data online once a day to trickle it out seems bad for all involved.

tveita 12y ago
I think it makes most sense to think of it as paying for access to the tape robot. If Amazon store your data in a tape archive (I don't know if they do, but they at least seem to have similar constraints), they can only access a small portion of the stored data at a time, so they need to control how often people request data. They could just rate limit everyone, but this way allows people to pay for priority in an emergency while still discouraging everyday read requests. The pricing makes more sense if you're a large user with data spanning several tapes than if you're in the single terabyte range, but the low limit still discourages you from making requests causally, which helps them keep their SLA. If they predict that you'll trickle out your small file, they can just read out everything on the first access and cache it online, so there's no extra trips to the archive for them.
- discodave 12y ago
  You assume Glacier uses tape storage... But the same principle still applies, they're trying to prevent people clogging up the network so when somebody does need to do an emergency restore there is lots of spare network capacity.
- breakingcups 12y ago
  Note: The following is speculation best I know. I'm recalling from memory what I've read on the internet written by someone who did not have a direct source. Glacier uses low-speed (5400 RPM) consumer drives, which they then clock down lower to save energy. Any given 'rack' only has enough power to power a few drives on that rack, the rest is powered down. To prevent multiple customers from all trying to pull their data out they needed to introduce a rate limiting system, which they did with this exorbitant pricing.