10 ms·

Glacier's pricing structure is complicated, but fortunately it's now fairly straightforward to set up a policy to cap your data retrieval rate and limit your co

by re 11y ago

Glacier's pricing structure is complicated, but fortunately it's now fairly straightforward to set up a policy to cap your data retrieval rate and limit your costs. This was only introduced a year ago, so if like Marko you started using Glacier before that it could be easy to miss, but it's probably something that anyone using Glacier should do. http://docs.aws.amazon.com/amazonglacier/latest/dev/data-retrieval-policy.html#data-retrieval-policy-using-console http://docs.aws.amazon.com/amazonglacier/latest/dev/data-ret... https://aws.amazon.com/blogs/aws/data-retrieval-policies-audit-logging-glacier/ https://aws.amazon.com/blogs/aws/data-retrieval-policies-aud...

vonklaus 11y ago
> fortunately it is fairly strait forward to cap your data retreival rates. Amazon has done a great job with this feature. By doing a poor job implementing something for an extremely narrow usecase, in a technology that is outdated and then providing the most complicated pricing structure surrounding every aspect of the product one can't helpbut use the feature: any other provider or service. Like, wtf would be the usecase for amazon glacier in 2016? I dont think I would put hubdreds of petabytes of sata into 20 year cold storage, and the author of this post certainly wouldnt use it again. The fact that i need to read 2 pages of pricing docs and then the 2 pages you linked to control them because I cant estimate them myself, is a sure sign this is absurd
- zcdziura 11y ago
  Tape storage is still the most optimal form of long-term storage. If you need to store things for an exceptionally long time, such as financial data, scientific data, etc, then you're going to get the most bang for your buck on tape.
  - barkingcat 11y ago
    Yes but if you were in this target market you'd likely want your information on your own tape with a reading infrastructure you control.
    - Sanddancer 11y ago
      Depends. Having an in-house key and shipping everything to Amazon encrypted means that you have all the infrastructure there and waiting, and not capable of being read. Additionally, that tape library would need to be stored, and periodically tested so that tapes can be rotated out as needed. Sending that data to a service like glacier means you've shown due diligence, but at the same time, don't need to maintain a schedule of testing every disk every year.
      - PhantomGremlin 11y ago
        don't need to maintain a schedule of testing every disk every year. So you trust Glacier or Google Nearline (or any similar provider) without testing? No testing ever??? I wouldn't feel happy with my critical data floating around in the cloud, without my checking it at least once a year to make sure it really exists. And once you do start verifying that data, you will incur all sorts of charges to access it.
    - vonklaus 11y ago
      this was what I meant. thanks. You can buy a 5tb hard drive for ~138.00. You could likely buy several of them at discount to get started. As you go forward in time, these will become much cheaper, allowing you to continue purchasing them on demand from the market for much less money. This allows you to trivially share, copy, move and retrieve that data quickly as well as fully control who has access and when. I am sure there are use cases for this but in a situation where you have petabyte scale data, it is often the case that you also have the infrastructure to save it. How many places would need to store >5tb of data a week that * don't have this capability in house * will almost never need to access it again. * will not need it in a timely manner, if they do need to access it again. * don't have the money to implement their dedicated server and storage on site for this purpose. I am not saying that this rules everyone out, but the prices are so low, and tape must be so annoying, I couldn't imagine why they keep offering this. Obviously, some peopel must be using it but in 2016 with storage prices being so low already, i don't know how many places have this amount of data and meet the above requirements.
  - vonklaus 11y ago
    The post states he paid $150 yo retrieve 60gb of data. For $150 you can buy a 5tb hdd drive. What usecase could the price delta make sense to have a 4 hour feedback loop and all of your important data locked in someone elses data center. the usecase where your data is so properly massive that this makes sense && you don't have the storage infrastructure in place, is so narrow that it doesnt make sense. It is basically one research student's crawl data Edit: also, s3 is pretty cheap. So again, i dont really see the usecase here. How much room is in the market between your own physical or digital system and amazon s3 or an equivalent. you would have to have a massive amount of data you dont care about and be very price sensitive.
    - 88e282102ae2e5b 11y ago
      Durability is one reason. A physical hard drive in my desk is so much more susceptible to destruction or loss or theft.
      - bigiain 11y ago
        This. The data I care about is already backed up on two different multi-TB drives at home, and another one at work. Glacier is the contingency for "something took out the original data and all three backups in two different locations 7 or 8km apart - if I'm still alive after whatever just happened, I'll consider whether or not to pay Amazon a grand or so to retrieve it quickly from Glacier, or wait ~20 months to get it all for single-digit-dollars".
        manigandham 11y ago
        If you're talking about personal data, why not just use backblaze and amazon's consumer unlimited cloud storage? That gives you 2 backup providers that can durably store everything and it's free and quick to access. Why deal with all the harddrives and glacier?
        88e282102ae2e5b 11y ago
        Neither of those seem free - am I looking in the wrong place? Also there's no Linux client for either AFAICT.
- jonas21 11y ago
  There are some use cases listed here, and they seem pretty reasonable to me: https://aws.amazon.com/glacier https://aws.amazon.com/glacier Lots of businesses have data retention requirements, and it can be difficult and time consuming to make sure this data is backed up in a way that is secure and can survive a catastrophe. The author's use case (and most other personal use cases) might not be a good fit for Glacier, but he's not the target market.
- Sanddancer 11y ago
  SOX compliance, legal requirements to save communications, etc. There are a lot of places where there are needs to maintain a huge amount of information that you're probably never going to need again. Not all products are for all people. If you foresee a need to recover a large amount of data all at once, then glacier's not for you. If you might occasionally need a filing from 6 years ago, then glacier would be great.
  - true_religion 11y ago
    It's not about recovering a large amount of data; its about recovering a large % of your own stored data. Amazon starts to charge you extra anytime you exceed restoring 5% of the data. If for example you save all your tax-related documents in Glacier, then you are audited then the accounts department or the government will want all the information. Not 5% of it. Not 10% of it. Everything. At that point Amazon will have you over a barrel, because getting the data out at a reasonable time frame will cost exponentially more than dripping out the data over the course of 20 months.
    - quanticle 11y ago
      >If for example you save all your tax-related documents in Glacier, then you are audited then the accounts department or the government will want all the information. Not 5% of it. Not 10% of it. Everything. Are you sure about that? I haven't worked with tax litigation specifically, but I've worked with e-discovery w.r.t. e-mail and I can assure you that no one ever asks for all the e-mails sent by a particular company over all time. It's always a matter of asking for the e-mails sent by or received by a select group of people, over a fairly discrete time period. For something like this, a Glacier store might make sense, if it was coupled with an online metadata cache stored in e.g. S3.
      - true_religion 11y ago
        With tax litigation the issue is that you have to prove you didn't simply shift money and accounting briefs around, and the only way to realistically prove it is to show all the statements in the time period that you'd required to keep them (I think that's the last 6 years). The government basically comes to you and says they think you owe X, and you have to prove that false to their satisfaction. The more data you give your CPA to work with, the better.

by re 11y ago

vonklaus 11y ago

> fortunately it is fairly strait forward to cap your data retreival rates. Amazon has done a great job with this feature. By doing a poor job implementing something for an extremely narrow usecase, in a technology that is outdated and then providing the most complicated pricing structure surrounding every aspect of the product one can't helpbut use the feature: any other provider or service. Like, wtf would be the usecase for amazon glacier in 2016? I dont think I would put hubdreds of petabytes of sata into 20 year cold storage, and the author of this post certainly wouldnt use it again. The fact that i need to read 2 pages of pricing docs and then the 2 pages you linked to control them because I cant estimate them myself, is a sure sign this is absurd

zcdziura 11y ago
Tape storage is still the most optimal form of long-term storage. If you need to store things for an exceptionally long time, such as financial data, scientific data, etc, then you're going to get the most bang for your buck on tape.
- barkingcat 11y ago
  Yes but if you were in this target market you'd likely want your information on your own tape with a reading infrastructure you control.
  - Sanddancer 11y ago
    Depends. Having an in-house key and shipping everything to Amazon encrypted means that you have all the infrastructure there and waiting, and not capable of being read. Additionally, that tape library would need to be stored, and periodically tested so that tapes can be rotated out as needed. Sending that data to a service like glacier means you've shown due diligence, but at the same time, don't need to maintain a schedule of testing every disk every year.
    - PhantomGremlin 11y ago
      don't need to maintain a schedule of testing every disk every year. So you trust Glacier or Google Nearline (or any similar provider) without testing? No testing ever??? I wouldn't feel happy with my critical data floating around in the cloud, without my checking it at least once a year to make sure it really exists. And once you do start verifying that data, you will incur all sorts of charges to access it.
  - vonklaus 11y ago
    this was what I meant. thanks. You can buy a 5tb hard drive for ~138.00. You could likely buy several of them at discount to get started. As you go forward in time, these will become much cheaper, allowing you to continue purchasing them on demand from the market for much less money. This allows you to trivially share, copy, move and retrieve that data quickly as well as fully control who has access and when. I am sure there are use cases for this but in a situation where you have petabyte scale data, it is often the case that you also have the infrastructure to save it. How many places would need to store >5tb of data a week that * don't have this capability in house * will almost never need to access it again. * will not need it in a timely manner, if they do need to access it again. * don't have the money to implement their dedicated server and storage on site for this purpose. I am not saying that this rules everyone out, but the prices are so low, and tape must be so annoying, I couldn't imagine why they keep offering this. Obviously, some peopel must be using it but in 2016 with storage prices being so low already, i don't know how many places have this amount of data and meet the above requirements.
- vonklaus 11y ago
  The post states he paid $150 yo retrieve 60gb of data. For $150 you can buy a 5tb hdd drive. What usecase could the price delta make sense to have a 4 hour feedback loop and all of your important data locked in someone elses data center. the usecase where your data is so properly massive that this makes sense && you don't have the storage infrastructure in place, is so narrow that it doesnt make sense. It is basically one research student's crawl data Edit: also, s3 is pretty cheap. So again, i dont really see the usecase here. How much room is in the market between your own physical or digital system and amazon s3 or an equivalent. you would have to have a massive amount of data you dont care about and be very price sensitive.
  - 88e282102ae2e5b 11y ago
    Durability is one reason. A physical hard drive in my desk is so much more susceptible to destruction or loss or theft.
    - bigiain 11y ago
      This. The data I care about is already backed up on two different multi-TB drives at home, and another one at work. Glacier is the contingency for "something took out the original data and all three backups in two different locations 7 or 8km apart - if I'm still alive after whatever just happened, I'll consider whether or not to pay Amazon a grand or so to retrieve it quickly from Glacier, or wait ~20 months to get it all for single-digit-dollars".
      - manigandham 11y ago
        If you're talking about personal data, why not just use backblaze and amazon's consumer unlimited cloud storage? That gives you 2 backup providers that can durably store everything and it's free and quick to access. Why deal with all the harddrives and glacier?
        88e282102ae2e5b 11y ago
        Neither of those seem free - am I looking in the wrong place? Also there's no Linux client for either AFAICT.
jonas21 11y ago
There are some use cases listed here, and they seem pretty reasonable to me: https://aws.amazon.com/glacier https://aws.amazon.com/glacier Lots of businesses have data retention requirements, and it can be difficult and time consuming to make sure this data is backed up in a way that is secure and can survive a catastrophe. The author's use case (and most other personal use cases) might not be a good fit for Glacier, but he's not the target market.
Sanddancer 11y ago
SOX compliance, legal requirements to save communications, etc. There are a lot of places where there are needs to maintain a huge amount of information that you're probably never going to need again. Not all products are for all people. If you foresee a need to recover a large amount of data all at once, then glacier's not for you. If you might occasionally need a filing from 6 years ago, then glacier would be great.
- true_religion 11y ago
  It's not about recovering a large amount of data; its about recovering a large % of your own stored data. Amazon starts to charge you extra anytime you exceed restoring 5% of the data. If for example you save all your tax-related documents in Glacier, then you are audited then the accounts department or the government will want all the information. Not 5% of it. Not 10% of it. Everything. At that point Amazon will have you over a barrel, because getting the data out at a reasonable time frame will cost exponentially more than dripping out the data over the course of 20 months.
  - quanticle 11y ago
    >If for example you save all your tax-related documents in Glacier, then you are audited then the accounts department or the government will want all the information. Not 5% of it. Not 10% of it. Everything. Are you sure about that? I haven't worked with tax litigation specifically, but I've worked with e-discovery w.r.t. e-mail and I can assure you that no one ever asks for all the e-mails sent by a particular company over all time. It's always a matter of asking for the e-mails sent by or received by a select group of people, over a fairly discrete time period. For something like this, a Glacier store might make sense, if it was coupled with an online metadata cache stored in e.g. S3.
    - true_religion 11y ago
      With tax litigation the issue is that you have to prove you didn't simply shift money and accounting briefs around, and the only way to realistically prove it is to show all the statements in the time period that you'd required to keep them (I think that's the last 6 years). The government basically comes to you and says they think you owe X, and you have to prove that false to their satisfaction. The more data you give your CPA to work with, the better.