7 ms·
Lower Cost S3 Storage Option and Glacier Price Reduction
- cperciva 11y agoThis looks like it would be perfect for Tarsnap -- the data Tarsnap stores is almost always kept for 30+ days, and it's almost always in objects of 128kB or more. The $0.01/GB for reads would be annoying (one of the reasons Tarsnap is hosted in EC2 is because it has free data transfer to and from S3; data is regularly retrieved and stored back after filtering out blocks marked for deletion) but it would be cheaper. One thing concerns me however: Standard – IA has an availability SLA of 99%. If this is just a reduced SLA but the actual availability is likely to be similar, that's fine. But if the actual availability is not expected to hit 99.9% -- say, if the backend implementation is "one copy in online storage, plus a backup in Glacier which gets retrieved if the online copy dies" that would be completely inadequate. Hopefully we'll get more details over time.
- toomuchtodo 11y agoBased on the post, it seems availability drops but durability remains the same. You might need to rest to get an object, and you'll be successful eventually.
- cperciva 11y agoRight. But it matters how much availability drops, and also what the correlation is between failures -- if they're completely uncorrelated but there's a 1% failure rate, you just retry, but if 1% of objects are going to be unavailable for the next four hours, that's a problem.
- derefr 11y agoFrom one of the customer quotes on the page: > it’s vital that customers have immediate, instant access to any of [our photos] at a moment’s notice – even if they haven’t been viewed in years. [IA] offers the same high durability and performance ... so we can continue to deliver the same amazing experience for our customers. The way this was phrased implies that this customer's use-case had a hard requirement that all of their data be in "online" storage at all times, and their satisfaction implies that IA does, in fact, hit this requirement. I'm not sure what the 99% SLA means given that.
- cperciva 11y agoHuh, I hadn't noticed that quote. Now I'm really confused.
- seiji 11y agoTo me it sounds like: S3 Classic: Your file gets replicated on 3 live HDs (and/or HDs backed by RAID arrays—not sure about the internal S3 storage topology). S3 Infrequent: Your file gets stored on 1 live HD (or single hardware redundancy component) and a copy in Glacier. If your live HD dies, your file will be automatically restored from Glacier to a new HD (but your data may be inaccessible during the automatic re-deploy). Glacier: offline bluray combined with error correction accessed by robot arms, temporarily restored to live HDs on demand.
- deleted 11y ago[deleted]
- breakingcups 11y agoI thought Glacier used "green" consumer drives clocked down even further to save power?
- acveilleux 11y agoWhichever they use, most disks are likely unpowered most of the time. Like the Facebook equivalent where they can only power 1 out of 12 disks at any time.
- 6t6t6 11y agoI understand that 99% doesn't mean that 1% of the objects will be always unaccesible. Instead, I guess that they mean is that they allow themselves to have up to 80 hours a year of downtime for any bucket.
- jeffbarr 11y agoThis is online storage. If a GET fails, just retry as usual (most higher-level libraries do this automatically, sometimes with a backoff mechanism).
- cperciva 11y agoIf a GET fails, just retry as usual Thanks! This is a very important detail which isn't documented anywhere: Retries are likely to succeed. A service where 1% of requests fail but failures are completely uncorrelated is far more usable than a service where 0.01% of requests fail but they keep on failing no matter how many times you retry them.
- zeckalpha 11y agoAdditionally, assuming your block data is being hash-addressed, i.e. not changing the S3 objects once they are in S3, adding CloudFront in front of your buckets may go a long way to increasing that percentage. However, its SLA is a little more involved (for better or worse): http://aws.amazon.com/cloudfront/sla/ http://aws.amazon.com/cloudfront/sla/ I'm not accessing S3 from EC2, though, so another benefit for me was it brought my S3 network costs way down.
- seiji 11y ago(constructive review) I had to read these sentences a few times to understand what you were trying to say: "You now have the choice of three S3 storage classes (Standard, Standard – IA, and Glacier) that are designed to offer 99.999999999% (eleven nines) of durability. Standard – IA has an availability SLA of 99%." No availability is mentioned for the others, but I assume it's 100%? Perhaps a simple table could help readers to scan and visually compare the values of two properties across three service classes?
- dbarlett 11y agoS3 Standard: "Designed for 99.99% availability over a given year" [1] Standard - IA: "Designed for 99.9% availability over a given year" [2] Glacier: "Retrieval jobs typically complete within 3-5 hours." [3] [1] https://aws.amazon.com/s3/storage-classes/#Amazon_S3_Standard https://aws.amazon.com/s3/storage-classes/#Amazon_S3_Standar... [2] https://aws.amazon.com/s3/storage-classes/#Infrequent_Access https://aws.amazon.com/s3/storage-classes/#Infrequent_Access [3] http://aws.amazon.com/glacier/faqs/#How_can_I_retrieve_data http://aws.amazon.com/glacier/faqs/#How_can_I_retrieve_data
- TheDong 11y agoOne possibility is the reduced SLA is to account for code bugs / youth, not for any expected difference. They might expect to be just as good as regular in the happy path, but are under-promising out of fear of some code-bug or other issue. Either way, I'd definitely not migrate too soon for exactly the above reason.
- Trisell 11y agoWith the cost per GB for Glacier Storage now. Any small to medium company would be fiscally irresponsible to not use it as a primary disaster recovery option. $84 per year per TB is ridiculous for geographically diverse storage.
- panarky 11y agoGlacier pricing is cheap only if you don't need to access it quickly. Glacier pricing is surprisingly complicated, and the actual cost can be much higher than $0.01 per GB-month if you don't read the fine print. The biggest gotcha is that you can only access 0.17% of the data you've stored in any given day without extra charges. So if you've stored 1000 GB, you can only access 1.7 GB per day for free. https://news.ycombinator.com/item?id=9184466 https://news.ycombinator.com/item?id=9184466
- skuhn 11y agoI think the math for Glacier actually gets a lot simpler if you look at it this way: Glacier is only cost effective if you never want to access that data ever again. There are actually use cases like this, when you will almost certainly never want to restore this data, but just in case you put it in Glacier. For anything that you expect to ever reasonably want to restore in a reasonable timeframe (like a MySQL database backup), it just doesn't make sense: it's too slow and too expensive.
- deleted 11y ago[deleted]
- deleted 11y ago[deleted]
- anarazel 11y agoIn many cases the recovery times won't be sufficient for that. If the bulk is archived or log data a couple days/weeks of recovery are OK, but of you need everything immediately...
- 11y ago
- hendry 11y agoIf I could have a git API to glacier I would be happy. I have tried various glacier clients and they all seem to suck. So I have trouble tracking exactly what I have stored there sanely. :( Unusable.
- phamilton 11y agoAre you using glacier specific clients? I just use an s3 client and configure the bucket to put everything in Glacier. For example, I have a cron job calling `s3cmd sync` for my photos on my iMac once a day.
- hendry 11y agoI'm aware of that easy API of getting stuff to glacier, but once it "expires" to glacier, how do you check what's actually there? It quickly becomes a nightmare for me. Hence I need git! http://natalian.org/2015/04/13/How_I_organise_my_media/ http://natalian.org/2015/04/13/How_I_organise_my_media/
- ilaksh 11y agoWhat about s3backer with a filesystem (https://code.google.com/p/s3backer/wiki/CreatingANewFilesystem https://code.google.com/p/s3backer/wiki/CreatingANewFilesyst...). Then you could use regular git. I think any active use of glacier is going to suck though. Just use that for archive.
- derobert 11y agogit-annex has a glacier remote which is supposed to work (I haven't personally used it yet). http://git-annex.branchable.com/special_remotes/glacier/ http://git-annex.branchable.com/special_remotes/glacier/
- venning 11y agoThis seems like a smart response to Google Cloud Nearline Storage. Slightly more expensive, but with the S3->IA->Glacier lifecycle mapping the ultimate costs may be lower with AWS, certainly the flexibility will be there. Amazon, staring at its mighty armory, goes on the hunt for a tiny chink to repair.
- obulpathi 11y agoYep, looks like AWS is catching up with Google in areas like these (Nearline, Container Engine).
- boulos 11y agoJust a clarification the difference in cost-per-byte stored is 25% (1.25 cents vs 1). The rates are all so small it's hard to see that but when you say $12.50 vs $10 per TiB/month, I think that makes it more "visible". As you get into the PiB range ($10k/month on Nearline) and then consider storage for a year you've got a difference of $30k/year/PiB. For individuals doing a small backup even 25% isn't huge in absolute numbers, but in the petabyte range it matters a lot. Disclaimer: I work on Compute Engine (and not GCS or Nearline).
- JoshTriplett 11y agoThese prices make the new Standard-IA storage significantly cheaper than the Reduced Redundancy Storage, even if you end up reading the data back. However, I find it interesting that in addition to the cost per GB to retrieve data, this new storage class also has a significantly higher per-request cost, too. Actually, it looks cheaper to upload an object as a different storage class and then transition it to Standard-IA, since PUT of IA costs $0.01 per 1000, but PUT of another class costs $0.005 per 1000, and the cost to transition another class to IA is $0.01 per 10000. It's a small difference ($0.04 per 10k objects), but if you store an obscene amount of data on S3, that seems like enough difference to matter.
- cperciva 11y agoI'm going to go out on a limb and guess that Standard-IA is basically Standard except with slow disks instead of fast. Slow disks have lower $/GB, but higher $/IOPS.
- JoshTriplett 11y agoSeems unlikely, given some of the other properties it has; more likely, Standard-IA has fewer copies of the data, and then a backup in Glacier, which explains why it has high durability but low availability.
- bscanlan 11y agoDoubtful. Almost certainly the same storage backend, but IA requests are more aggressively throttled, so there's more chance of requests being rejected. S3's capacity planning then requires less request processing infrastructure to serve IA, so it's cheaper to provide than regular S3.
- gaul 11y agoUpdated cross-provider comparison: http://gaul.org/object-store-comparison/ http://gaul.org/object-store-comparison/
- aembleton 11y agoThanks for that. After looking at that comparison, I came across https://www.runabove.com/index.xml https://www.runabove.com/index.xml who provide a good deal. Storage at 1c/GB/month, outgoing traffic at 1c/GB/month and no charge for incoming traffic. Data is replicated 3 times.
- hackerboos 11y agoRun Above is managed by OVH. Very cheap but not the most reliable, at least on their dedicated servers.
- finalight 11y agoso..does S3 objects becomes IA automatically if it's access infrequently, or do we have to do the implementation on our side?
- omribahumi 11y agoJust released a tool Yesterday to give an insight on your S3 bucket size: https://github.com/EverythingMe/ncdu-s3 https://github.com/EverythingMe/ncdu-s3
- gallamine 11y agoCan you say a few words about this. What's the advantage over doing a recursive ls with something like s3cmd?
- omribahumi 11y agoJust like the difference between ncdu and regular du. It lets you drill down into the directory hierarchy and spot the exact dir/file that consumes a lot of space.
- ck2 11y agoSpent the past 10 minutes just looking for a simple grid chart of pricing on S3 and gave up. and "infrequent access" is not on the price calculator wtf is it so complicated to compare
- PhantomGremlin 11y agoDid you see this? https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/ It has quite a lot of info. Edit: you must view that page with JS enabled, otherwise no prices are shown. Perhaps that was your problem?
- ck2 11y agoDidn't show in Firefox, had to open it in Chrome to see the table, thanks Firefox logs the dreaded "Error: InvalidStateError: A mutation operation was attempted on a database that did not allow mutations." which is known bad coding problem.
- devit 11y agoAren't these services still wildly overpriced? At $0.0125/GB/month, that means it costs $75 for 6TB per month. But a 6TB hard drive costs less than $300, which means that assuming the data is stored on 3 hard drives for redundancy, they break even in less than a year. However, hard drives seems to last at least 3-5 years on average, so this service seems to be priced at least 3-5 times as much as it costs to Amazon. And there is even a $0.01/GB charge for retrieval on top. There are other costs, but they should be relatively small at scale. Am I missing something? If not, why doesn't anybody compete with Amazon and provide more reasonable pricing?
- ju-st 11y agoGlacier isn't even using hard drives, so their per TB costs should be way lower (raw storage and infrastructure).
- kalleboo 11y ago> Glacier isn't even using hard drives That's still speculation right? Some have theorized they use offline harddisks
- acveilleux 11y agoI would expect offline HDDs as well, possibly combined with tapes for secondary copies to meet durability requirements.
- zzleeper 11y agoMy guess is off site backup is expensive. Also add the electricity costs, CPU etc, engineers, rent, etc. I That probably takes the margin from 80% of price to maybe 40%, in line with most retailers and whatnot
- discodave 11y agoGo look up one of James Hamilton's talks, he breaks down the cost of running datacenters/cloud. 'Servers' are ~50% of the cost, hard drives are of course less than 100% of server costs. So you need to at least double the cost of the hard drive. Tripling the cost of hard drives might be a good assumption for S3/Glacier because you could assume they are building servers where hard drives make up 2/3 the cost of the server.
- amatix 11y agoWould be nice to apply this to EBS snapshot storage too, that can add up mighty-quick.
- timdierks 11y agoHow does Amazon claime 11 9's of durability when the chance of an asteroid extinction event is roughly 1000x as high? This isn't a joke: I can't find any documentation on the risk model that lets them estimate 11 9's and what class of risks it includes.