12 ms·
Ice cold archive storage
- tpetry 8y agoThe interesting part it's cheaper than AWS Glacier ($4 per TB per month) and slightly more expansive than AWS Glacier Deep Archive ($0.99 per TB per month) but the data is available immediately and not in hours like glacier where you have to pay a hefty premium for faster access to the data.
- duxup 8y agoI use Glacier as a sort of backup of my backups .... and was thinking about Glacier Deep, but this is tempting too.
- lucb1e 8y agoIn case you're still thinking about options, I would be fine to host a variant of this for you: https://old.reddit.com/r/DataHoarder/comments/7rjcdn/home_made_non_gmo_cruelty_free_offsite_backup/ https://old.reddit.com/r/DataHoarder/comments/7rjcdn/home_ma...
- trpc 8y agooff topic question: what is the lowest price for their Compute Engine VMs (EC2 alternative) currently? EDIT: WTF is wrong with this website? probably 45+ yo people downvoting every fucking comment they don't like? WTF
- deleted 8y ago[deleted]
- tpetry 8y agoThey downvoted you because it would be really easy for you to look the prices up by yourself. Will take only a few seconds: * Go to https://cloud.google.com/compute/ https://cloud.google.com/compute/ * Scroll down to the pricing information * Click on the link for the price list
- mercwear 8y agoWhat does the age of someone have to do with them downvoting you? You asked a question that you could have easily answered yourself.
- scurvy 8y agoWhile one major use of something like this would be backups, how does one handle these backup sets with respect to GDPR requests? The window to respond is 30 days, so keeping backups longer than say 25 days seems cumbersome. You would need hot access to the sets to load them up and delete the data.
- tpetry 8y agoEncrypt the backups and store the encryption key in a normal non-archival bucket.
- jsnell 8y agoEncrypt backup data with a per-user key, keep the keys only in hot storage, delete the key when a user is deleted.
- anoncake 8y agoWont that make the backups useless in case of a data loss (i.e. always)?
- jsnell 8y agoYou don't keep a single copy of each key, but store enough redundant copies to get the proper number of nines. Preferably that's redundant geographically, in terms of storage technology, and in write frequency. The important part is just that the keys don't end up in long term cold storage. Either it's only retained for a short period (e.g. tape backups that get rotated after two weeks), or it supports live deletion.
- shittyadmin 8y agoInteresting, unlike Glacier this is significantly cheaper than Backblaze B2, meaning I might have to reconsider how I do my backups again. Any good backup tools supporting this type of service? I rely on Restic at the moment which seems to need fast read access to data, but their incremental snapshotting is great. It'd be ideal if I could find something like that supporting these "cold storage" solutions.
- Aeolun 8y agoI need more information than this post provides before switching archival solutions. Actually, since it’s google I likely wouldn’t consider them regardless.
- jamesjamesm 8y agoI've never tried it, but I know https://www.arqbackup.com/ https://www.arqbackup.com/ supports Google Cloud.
- tedmiston 8y agoThe concept, idea, and flexibility of Arq is great, ideal even. The amount of control is nice. I wish it were open source. The actual product is pretty painful when you need to do a recovery, especially if you don't know where the file lived on disk. I haven't tried newer Arq Cloud Backup destination to see if it improves the search experience. That said my experience is from more than a year ago and I would try it again if they were able to bring their search on par with current consumer backup offerings.
- mmahemoff 8y agoAWS's new Glacier Deep is actually cheaper than Google's Ice Cold, $1/TB/month. https://aws.amazon.com/about-aws/whats-new/2019/03/S3-glacier-deep-archive/ https://aws.amazon.com/about-aws/whats-new/2019/03/S3-glacie...
- deleted 8y ago[deleted]
- patrickg_zill 8y agoWhat are the transfer costs for storage and retrieval?
- tsukurimashou 8y agoI was also looking for that, the only piece of info about that was: Unlike tape and other glacially slow equivalents, we have taken an approach that eliminates the need for a separate retrieval process and provides immediate, low-latency access to your content. Access and management are performed via the same consistent set of APIs used by our other storage classes, with full integration into object lifecycle management so that you can tier cold objects down to optimize your total cost of ownership.
- tpetry 8y agoIt seems to have the same pricing like the other storage classes: No fees for accessing the files within the same region and the typical bandwidth fees if the backups will be downloaded to somewhere else.
- couterSpell 8y agoIntuitively there is some cost for retrieval. Otherwise you'd just use the cold storage to store everything.
- deleted 8y ago[deleted]
- tpetry 8y agoLike every cloud storage file access costs money, so the operation to access it. But its so minimal its basically non existent for a backup solution.
- votepaunchy 8y agoNearline and Coldline have a per-byte retrieval cost in addition to the increased-but-still-low cost per operation. https://cloud.google.com/storage/pricing https://cloud.google.com/storage/pricing
- idlewords 8y agoWhat is the meaning of the claim about "99.999999999% annual durability"? Does that mean one chance in 100B of an object being unretrievable?
- ajay-d 8y agoBackblaze wrote a good piece on that https://www.backblaze.com/blog/cloud-storage-durability/ https://www.backblaze.com/blog/cloud-storage-durability/
- idlewords 8y agoThank you for that link! It's very interesting to me that all the calculations laid out here assume independent failure events.
- ahelwer 8y agoIt is universal practice within cloud service providers to span redundancies across "fault domains" - basically things which could make failures correlated, like being in the same machine/power strip/datacenter/geo region. If you assume your fault domain analysis is good, then your failures should be independent. Many global outages are the result of a previously-unidentified fault domain, like the Azure certificate issues. Of course past a certain point it becomes unimportant - who cares about your data if an asteroid takes out every datacenter on Earth at once?
- gnulinux 8y agoNothing in engineering is 100%. As far as engineering goes 11 9s is pretty much as good as you can get. For comparison, AWS S3 and Glacier are 11 9s durability too.
- remus 8y agoIt's worth bearing in mind the difference between durability and availability. Durability is roughly the chance of losing your data over a given time span (in this case a year), whereas availability is about how reliably you can access the data (and is almost certainly a lot lower than 11 9s). A service can be very durable but have very poor availability.
- penagwin 8y agoDoes anybody know what the retrieval fees will likely look like? I've been wary of most of the "cloud archival" solutions because while they're cheap to put data into, they seem charge you a billion dollars to actually retrieve it.
- ocdtrekkie 8y agoFWIW, this is still an ideal model for backup storage: If your more regular backup model is robust and your network is well-secured, you'll never need retrieval. And if you need it, you need it, and it's justifiable to spend big to save your business.
- penagwin 8y agoI'm mostly just evaluating it for personal/hobby backups (few terabytes), I know business will for sure look at it differently.
- lucb1e 8y agoSame here. As a company you can go "we need this to save our asses, I don't care if it costs $50k in a 4 person company", but personally I kind of do care about the cost for retrieval... I've been comparing cloud storage prices to hard drive prices for years now. My first thought when seeing the storage prices was "huh, that might actually be worth it", but depending on the retrieval costs, you might still want to roll your own no matter the storage costs. For private use, I am (was?) planning a variant of this as soon as I am finished doing a server migration: https://old.reddit.com/r/DataHoarder/comments/7rjcdn/home_made_non_gmo_cruelty_free_offsite_backup/ https://old.reddit.com/r/DataHoarder/comments/7rjcdn/home_ma...
- z3t4 8y agoAlso compare with ftp and storage vps. Cheapest ive seen is 1$ per TB /month.
- 8y ago
- fiatjaf 8y agoGoogle offers some interesting services, but their API is always so awfully complicated and cumbersome that I've given up entirely trying to use anything.
- jpatokal 8y agoIf you can use cp to move files around, you can use gsutil to do the same for GCS. https://cloud.google.com/storage/docs/gsutil https://cloud.google.com/storage/docs/gsutil
- yread 8y ago> Unlike tape and other glacially slow equivalent shots fired I like when multinational corporations with revenues the size of midsized countries engage in some childish puns
- bzillins 8y agoThe title seems like a reference to the 2003 Outkast song Hey Ya! https://www.youtube.com/watch?v=PWgvGjAhvIw https://www.youtube.com/watch?v=PWgvGjAhvIw Alright alright alright!
- remus 8y agoAnyone care to speculate on the technology that allows them to offer the fast retrieval times and low cost per GB?
- puzzle 8y ago"Fast" is relative here. It's fast compared to Glacier and others, but it's going to be slower than the more expensive tiers. You might not need to speculate much about how it works, it's probably implemented as described by Google themselves in slides 22ff. here: http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Keynote.pdf http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Ke... (In a nutshell, pack some hot data with a lot of cold data on many large drives, then put a Flash-based cache in front of them to get long tail performance predictability back.)
- remus 8y agoThanks for the link! Interesting stuff.
- puzzle 8y agoThere was also a talk about the low level storage service and the performance isolation work that allows it to mix batch and latency-sensitive traffic on the same drive, but it doesn't seem to have been recorded: http://www.pdl.cmu.edu/SDI/2012/101112.html http://www.pdl.cmu.edu/SDI/2012/101112.html Gory details are in the patents, 9781054, 9262093 and 8612990, which I'm not linking directly, because your lawyers might not approve. There's even a follow up, 10257111. It's so new, from two days ago, that Google Patents can't find it, while Justia can.
- whoisjuan 8y agoYou have to keep the data in that class for a certain period of time. That's the drawback. You can access the data at that price as long as you keep it there for a long time. So I suspect this is not a fully cold storage. That's why they can retrieve the data faster. Seems more like an economics hack (Longer commitment to keep the data, allows them to buy and operate the storage hardware/software at a cost that can be amortized against those commitments)
- imagetic 8y agoJust don't think of it as something you'll ever want to restore unless the building burns down and you've lost everything. Glaciers restore costs had a lot of fees in my one experience. We could have bought several RAID units for the price of a fast restore. If you asked for it back over a long period of time, the price dropped dramatically.
- OrgNet 8y agoIt's a pretty good price but assuming you are storing 8TB and you get your own drive, the drive would pay for itself in about 14 months... so you would basically get the next 4 years for free if you are willing to manage it...
- dwild 8y agoWill that storage have "11 9’s annual durability" and stored in multiple location? Let say that you only need to write to it once, have 2 secure location available for free, that would still means that you need 2 of them which would pay for itself in 28 months then. Sure it's "cheaper" but it's far from being as good and the price difference isn't that big.
- votepaunchy 8y agoOptimally running your own drives also assumes that you can fill the drives ... and the next byte doubles your cost.
- dredmorbius 8y agoContrast: Petabox, from the Internet Archive. https://archive.org/web/petabox.php https://archive.org/web/petabox.php Density: 1.4 PetaBytes / rack Power consumption: 3 KW / PetaByte No Air Conditioning, instead use excess heat to help heat the building. Raw Numbers as of August 2014: 4 data centers, 550 nodes, 20,000 spinning disks Wayback Machine: 9.6 PetaBytes Books/Music/Video Collections: 9.8 PetaBytes Unique data: 18.5 PetaBytes Total used storage: 50 PetaBytes Costs are $2/GB, lifetime, I believe. https://help.archive.org/hc/en-us/articles/360014755952-Archive-org-Information https://help.archive.org/hc/en-us/articles/360014755952-Arch...
- JoblessWonder 8y agoGoogle has burned people so many times with shuttering products with little to no warning I'd be hesitant to trust them with my long term data storage.
- yjftsjthsd-h 8y agoEh... for consumer stuff, sure, and perhaps even for new/experimental GCP features. But this is storage, a core function, on GCP, an enterprise service with actual contracts and SLAs attached.