8 ms·
Why Tarsnap doesn't use Glacier
- michaelt 14y agoI'm confused about the reason a block can't exist in both S3 and Glacier at the same time, if the deduplication code decides that block is needed in a new archive. Why couldn't you simply have a rule that each file is either S3 or Glacier, and S3 lists-of-blocks can only reference other S3 blocks, while Glacier lists of blocks can only reference other Glacier blocks? In the worst case, where every block was in both archives, this would only increase costs by 10% if Glacier costs a tenth what S3 costs.
- cperciva 14y agoWhat happens when you decide that you don't want that new archive to be in S3 any more and tell Tarsnap to migrate it over to Glacier?
- michaelt 14y agoTo S3 it's like you just deleted the file; to Glacier it's like you just created the file.
- cperciva 14y agoExcept that then we're trying to store two different blocks in Glacier with the same hash ID.
- alexfoo 14y agoFrom that I assume that if a block's hash matches something that's already in the archives then you retrieve the archive block(s) with the same hash ID in order to verify that it is exactly the same (byte-for-byte)? And this wouldn't be possible with Glacier as you can't just retrieve the block from storage to check there and then. Do you have any stats on the number of collisions you've seen?
- cperciva 14y agoAnother reason, which I decided deserved to be written up as a separate blog post, is the surprising nature of the "data retrieval" component of Glacier's pricing model: http://news.ycombinator.com/item?id=4475319 http://news.ycombinator.com/item?id=4475319
- tedunangst 14y agoI was thinking about one time, single shot backups, but encrypted wtih tarsnap. No dedup. After all, storage is cheap and this is supposed to be zuper worst case scenario stuff, so maybe I don't want to go to the trouble of reconstructing the backup. I want to know it's there. But my interest is only theoretical.
- aristidb 14y agoIf you have a command line tool to access Glacier, I think what you can do is make an archive (with tar), encrypt it with gpg, and just use the command line tool to upload the encrypted archive. So you don't need tarsnap for that, but if you want fancy deduplicating incremental encrypted backup, that's where tarsnap shines.
- tedunangst 14y agoYes, but I'm already using tarsnap. Running it with "--backup-to-glacier" is easier than rolling my own. I don't want two passwords, two super duper secret keys, two online services I have to pay, ....
- cperciva 14y agoSounds like setting up a Glaciated Tarsnap machine would do what you want, then? Or maybe I'm misunderstanding what you're saying.
- tedunangst 14y agoMore or less. But I may also want to use the same machine key to perform smaller incremental backups (smaller data set), and I definitely don't want to have to print out two different keys to store in my safe. Just thinking here.
- ChuckMcM 14y agoWhile I think this is a completely reasonable thing (not storing tarsnap data in Glacier), I'm not sure if the reasoning holds beyond, "For the way we designed tarsnap it doesn't make sense." When you look at how GFS is implemented or Bigtable, or Blekko's NoSQL data store, there are layers, with the meta data as a pretty separable layer from the data. In all of the cases for these large stores that separation was put in to facilitate putting the meta data into lower latency storage than the 'bulk' data which facilitates fast access. As cperciva relates, tarsnap spends a lot of time de-duplicating data (or writing only one copy of a block with identical contents) which is great for reducing your overall storage footprint which lowers your costs. But if the cost of storage is much much smaller than the retrieval cost, then your design methodology would be different. So I would not be surprised if there was a way to build a function equivalent product to tarsnap that had lower storage costs if the bulk data was in glacier and the index data was in S3, or if the index data was designed such that a document recovery was exactly two retrievals (a catalog, and then the data). And of course such a system would not de-duplicate as that would result in potentially more retrievals. It seems that if the price delta ratio between Glacier and S3 was higher than the de-dupe ratio, then Glacier would 'win' with no de-dupe. Else de-dupe would still win. Thoughts?
- pdovy 14y agoBased on the article it seems like the motivation for the deduplication is as much bandwidth conservation as reduction of storage costs. What you're proposing would make sense if storage costs are your only concern, but it shuts out bandwidth limited users.
- ChuckMcM 14y agoAn excellent point. So capturing the bandwidth savings and the storage cost savings suggests an interesting mid-tier between longer term storage and shorter term.
- thisishugo 14y agoThere's no bandwidth cost associated with transferring data TO Tarsnap (that is, data transfer into EC2/S3 is free of charge), and if you have to get a file out the cost is the same whether you store files or "blocks" (I'm assuming the block disassembly is done on EC2). So our hypothetical Tarsnap alternative could store file metadata in S3, recently modified files in S3 also (quicker retrieval) and push older files into Glacier (can be retrieved when needed, cheaper storage if (as is likely) never accessed). It will take marginally more time and bandwidth to transfer a file every time there is a change than to only transfer changed blocks (i.e. parts of files that have changed) but for 90% of users I would bet the (significant?) decrease in long-term storage cost would make that worthwhile.
- danielweber 14y agoI like articles like this. 1. Explains what the new whiz-bang technology is. 2. Explains what Tarsnap is. 3. Provides technical explanation that lists both the upsides of the new technology along with the downsides.
- cperciva 14y agoThanks! I like blog posts like this too -- they appeal to my didactic instincts.
- mapgrep 14y agoAgreed. Too often we hear of new tech via "this is the best thing ever" commentary, followed a few months later by the inevitable "actually that thing sucks" commentary. Much better is "here are the trade offs, based on actual usage" commentary like this. I wish there was more stuff like this during the first wave of NoSQL mania.
- nchuhoai 14y agoDoes anyone know of products that already incoorperate Glacier. Basically, I am looking for a solution to back up my photo library. It's one big file with a "Write Once, Read hopefully never" Access Policy. I guess I could go and push it to Glacier by myself, but I'd rather have a library/product do it for me
- ghshephard 14y agoGlacier was rolled out on August 21st - I don't think anyone has rolled out code to support it in a consumer friendly fashion - but drop a line to http://www.haystacksoftware.com/arq/ http://www.haystacksoftware.com/arq/ and encourage them to give you the ability to target Glacier for your Archives, and I bet you'll see something fairly soon.
- Yrlec 14y agoMy startup Degoo (http://degoo.com http://degoo.com) is working on it. We're building a P2P Backup system were we will be using Glacier as a fall-back whenever we don't have enough reliable peers. We're planning on launching our private beta in the coming months.
- ocharles 14y agoGreat write up, I've been curious about it, but I'm not particularly bothered. I think what I'm going to end up doing is to basically tar ~/media, pipe it through GPG, and pipe it through some upload-to-glacier script. Maybe automate that once a month or something and then have tarsnap for much smaller/rapidly changing stuff. As it stands now I have 100gb in ~/media that's costing me about $1/day, so it'd be nice to reduce that.
- RexRollman 14y agoCperciva, Could a similar system to Tarsnap exist for Windows? I'm not asking you to implement it, I'm just curious.
- magic_haze 14y agoTarsnap runs without any problems under cygwin/windows right now. It is very easy to set up: the Tarsnap downloads page lists all the dependencies - you just need to make sure to include them in the Cygwin setup and then run his makefile. Frankly, his setup instructions are far more easier to follow than certain other gui-based installs that look shiny but induce far more stress trying to guess how to opt-out of the complementary crapware (e.g., Skype, Adobe Reader, utorrent...) Also, as he notes in his install page, it _definitely_ is worth checking out the source code: one of the cleanest C source I've ever seen, and very educational.
- RexRollman 14y agoThank you for the info!!
- unreal37 14y agoI think the best use-case for Glacier is large files that you would absolutely hate to lose. The thing that people cry over when their laptops get stolen. For home use, all the family photos and videos. An archive of your emails (outlook.pst) because a lot of important data is stored in there. All your taxes and accounting data from years past. Bulky stuff that takes up space on your home system, but isn't used daily. In business, many companies use "Iron Mountain" to archive their paperwork. Old invoices, reports, things that were important in the past and may someday be needed. That's what Glacier is for. Glacier is archiving, not backup. You might want to take advantage of cheap storage by keeping your backups there, but that's a different issue.
- fusiongyro 14y agoI don't get the impression Glacier is right for any home user. The pricing scheme is just too weird, and there are weird delays and time limits involved. I believe you have 24 hours to retrieve your data before incurring more fees when you do a retrieval from Glacier--but if I store a backup of 150 GB of photos and music and fetch the whole thing, I'm not going to be able to get that from Amazon to my home computer in one day. Avoiding that problem will necessitate other expenditures. In order to avoid the retrieval costs you'll have to limit retrievals to a small fraction of what you have stored. In my example, if I lose all my photos and music I'm going to want to restore the whole thing. Ignoring the transfer problem above, you're looking at paying for transferring 95% of the archive, 142.5 GB. I find Glacier's pricing model so difficult to comprehend I couldn't even guess at what that would cost, but Colin's math shows that what looks like it should cost $0.02 (retrieving 2 GB once at $0.01 per GB) winds up costing $3.60 (peak rate, percentage of archive fetched, etc.), so I wouldn't hold out a lot of hope for our use case of fetching the entire 150 GB archive. As you say, this is certainly something businesses that need to archive lots of stuff may be able to use (if they can navigate the pricing structure) I just don't see a home user getting anything out of it but frustration and a confusing bill.
- warp 14y agoIndeed the pricing is difficult to estimate. I followed along with [1] in a spreadsheet to get a feel for it for my usage. In my case I am looking at making a full backup of a 1TB usb disk. Doing a full restore on a 20Mbit/s (= 9 GB/hour) would probably take about 5 days. In that case I'll be paying $65 in retrieval fees for the full restore. (I'm hoping that includes bandwidth costs, that isn't clear to me from the FAQ). $65 for a full restore seems reasonable for something I expect to never need to do. [1] http://aws.amazon.com/glacier/faqs/#How_will_I_be_charged_when_retrieving_large_amounts_of_data_from_Amazon_Glacier http://aws.amazon.com/glacier/faqs/#How_will_I_be_charged_wh...
- cynicalkane 14y agoI'd bet that this use case is pretty common: I have only a small amount of data (a few hundred MB in my case) that I deem day-to-day critical, but hundreds of GB that I don't want to lose. My backup solution is effective but primitive: parchives on two offline hard drives, one of which is at my parents' house seven hours away. Backing up takes little effort but it's easy to forget, especially at the "remote site". It's really annoying that there's lots of easy backup solutions for online data, but nothing for cheap backup of large amounts of data that can afford to be offline for a while. Glacier is the perfect solution but I dread having to figure out whatever I have to do--divide things into a 180-part archive and download it over a month?--to get data back. The first glaciated backup solution on the market that I think I can trust will get my dollars almost immediately.
- j_s 14y agoSome people might not be aware that CrashPlan supports backing up between your own PCs (eg. to your parents) for free: http://www.crashplan.com/consumer/crashplan.html http://www.crashplan.com/consumer/crashplan.html
- genwin 14y agoYour remote site method is my method too. I'm not worried about city-wide catastrophes, so I keep the spare drives closer for easier updating.
- veyron 14y agoI've got Terabytes of data that need to be archived and possibly recalled later (trading logs and market data preserved in case of an audit). I'm not at the scale of Glacier. If TarSnap had a readonly vault (files in said vault would never change) then it would be able to distinguish the files and split between the Glacier and S3 offerings.
- ghshephard 14y agoDrop a note to http://www.haystacksoftware.com/arq/ http://www.haystacksoftware.com/arq/ - I'll wager they will have an option to create "Glaciered Archives" of folders within a few months. The developer there has already expressed some interest, and suggested that Arq would lend itself to pointing at a folder and storing it off to a Glacier Archive (with all the caveats around 4 Hour delay, 24 Hour of Availability, Costs associated with restores, etc...)
- sreitshamer 14y ago[I'm the developer] I'm hoping storing the metadata in S3 and the actual file data in Glacier would work well. I'm still looking into it. [edited for clarity]
- ghshephard 14y agoThat's awesome news. Just being able to take a folder on my Drive, and say, "Archive this 5 Gigabytes of photos for the next 50 years" - and know it will cost me around $30 (and, as time goes on, likely less as the storage drops) and I won't have to worry (much) about it, will be a big win - even if you don't come up with an easy way of integrating with the normal S3 backups - I bet a lot of people will love that feature.