9 ms·

Duplicity + S3: Easy, cheap, encrypted, automated full-disk backups

171 points by tortilla 13y ago

aroch 13y ago
I use Arq[1] on my macs and it does the same thing but handles all the "complicated" bits that I'd rather be programmatically dealt with. I have 6 macs, about 2TB of backups in Glacier and a ~20GB "system image" backup in hot-storage on S3. [1] http://www.haystacksoftware.com/arq/ http://www.haystacksoftware.com/arq/
- makmanalp 13y ago
  How do you like glacier? It seems it's fairly cheap until you want your data back. Any insights?
  - aidos 13y ago
    Personally I've factored that in to my cost of recovery. It's going to cost me about £120 to restore my data over the course of a week. It depends on your use case but for me - in the event that glacier is the last point I can get my data from, £120 is a small price to pay to get it back. I used backblaze before and it was about the same to get them to post out a hdd (which was an awesome service on the one occasion I had to use it).
    - toomuchtodo 13y ago
      It's too bad Amazon doesn't offer a discounted rate to restore from Glacier if you pay upfront.
      - zz1 13y ago
        Sorry to chase you between threads, but yes, I'd like to get in touch for the Getty files! Care to share contact info?
        toomuchtodo 13y ago
        m8r-fhf8r1@mailinator.com (apologies, would prefer not to share my public email address here) I'll reply back from my personal email account.
    - _red 13y ago
      In addition, Glacier should be thought of as "offline" storage , like a tape rotation scheme. When thought of that way, the cost for restoration is non-existent. Its not for "I may need this file next week" (thats what your local NAS / Time Machine is for...Glacier is for "what happens if this place burns down". At that point a week and £120 is not high on the list of your concerns...
  - aroch 13y ago
    I like it quite a bit, Arq is my "backup of last resort" and what I use for provision new macs if they're not with me. I have daily/weekly/monthly local backups and semi-local, bi-weekly backups (stored offsite or in a building with a different mac). Setting up a new Mac remotely costs about $30 for the 60GB provisioning profile to be downloaded and restored (maybe more or less depending on the internet connection at the place). I pay a few hundred yearly for backups that I have no doubt will be there if I need them.
- aidos 13y ago
  I also use Arq and recommend it to everyone on OS X. It has a sane transparent format (modelled on git), it can store to Glacier and it doesn't periodically explode like Back Blaze used to. Interested to hear more about your "system image". How do you do it and what do you store in it? Documents / preferences / Library etc? How would you use it in a recovery situation?
  - aroch 13y ago
    It's a separate backup vault in Arq that's maintained by clean OS install + my essential documents, dotfiles, and an encrypted PII volume with scans of passport/SSC/Drivers license/Birth certificate. It's all on a 60GB SSD that's booted every week to update to the latest versions of everything and then to be Arq backed up. Restoration is downloading the vault to a new SSD and shoving it in a computer. The "root" user is included in the backup as an encrypted volume. My setup's a little strange in that the OS disks on my computers are filevaulted and each user's homedir is also in an encrypted volume only they have the password too that's mounted to the correct /Users/folder with fstab. (It'd probably be helpful to note that I have a hackintosh this is done on with 10 internal bays, in which one resides this SSD with a clone of this SSD elsewhere)
    - sreitshamer 13y ago
      Could you write that up in a blog post please? I'd love to read how that works. (I'm the author of Arq)
      - aroch 13y ago
        I could probably whack something together, but what specifically were you wondering about?
        sreitshamer 13y ago
        I'm wondering about the whole process. It sounds like an external drive that you periodically copy your documents, dot files etc to, and then let Arq back up? Or is it something else? (You talk about "booting" the disk -- do you mean booting the computer from that disk, or just mounting it?) Also I didn't understand the "root" user part. And is this a multi-user computer? You mentioned something about "each user".
- apaprocki 13y ago
  Another happy Arq user here -- I just use it for Glacier backup, no need for hot S3 storage.
- OWaz 13y ago
  Has anyone been able to accurately figure out how much Glacier will costs once storage is retrieved from it?
  - aroch 13y ago
    Here's my last bill: $0.010 per GB / month - Storage 2112.146 GB-Mo 21.12 $0.050 per 1,000 Requests 40,447 Requests 2.02 My last "restoration" was actually setting up a new iMac at a remote site involved downloading ~60GB over ~20hour and cost me about $30
  - mjn 13y ago
    Above the free retrieval quota (5% of your storage per month), pricing is based entirely on your peak hourly retrieval rate for the month, billed at $7.20 x [peak rate measured in GB/hr]. So for example, if you retrieve 100 GB in one hour, that's $720. If you retrieve 10 TB at the rate of 100 GB/hr for 100 hours, that's also $720. Translated into bandwidth (assuming you're pulling at this rate for at least an hour), it's about $3 per Mbps. So if you limit your retrieval to 10 Mbps, you'll pay $30, regardless of how much data you retrieve. This makes it fairly easy to cap your expenditure, if you have a client that can stagger its requests appropriately (you need to throttle the Glacier retrieval requests, possibly using Glacier's range requests if you have very large files, not throttle at the network level).
- ivanhoe 13y ago
  Arq is simple and works great, although it seems that a half of my S3 bills are for the thousands of requests that Arq makes back and forth. Simpler upload & forget backup model probably could save a few bucks...
  - aroch 13y ago
    I make, on average, 40K requests a month which is like $2 while my storage costs are ~$22. At the low end you'll be paying what seems like a lot for requests but as things scale your requests costs become a much smaller percentage. You an cut down on requests by moving to daily backups (what I do) and narrowing down what you're backing up. Backing up your applications folders will take a ton of requests (Some apps have upwards of 1000 assets whose bless values may change if you open the application and this trigger Arq)
- aesptux 13y ago
  Do you recommend Glacier over Crashplan? I'm thinking about switching
  - ye 13y ago
    They are very different. Glacier takes hours just to get the listing of files, or start the file download. Their pricing is very different. Crashplan charges per computer. Glacier charges per GB of storage and per GB of transfer out.
    - aesptux 13y ago
      In what cases would be better to use Glacier? I don't have too many GB, ~30GB.
      - TkTech 13y ago
        Glacier is for when you want to store essentially never-changing backups that you expect to very, very rarely access. You can store a couple GB or petabytes. Think an archive of family videos and photos spanning 30 years, or all the tax documents for a large enterprise.
      - aroch 13y ago
        A 30GB backup, that's updated nightly, on AWS Glacier will cost somewhere in the neighborhood of $1 a month. A 300GB backup somewhere around $5 on AWS.
      - ye 13y ago
        Glacier is cheaper for small backups that you almost never touch.
- dewiz 13y ago
  Is there something like Arq compatible with both OSX and Windows ? Features I'm interested in: - must use S3 or similar, as long as it's private and "unlimited" - must support multiple clients, even though I don't need real time sync - should encrypt on the client without need of a separate sw I've been using Dropbox + CloudFogger so far, but dropbox doesn't scale anymore with the quantity of data I have, mostly due to the number of files (and I don't like depending on two different sw for one task).
  - aroch 13y ago
    The only two I have used are git-annex [1] and duplicati [2]. I much prefer git-annex but the windows version isn't so stable. [1] http://git-annex.branchable.com/ http://git-annex.branchable.com/ [2] http://www.duplicati.com/ http://www.duplicati.com/
zobzu 13y ago
duplicity + anything = easy,cheap,encrypted, automated full-disk/incremental/etc backups. duplicity's simple yet pretty awesome.
bkmartin 13y ago
How would one do something similarly easy and economically efficient for a Windows platform? We've looked at different services but they are always so expensive. I would love a piece of software that we can place on each server and let it do its thing.... of course there is always SQL Server to contend with, again... expensive solutions exist.
- uptown 13y ago
  Check out Crashplan. You can use their software for incremental backups to the destination of your choice at no cost, and it's multi-platform provided you install Java.
  - pwenzel 13y ago
    Another Crashplan fan here. You can also install the Crashplan client on headless Linux-based NAS devices, so long as they can run Java (such as ReadyNAS).
- aidos 13y ago
  Personally I'd probably use something to sync the lot over to a linux box and do the rest of the work from there. Depends on how much data you need to sync / how many servers are involved. Could be a pain to restore too (depending on the type of data you're backing up) so you would want to weigh all of that up too. With the Sql Server backups you could run periodic backups from Sql Server itself and have those backed up - depending on the data this could be really suboptimal but I think sql server allows incremental backups in the backup set, doesn't it? Been a number of years since I've had to deal with it. You can use some sort of rsync alternative to get the content from the windows boxes to the linux machine (cwrsync is recommended - though I've never used it). http://stackoverflow.com/questions/528298/rsync-for-windows https://www.itefix.no/i2/cwrsync
- kevinoid 13y ago
  I feel your pain. I actually ported Duplicity to Windows about 3 years ago for a client, but the code was never merged (or commented on).[1] In the end, that client decided that it was easier to use a commercial package, given their typical use case. Tried Carbonite (didn't work, support unhelpful) but settled on JungleDisk. YMMV. 1. https://code.launchpad.net/~kevinoid/duplicity/windows-port https://code.launchpad.net/~kevinoid/duplicity/windows-port
jewel 13y ago
The issue I have with duplicity is that every N backups have to be a full backup, so a large backup (such as my photo collection) becomes impossible.
- orthecreedence 13y ago
  Is this a technical limitation? I'm not seeing anything in the docs that forces you to do a full backup after N amount of backups (maybe I missed this).
  - brucehart 13y ago
    There is no technical limitation. You can do as many incremental backups as you want and are not forced to do a full backup after the initial one. As a practical matter though, I've found it's a good idea to occasionally do a full backup because the metadata can get out of sync (especially if something goes wrong like you lose your connection during an incremental backup).
  - Spooky23 13y ago
    You can do many many incrementals, but the restoration process, especially from a remote source is incredibly slow.
    - zobzu 13y ago
      its not that bad. i do it from a shiva plug over a 10mbit DSL link for a few dozen gigs regularly. its as fast as the DSL will go.
  - ef4 13y ago
    In addition to the other reasons given, there's a reliability problem in relying on an ever-growing chain of incremental backups: you can only recover your latest state if every increment in the chain is uncorrupted. I would much rather use an incremental backup that records the diffs in reverse, so that the latest state is stored explicitly, and the earlier versions are recovered through a chain of reverse diffs. Duplicity's security goals make it hard to implement this kind of reverse diff structure.
PlaneSploit 13y ago
Duplicity has consistently failed to restore for me in the form of deja-dup on Ubuntu.
- tenfingers 13y ago
  Restoring from duplicity takes a proportional amount of time related to the backup chain length, as archives are stored as forward-deltas. Nothing against forward deltas, but just consider that: - restoring time increases with increment count - risk of corruption of a single archives will make all further increments worthless. These are very important things to consider when doing a backup. Currently, you have to limit the chain length by performing a full backup every N increments (with N being as short as possible), which defeats the purpose of efficient increments. I have requested the ability to specify manually the ancestor of the increments (by time or count), so that one could implement a non-linear hierarchy with multiple levels like one normally does with tar/timestamps, but the request was dubbed as "unnecessary given the delta efficiency" (despite the fact that efficiency is just one variable). Having a 3 level backup (daily, weekly, monthly) would make duplicity much more space efficient, reduce the number of full backups needed, and shorten the chains to the level that would make restores of "last year" actually _possible_. I sent several patches to fix duplicity behavior with files larger than 1gb (by limiting block counts), which got integrated, but are still a far cry to make duplicity work decently as a whole-system backup&restore solution. It's just too slow. And like you said, several bugs afflicted duplicity in the past that would make restore fail in many circumstances. I also debugged my share of issues, which led me to think that very few people actually tried to restore from arbitrary increments with duplicity and/or used them to archive a large system. Many got fixed, but I won't consider duplicity again until I can control the incremental ancestor and reduce the chain lengths (and it's silly to think that "rdiffdir", distributed with duplicity, would allow for that easily). Nowdays I use rdiff-backup, and use a second script to tar/encrypt the deltas after the backup has completed. I'm keeping an eye on "bup" (https://github.com/bup/bup https://github.com/bup/bup), but I cannot backup "forever", thus without the ability of purging old versions it's only useful in a limited set of cases.
  - rlpb 13y ago
kbar13 13y ago
duplicity + s4[0], because s4 is greater than s3. [0] https://leastauthority.com/ https://leastauthority.com/
deleted 13y ago
[deleted]
jd 13y ago
Quick correction: it's not true that rdiff-backup has to be installed on the remote server. Rdiff-backup works brilliantly with a dumb target for storage and that's how we use it in production. Our servers periodically run rdiff-backup of all important data to a /backup partition on the same server. Then this /backup partition -- with the backup version history and metadata -- is rsynced to various dumb backup storage locations. We've tried many backup solutions, and rdiff-backup is by far the fastest and most robust backup program we know.
- ars 13y ago
  > is rsynced to various dumb backup storage locations. Rsync requires a program on the other end. True, it's a more commonly installed program than rdiff-backup but it's not a dump backup.
  - andreasvc 13y ago
    The difference is that rdiff-backup requires the exact same version on both ends.
- jackalope 13y ago
  You might want to reconsider this approach. I do something similar, but my remote backup also uses rdiff-backup. While it's awfully tempting to rsync the local backup to a remote location, it puts your entire backup at risk because rsync doesn't care about maintaining the state that rdiff-backup depends on. If your local disk is failing with transient errors, or your local /backup mount point disappears, the next rsync could mirror the corruption, leaving you without any backup at all. It's not worth the time savings and after the initial full backup the network overhead is negligible, even over ssh.
dotemacs 13y ago
What's the benefit of Duplicity over Tarsnap http://www.tarsnap.com/ http://www.tarsnap.com/ ? Thanks
- reidrac 13y ago
  Duplicity is an open source project that can be used with different types of storage (I use them with OpenStack Object Storage instead of S3). You can host your own backups if you want (supports ftp, sftp, ... check the docs: http://duplicity.nongnu.org/duplicity.1.html http://duplicity.nongnu.org/duplicity.1.html), but Tarsnap is an hosted solution.
- aquark 13y ago
  I played a bit with both and they both worked well. Tarsnap: easier to set up and use Duplicity: fully open source, and I think the restore speed was significantly faster ... but I've not properly benchmarked it apples-to-apples. If you are backing up a lot of frequently changing files (eg. database backups) then tarsnap's bandwidth charges (probably caused by the underlying EC2 charges) can swamp the storage cost. Running duplicity against rsync.net avoids this.
  - gingerlime 13y ago
    restore speed was significantly faster with duplicity? I haven't used tarsnap yet, but one of my main gripes with duplicity is the rather slow restore speed. Is tarsnap even slower?
    - tedunangst 13y ago
      Depends. tarsnap restore times should be constant regardless of which backup you pick, since it's just downloading the relevant blocks. duplicity does full backup + diffs, so if your data changes quite a bit you can end up downloading a lot of data that doesn't land on disk. (i believe that's right, not verified.)
sahaskatta 13y ago
I've been using BackBlaze. I find it both cost effective and time saving. Here's why: Price: It costs only $3.96 a month if you pay for two years up front. ($4.17 a month if you pay for only 1 year up front.) They offer unlimited storage of everything on internal and external HDDs. 30 days of file revisions too. I think I have close to 1TB of data backed up with their service. (Using Duplicity w/ S3 would cost me $90 a MONTH in comparison.) Time: I've always been using Windows File History to backup onto a external USB HDD, but I wanted an off-site backup. I initially started backing up to S3 myself and even dabbled with Glacier. However, maintaining this setup proved to be quite messy. Dealing with encryption and decryption was painful as well. Having a simple foolproof interface was worth it.
- alex_doom 13y ago
  I've been using them for the past two years as a remote backup. Locally, Time Machine. I don't even have to think about it which is the most appeal to me. I come home and everything gets backed up around midnight.
Spooky23 13y ago
I couldn't disagree more. Duplicity is awesome, I had it running in a prod environment that had some intense security requirements (addressed with GPGcards for keying) But for system backups, you need something simple and easy. Just get arq/etc.
phusion 13y ago
Ahh the bums who think they have right to my name, despite me using it since '00 or so. This still looks like an interesting service.
- toyg 13y ago
  care to elaborate?
  - jlgreco 13y ago
    They both call themselves "phusion".
zzzeek 13y ago
I use duplicity extensively and I've written my own frontend around it. Duplicity itself works very well, though has many areas in which it could improve. One is, it could play within the Python ecosystem more nicely; it doesn't install with the usual setup.py mechanism and also is very resistant to any kind of API/in-process usage; it's various features are locked up tight within its commandline interface so writing frontends means you pretty much have to pipe to a process to work with it. Another is that it needs a feature known as "synthetic backup". If you read up on how people use duplicity, a common theme is the need to "run a full backup" periodically. This is because you don't want to have months worth of backups as an endless long stream of incremental files; it becomes unwieldy and difficult to restore from. In my case, I'm backing up files from my laptop, and a full backup takes hours - I'd rather some process exists which can directly squash a series of incremental backup files into a full backup, and write them back out to S3. I'm actually doing this now, though in a less direct way; my front-end includes a synthetic backup feature which I run on my synology box - it restores from the incremental backup to a temp directory, then pushes everything back up as a new, full backup. My laptop doesn't need to remain open, it only needs to push small incremental files, and I get a nice full backup of everything nightly.
tunesmith 13y ago
Just for clarification, doesn't this backup approach only give you your most recent backup? In other words, is it true that it doesn't do "versions", i.e. doesn't let you restore from n backups ago?
bowlofpetunias 13y ago
The main reason why I prefer Duplicity is its portability. I can easily use the same setup for backups over ssh, rsync, s3, ftp, whatever.
oliwarner 13y ago
Cheap? Pish. A lot of server hosts have a local backup solution. For a 2GB Linode, we're talking about backing up 96GB of storage. More than that, three backups are made (daily, weekly and 2-weekly). 288GB of active backup storage. Handled for you. For $10 a month. Amazon is over twice that before you even consider the hassle of setting it up and restoring backups.
- harshreality 13y ago
  Until linode breaks or is compromised and your server and backups both get broken or hacked and deleted. I don't trust hosting my server and backups with the same company. I backup to s3 with s3 credentials that forbid removal; files are expired out of the duplicity-related s3 buckets using native amazon functionality, not duplicity. It does mean there are some stale incrementals, but that's a small price to pay.
- qwertyboy 13y ago
  Homepage at linode.com lists 2GB Linode at $40 a month. Am I missing something?
banachtarski 13y ago
I know it's not the point of the article, but "Duplicity" is a horrible name. It has negative connotations of dishonesty and is only in the same family as the word "duplication."
engates 13y ago
Duplicity also works with Rackspace Cloud files as detailed here: http://gsusmonzon.blogspot.com/2013/07/backup-with-duplicity-and-rackspace.html http://gsusmonzon.blogspot.com/2013/07/backup-with-duplicity...
nodata 13y ago
duplicity is nice, but it doesn't do dedupe. It's also slow. I currently prefer http://liw.fi/obnam/ http://liw.fi/obnam/ which does dedupe and every backup is a snapshot. Only downside is that encrypted backups are broken if you use the latest gpg.
- 2bluesc 13y ago
  I use obnam too. Way better then duplicity with generation snapshots.
aryastark 13y ago
This article is about backups for servers. I think some people are missing that part. For my personal home machines, I was recently researching backup systems. I decided against cloud backups and opted for the simple solution of a USB 3.0 drive. It's cheap, and if I ever need to get my backup quickly, I just pick it up and take it. For critical documents, I can't imagine most people have more than a few megabytes anyways. That stuff can be archived/encrypted periodically and sent to cloud email (gmail, yahoo) much cheaper and easier than worrying with S3. For the actual backups, I'm using Linux so I went with rsync. See here for roughly how it's done: http://www.mikerubel.org/computers/rsync_snapshots/ http://www.mikerubel.org/computers/rsync_snapshots/ The USB drive is encrypted, I then tell rsync to use hard links and it does the incremental backup. The nice thing about hard links is you can get at any file you want to, in any backup you want to while preserving space. Files in new backups that exist in prior backups are simply hard links. Then you just "rm" older backups when no longer needed. It's an incredibly elegant solution on Linux, especially if you combine it with LVM snapshots.
- ineedtosleep 13y ago
  I've researched this for an extremely long time as well and came to very similar conclusions: * Small-ish type stuff on SpiderOak and Dropbox (both on free tiers). I assume both of these are "compromised". * Data that I want to keep /private/ go into an encrypted USB flash drive and onto Tarsnap. I doubt this will ever get larger than 10-20MB. * I have ~5 full HDD images of Linux boxes that I need to keep for archival purposes and storing that on a cloud backup service would be a pain to upload and download. Those are on a Synology 213+ and on a larger capacity USB3 drive. I'm still evaluating git-annex, but right now I can't seem to find a workflow to my liking. I feel like it /could/ be something I'd use to replace most of the above, but I can't see it yet.
  - edoloughlin 13y ago
    I'm still evaluating git-annex I thought git-annex would be a good solution but I can't seem to get it to stop locking files I'm working on without using its 'unsafe' mode. I can't seem to figure out the workflow it was designed to support.
fwenzel 13y ago
S3 is actually not that cheap. I have well over 100 GB of data to back up (and that's not even a full backup by any means, just the stuff I don't want to lose and can't get back by redownloading and reinstalling things), which racks up a significant bill on S3. CrashPlan isn't cheap either but the cheapest ones of the bunch, so I use them instead. Actually getting the data back in the case of disaster would take a while but I could certainly restore the most important things first, then backfill the rest as I go.
JimmaDaRustla 13y ago
I'm currently using Duplicity and a raspberry pi to push my encrypted files off of a NAS to a VPS on a daily basis. Works out to be $1.38/month for 50gb storage.
kudu 13y ago
If you're interested in this, you should look into DreamObjects (http://www.dreamhost.com/cloud/dreamobjects/ http://www.dreamhost.com/cloud/dreamobjects/). It's a redundant cloud storage service which is compatible with the S3 and Swift protocols at less than half the price ($0.05/GB). It can be used with duplicity (http://www.dreamhost.com/dreamscape/2013/02/11/backing-up-to-dreamobjects-with-duplicity/ http://www.dreamhost.com/dreamscape/2013/02/11/backing-up-to...).
buster 13y ago
OOOrrrr... QNAP home-disk array + deja-dup = automated, really cheap full-disk backup NOT stored on some US-NSA-server..
CyberThijs 13y ago
Has anyone tried the new Glacier-clone of OVH [0] yet? At 0.008 EUR/GBP it seems to be priced very reasonable, and there are no crazy retrieval price structures (altrough retrieval has a 4 hour lead time, just like Glacier). [0] http://www.ovh.co.uk/cloud/archive/ http://www.ovh.co.uk/cloud/archive/
dalore 13y ago
Isn't silence-unless-failed just a way of doing > /dev/null ? Since the stderr isn't redirected cron will get the errors but the other stuff goes to null.
adamonduty 13y ago
I'm hoping that Space Monkey (http://www.spacemonkey.com/ http://www.spacemonkey.com/) helps solve some of this. It backups up to a local appliance-style box, then syncs with other space monkey boxes in a peer to peer manner. Like cloud storage, but cheaper at $10/month for 1TB.

171 points by tortilla 13y ago

aroch 13y ago

I use Arq[1] on my macs and it does the same thing but handles all the "complicated" bits that I'd rather be programmatically dealt with. I have 6 macs, about 2TB of backups in Glacier and a ~20GB "system image" backup in hot-storage on S3. [1] http://www.haystacksoftware.com/arq/ http://www.haystacksoftware.com/arq/

makmanalp 13y ago
How do you like glacier? It seems it's fairly cheap until you want your data back. Any insights?
- aidos 13y ago
  Personally I've factored that in to my cost of recovery. It's going to cost me about £120 to restore my data over the course of a week. It depends on your use case but for me - in the event that glacier is the last point I can get my data from, £120 is a small price to pay to get it back. I used backblaze before and it was about the same to get them to post out a hdd (which was an awesome service on the one occasion I had to use it).
  - toomuchtodo 13y ago
    It's too bad Amazon doesn't offer a discounted rate to restore from Glacier if you pay upfront.
    - zz1 13y ago
      Sorry to chase you between threads, but yes, I'd like to get in touch for the Getty files! Care to share contact info?
      - toomuchtodo 13y ago
        m8r-fhf8r1@mailinator.com (apologies, would prefer not to share my public email address here) I'll reply back from my personal email account.
  - _red 13y ago
    In addition, Glacier should be thought of as "offline" storage , like a tape rotation scheme. When thought of that way, the cost for restoration is non-existent. Its not for "I may need this file next week" (thats what your local NAS / Time Machine is for...Glacier is for "what happens if this place burns down". At that point a week and £120 is not high on the list of your concerns...
- aroch 13y ago
  I like it quite a bit, Arq is my "backup of last resort" and what I use for provision new macs if they're not with me. I have daily/weekly/monthly local backups and semi-local, bi-weekly backups (stored offsite or in a building with a different mac). Setting up a new Mac remotely costs about $30 for the 60GB provisioning profile to be downloaded and restored (maybe more or less depending on the internet connection at the place). I pay a few hundred yearly for backups that I have no doubt will be there if I need them.
aidos 13y ago
I also use Arq and recommend it to everyone on OS X. It has a sane transparent format (modelled on git), it can store to Glacier and it doesn't periodically explode like Back Blaze used to. Interested to hear more about your "system image". How do you do it and what do you store in it? Documents / preferences / Library etc? How would you use it in a recovery situation?
- aroch 13y ago
  It's a separate backup vault in Arq that's maintained by clean OS install + my essential documents, dotfiles, and an encrypted PII volume with scans of passport/SSC/Drivers license/Birth certificate. It's all on a 60GB SSD that's booted every week to update to the latest versions of everything and then to be Arq backed up. Restoration is downloading the vault to a new SSD and shoving it in a computer. The "root" user is included in the backup as an encrypted volume. My setup's a little strange in that the OS disks on my computers are filevaulted and each user's homedir is also in an encrypted volume only they have the password too that's mounted to the correct /Users/folder with fstab. (It'd probably be helpful to note that I have a hackintosh this is done on with 10 internal bays, in which one resides this SSD with a clone of this SSD elsewhere)
  - sreitshamer 13y ago
    Could you write that up in a blog post please? I'd love to read how that works. (I'm the author of Arq)
    - aroch 13y ago
      I could probably whack something together, but what specifically were you wondering about?
      - sreitshamer 13y ago
        I'm wondering about the whole process. It sounds like an external drive that you periodically copy your documents, dot files etc to, and then let Arq back up? Or is it something else? (You talk about "booting" the disk -- do you mean booting the computer from that disk, or just mounting it?) Also I didn't understand the "root" user part. And is this a multi-user computer? You mentioned something about "each user".
apaprocki 13y ago
Another happy Arq user here -- I just use it for Glacier backup, no need for hot S3 storage.
OWaz 13y ago
Has anyone been able to accurately figure out how much Glacier will costs once storage is retrieved from it?
- aroch 13y ago
  Here's my last bill: $0.010 per GB / month - Storage 2112.146 GB-Mo 21.12 $0.050 per 1,000 Requests 40,447 Requests 2.02 My last "restoration" was actually setting up a new iMac at a remote site involved downloading ~60GB over ~20hour and cost me about $30
- mjn 13y ago
  Above the free retrieval quota (5% of your storage per month), pricing is based entirely on your peak hourly retrieval rate for the month, billed at $7.20 x [peak rate measured in GB/hr]. So for example, if you retrieve 100 GB in one hour, that's $720. If you retrieve 10 TB at the rate of 100 GB/hr for 100 hours, that's also $720. Translated into bandwidth (assuming you're pulling at this rate for at least an hour), it's about $3 per Mbps. So if you limit your retrieval to 10 Mbps, you'll pay $30, regardless of how much data you retrieve. This makes it fairly easy to cap your expenditure, if you have a client that can stagger its requests appropriately (you need to throttle the Glacier retrieval requests, possibly using Glacier's range requests if you have very large files, not throttle at the network level).
ivanhoe 13y ago
Arq is simple and works great, although it seems that a half of my S3 bills are for the thousands of requests that Arq makes back and forth. Simpler upload & forget backup model probably could save a few bucks...
- aroch 13y ago
  I make, on average, 40K requests a month which is like $2 while my storage costs are ~$22. At the low end you'll be paying what seems like a lot for requests but as things scale your requests costs become a much smaller percentage. You an cut down on requests by moving to daily backups (what I do) and narrowing down what you're backing up. Backing up your applications folders will take a ton of requests (Some apps have upwards of 1000 assets whose bless values may change if you open the application and this trigger Arq)
aesptux 13y ago
Do you recommend Glacier over Crashplan? I'm thinking about switching
- ye 13y ago
  They are very different. Glacier takes hours just to get the listing of files, or start the file download. Their pricing is very different. Crashplan charges per computer. Glacier charges per GB of storage and per GB of transfer out.
  - aesptux 13y ago
    In what cases would be better to use Glacier? I don't have too many GB, ~30GB.
    - TkTech 13y ago
      Glacier is for when you want to store essentially never-changing backups that you expect to very, very rarely access. You can store a couple GB or petabytes. Think an archive of family videos and photos spanning 30 years, or all the tax documents for a large enterprise.
    - aroch 13y ago
      A 30GB backup, that's updated nightly, on AWS Glacier will cost somewhere in the neighborhood of $1 a month. A 300GB backup somewhere around $5 on AWS.
    - ye 13y ago
      Glacier is cheaper for small backups that you almost never touch.
dewiz 13y ago
Is there something like Arq compatible with both OSX and Windows ? Features I'm interested in: - must use S3 or similar, as long as it's private and "unlimited" - must support multiple clients, even though I don't need real time sync - should encrypt on the client without need of a separate sw I've been using Dropbox + CloudFogger so far, but dropbox doesn't scale anymore with the quantity of data I have, mostly due to the number of files (and I don't like depending on two different sw for one task).
- aroch 13y ago
  The only two I have used are git-annex [1] and duplicati [2]. I much prefer git-annex but the windows version isn't so stable. [1] http://git-annex.branchable.com/ http://git-annex.branchable.com/ [2] http://www.duplicati.com/ http://www.duplicati.com/

zobzu 13y ago

duplicity + anything = easy,cheap,encrypted, automated full-disk/incremental/etc backups. duplicity's simple yet pretty awesome.

bkmartin 13y ago

How would one do something similarly easy and economically efficient for a Windows platform? We've looked at different services but they are always so expensive. I would love a piece of software that we can place on each server and let it do its thing.... of course there is always SQL Server to contend with, again... expensive solutions exist.

uptown 13y ago
Check out Crashplan. You can use their software for incremental backups to the destination of your choice at no cost, and it's multi-platform provided you install Java.
- pwenzel 13y ago
  Another Crashplan fan here. You can also install the Crashplan client on headless Linux-based NAS devices, so long as they can run Java (such as ReadyNAS).
aidos 13y ago
Personally I'd probably use something to sync the lot over to a linux box and do the rest of the work from there. Depends on how much data you need to sync / how many servers are involved. Could be a pain to restore too (depending on the type of data you're backing up) so you would want to weigh all of that up too. With the Sql Server backups you could run periodic backups from Sql Server itself and have those backed up - depending on the data this could be really suboptimal but I think sql server allows incremental backups in the backup set, doesn't it? Been a number of years since I've had to deal with it. You can use some sort of rsync alternative to get the content from the windows boxes to the linux machine (cwrsync is recommended - though I've never used it). http://stackoverflow.com/questions/528298/rsync-for-windows https://www.itefix.no/i2/cwrsync
kevinoid 13y ago
I feel your pain. I actually ported Duplicity to Windows about 3 years ago for a client, but the code was never merged (or commented on).[1] In the end, that client decided that it was easier to use a commercial package, given their typical use case. Tried Carbonite (didn't work, support unhelpful) but settled on JungleDisk. YMMV. 1. https://code.launchpad.net/~kevinoid/duplicity/windows-port https://code.launchpad.net/~kevinoid/duplicity/windows-port

jewel 13y ago

The issue I have with duplicity is that every N backups have to be a full backup, so a large backup (such as my photo collection) becomes impossible.

orthecreedence 13y ago
Is this a technical limitation? I'm not seeing anything in the docs that forces you to do a full backup after N amount of backups (maybe I missed this).
- brucehart 13y ago
  There is no technical limitation. You can do as many incremental backups as you want and are not forced to do a full backup after the initial one. As a practical matter though, I've found it's a good idea to occasionally do a full backup because the metadata can get out of sync (especially if something goes wrong like you lose your connection during an incremental backup).
- Spooky23 13y ago
  You can do many many incrementals, but the restoration process, especially from a remote source is incredibly slow.
  - zobzu 13y ago
    its not that bad. i do it from a shiva plug over a 10mbit DSL link for a few dozen gigs regularly. its as fast as the DSL will go.
- ef4 13y ago
  In addition to the other reasons given, there's a reliability problem in relying on an ever-growing chain of incremental backups: you can only recover your latest state if every increment in the chain is uncorrupted. I would much rather use an incremental backup that records the diffs in reverse, so that the latest state is stored explicitly, and the earlier versions are recovered through a chain of reverse diffs. Duplicity's security goals make it hard to implement this kind of reverse diff structure.

PlaneSploit 13y ago

Duplicity has consistently failed to restore for me in the form of deja-dup on Ubuntu.

tenfingers 13y ago
Restoring from duplicity takes a proportional amount of time related to the backup chain length, as archives are stored as forward-deltas. Nothing against forward deltas, but just consider that: - restoring time increases with increment count - risk of corruption of a single archives will make all further increments worthless. These are very important things to consider when doing a backup. Currently, you have to limit the chain length by performing a full backup every N increments (with N being as short as possible), which defeats the purpose of efficient increments. I have requested the ability to specify manually the ancestor of the increments (by time or count), so that one could implement a non-linear hierarchy with multiple levels like one normally does with tar/timestamps, but the request was dubbed as "unnecessary given the delta efficiency" (despite the fact that efficiency is just one variable). Having a 3 level backup (daily, weekly, monthly) would make duplicity much more space efficient, reduce the number of full backups needed, and shorten the chains to the level that would make restores of "last year" actually _possible_. I sent several patches to fix duplicity behavior with files larger than 1gb (by limiting block counts), which got integrated, but are still a far cry to make duplicity work decently as a whole-system backup&restore solution. It's just too slow. And like you said, several bugs afflicted duplicity in the past that would make restore fail in many circumstances. I also debugged my share of issues, which led me to think that very few people actually tried to restore from arbitrary increments with duplicity and/or used them to archive a large system. Many got fixed, but I won't consider duplicity again until I can control the incremental ancestor and reduce the chain lengths (and it's silly to think that "rdiffdir", distributed with duplicity, would allow for that easily). Nowdays I use rdiff-backup, and use a second script to tar/encrypt the deltas after the backup has completed. I'm keeping an eye on "bup" (https://github.com/bup/bup https://github.com/bup/bup), but I cannot backup "forever", thus without the ability of purging old versions it's only useful in a limited set of cases.
- rlpb 13y ago

kbar13 13y ago

duplicity + s4[0], because s4 is greater than s3. [0] https://leastauthority.com/ https://leastauthority.com/

deleted 13y ago

[deleted]

jd 13y ago

Quick correction: it's not true that rdiff-backup has to be installed on the remote server. Rdiff-backup works brilliantly with a dumb target for storage and that's how we use it in production. Our servers periodically run rdiff-backup of all important data to a /backup partition on the same server. Then this /backup partition -- with the backup version history and metadata -- is rsynced to various dumb backup storage locations. We've tried many backup solutions, and rdiff-backup is by far the fastest and most robust backup program we know.

ars 13y ago
> is rsynced to various dumb backup storage locations. Rsync requires a program on the other end. True, it's a more commonly installed program than rdiff-backup but it's not a dump backup.
- andreasvc 13y ago
  The difference is that rdiff-backup requires the exact same version on both ends.
jackalope 13y ago
You might want to reconsider this approach. I do something similar, but my remote backup also uses rdiff-backup. While it's awfully tempting to rsync the local backup to a remote location, it puts your entire backup at risk because rsync doesn't care about maintaining the state that rdiff-backup depends on. If your local disk is failing with transient errors, or your local /backup mount point disappears, the next rsync could mirror the corruption, leaving you without any backup at all. It's not worth the time savings and after the initial full backup the network overhead is negligible, even over ssh.

dotemacs 13y ago

What's the benefit of Duplicity over Tarsnap http://www.tarsnap.com/ http://www.tarsnap.com/ ? Thanks

reidrac 13y ago
Duplicity is an open source project that can be used with different types of storage (I use them with OpenStack Object Storage instead of S3). You can host your own backups if you want (supports ftp, sftp, ... check the docs: http://duplicity.nongnu.org/duplicity.1.html http://duplicity.nongnu.org/duplicity.1.html), but Tarsnap is an hosted solution.
aquark 13y ago
I played a bit with both and they both worked well. Tarsnap: easier to set up and use Duplicity: fully open source, and I think the restore speed was significantly faster ... but I've not properly benchmarked it apples-to-apples. If you are backing up a lot of frequently changing files (eg. database backups) then tarsnap's bandwidth charges (probably caused by the underlying EC2 charges) can swamp the storage cost. Running duplicity against rsync.net avoids this.
- gingerlime 13y ago
  restore speed was significantly faster with duplicity? I haven't used tarsnap yet, but one of my main gripes with duplicity is the rather slow restore speed. Is tarsnap even slower?
  - tedunangst 13y ago
    Depends. tarsnap restore times should be constant regardless of which backup you pick, since it's just downloading the relevant blocks. duplicity does full backup + diffs, so if your data changes quite a bit you can end up downloading a lot of data that doesn't land on disk. (i believe that's right, not verified.)

sahaskatta 13y ago

I've been using BackBlaze. I find it both cost effective and time saving. Here's why: Price: It costs only $3.96 a month if you pay for two years up front. ($4.17 a month if you pay for only 1 year up front.) They offer unlimited storage of everything on internal and external HDDs. 30 days of file revisions too. I think I have close to 1TB of data backed up with their service. (Using Duplicity w/ S3 would cost me $90 a MONTH in comparison.) Time: I've always been using Windows File History to backup onto a external USB HDD, but I wanted an off-site backup. I initially started backing up to S3 myself and even dabbled with Glacier. However, maintaining this setup proved to be quite messy. Dealing with encryption and decryption was painful as well. Having a simple foolproof interface was worth it.

alex_doom 13y ago
I've been using them for the past two years as a remote backup. Locally, Time Machine. I don't even have to think about it which is the most appeal to me. I come home and everything gets backed up around midnight.

Spooky23 13y ago

I couldn't disagree more. Duplicity is awesome, I had it running in a prod environment that had some intense security requirements (addressed with GPGcards for keying) But for system backups, you need something simple and easy. Just get arq/etc.

phusion 13y ago

Ahh the bums who think they have right to my name, despite me using it since '00 or so. This still looks like an interesting service.

toyg 13y ago
care to elaborate?
- jlgreco 13y ago
  They both call themselves "phusion".

zzzeek 13y ago

I use duplicity extensively and I've written my own frontend around it. Duplicity itself works very well, though has many areas in which it could improve. One is, it could play within the Python ecosystem more nicely; it doesn't install with the usual setup.py mechanism and also is very resistant to any kind of API/in-process usage; it's various features are locked up tight within its commandline interface so writing frontends means you pretty much have to pipe to a process to work with it. Another is that it needs a feature known as "synthetic backup". If you read up on how people use duplicity, a common theme is the need to "run a full backup" periodically. This is because you don't want to have months worth of backups as an endless long stream of incremental files; it becomes unwieldy and difficult to restore from. In my case, I'm backing up files from my laptop, and a full backup takes hours - I'd rather some process exists which can directly squash a series of incremental backup files into a full backup, and write them back out to S3. I'm actually doing this now, though in a less direct way; my front-end includes a synthetic backup feature which I run on my synology box - it restores from the incremental backup to a temp directory, then pushes everything back up as a new, full backup. My laptop doesn't need to remain open, it only needs to push small incremental files, and I get a nice full backup of everything nightly.

tunesmith 13y ago

Just for clarification, doesn't this backup approach only give you your most recent backup? In other words, is it true that it doesn't do "versions", i.e. doesn't let you restore from n backups ago?

bowlofpetunias 13y ago

The main reason why I prefer Duplicity is its portability. I can easily use the same setup for backups over ssh, rsync, s3, ftp, whatever.

oliwarner 13y ago

Cheap? Pish. A lot of server hosts have a local backup solution. For a 2GB Linode, we're talking about backing up 96GB of storage. More than that, three backups are made (daily, weekly and 2-weekly). 288GB of active backup storage. Handled for you. For $10 a month. Amazon is over twice that before you even consider the hassle of setting it up and restoring backups.

harshreality 13y ago
Until linode breaks or is compromised and your server and backups both get broken or hacked and deleted. I don't trust hosting my server and backups with the same company. I backup to s3 with s3 credentials that forbid removal; files are expired out of the duplicity-related s3 buckets using native amazon functionality, not duplicity. It does mean there are some stale incrementals, but that's a small price to pay.
qwertyboy 13y ago
Homepage at linode.com lists 2GB Linode at $40 a month. Am I missing something?

banachtarski 13y ago

I know it's not the point of the article, but "Duplicity" is a horrible name. It has negative connotations of dishonesty and is only in the same family as the word "duplication."

engates 13y ago

Duplicity also works with Rackspace Cloud files as detailed here: http://gsusmonzon.blogspot.com/2013/07/backup-with-duplicity-and-rackspace.html http://gsusmonzon.blogspot.com/2013/07/backup-with-duplicity...

nodata 13y ago

duplicity is nice, but it doesn't do dedupe. It's also slow. I currently prefer http://liw.fi/obnam/ http://liw.fi/obnam/ which does dedupe and every backup is a snapshot. Only downside is that encrypted backups are broken if you use the latest gpg.

2bluesc 13y ago
I use obnam too. Way better then duplicity with generation snapshots.

aryastark 13y ago

This article is about backups for servers. I think some people are missing that part. For my personal home machines, I was recently researching backup systems. I decided against cloud backups and opted for the simple solution of a USB 3.0 drive. It's cheap, and if I ever need to get my backup quickly, I just pick it up and take it. For critical documents, I can't imagine most people have more than a few megabytes anyways. That stuff can be archived/encrypted periodically and sent to cloud email (gmail, yahoo) much cheaper and easier than worrying with S3. For the actual backups, I'm using Linux so I went with rsync. See here for roughly how it's done: http://www.mikerubel.org/computers/rsync_snapshots/ http://www.mikerubel.org/computers/rsync_snapshots/ The USB drive is encrypted, I then tell rsync to use hard links and it does the incremental backup. The nice thing about hard links is you can get at any file you want to, in any backup you want to while preserving space. Files in new backups that exist in prior backups are simply hard links. Then you just "rm" older backups when no longer needed. It's an incredibly elegant solution on Linux, especially if you combine it with LVM snapshots.

ineedtosleep 13y ago
I've researched this for an extremely long time as well and came to very similar conclusions: * Small-ish type stuff on SpiderOak and Dropbox (both on free tiers). I assume both of these are "compromised". * Data that I want to keep /private/ go into an encrypted USB flash drive and onto Tarsnap. I doubt this will ever get larger than 10-20MB. * I have ~5 full HDD images of Linux boxes that I need to keep for archival purposes and storing that on a cloud backup service would be a pain to upload and download. Those are on a Synology 213+ and on a larger capacity USB3 drive. I'm still evaluating git-annex, but right now I can't seem to find a workflow to my liking. I feel like it /could/ be something I'd use to replace most of the above, but I can't see it yet.
- edoloughlin 13y ago
  I'm still evaluating git-annex I thought git-annex would be a good solution but I can't seem to get it to stop locking files I'm working on without using its 'unsafe' mode. I can't seem to figure out the workflow it was designed to support.

fwenzel 13y ago

S3 is actually not that cheap. I have well over 100 GB of data to back up (and that's not even a full backup by any means, just the stuff I don't want to lose and can't get back by redownloading and reinstalling things), which racks up a significant bill on S3. CrashPlan isn't cheap either but the cheapest ones of the bunch, so I use them instead. Actually getting the data back in the case of disaster would take a while but I could certainly restore the most important things first, then backfill the rest as I go.

JimmaDaRustla 13y ago

I'm currently using Duplicity and a raspberry pi to push my encrypted files off of a NAS to a VPS on a daily basis. Works out to be $1.38/month for 50gb storage.

kudu 13y ago

If you're interested in this, you should look into DreamObjects (http://www.dreamhost.com/cloud/dreamobjects/ http://www.dreamhost.com/cloud/dreamobjects/). It's a redundant cloud storage service which is compatible with the S3 and Swift protocols at less than half the price ($0.05/GB). It can be used with duplicity (http://www.dreamhost.com/dreamscape/2013/02/11/backing-up-to-dreamobjects-with-duplicity/ http://www.dreamhost.com/dreamscape/2013/02/11/backing-up-to...).

buster 13y ago

OOOrrrr... QNAP home-disk array + deja-dup = automated, really cheap full-disk backup NOT stored on some US-NSA-server..

CyberThijs 13y ago

Has anyone tried the new Glacier-clone of OVH [0] yet? At 0.008 EUR/GBP it seems to be priced very reasonable, and there are no crazy retrieval price structures (altrough retrieval has a 4 hour lead time, just like Glacier). [0] http://www.ovh.co.uk/cloud/archive/ http://www.ovh.co.uk/cloud/archive/

dalore 13y ago

Isn't silence-unless-failed just a way of doing > /dev/null ? Since the stderr isn't redirected cron will get the errors but the other stuff goes to null.

adamonduty 13y ago

I'm hoping that Space Monkey (http://www.spacemonkey.com/ http://www.spacemonkey.com/) helps solve some of this. It backups up to a local appliance-style box, then syncs with other space monkey boxes in a peer to peer manner. Like cloud storage, but cheaper at $10/month for 1TB.