5 ms·

Over a year old article, but I'm still curious; what options are there for backups these days?

by iamtew 12y ago

wazoox 12y ago
Apparently BareOS is making progress, and has now scripting capabilities. Go BareOS! Die, Bacula!
xorcist 12y ago
That depends on what you requirements are. Do you need application specific backups (which?), windows support, mac support, hierarchical storage, tape support? For personal/development use on Linux, you should absolutely take a look at obnam (and the similar attic and bup). They are simple software that do dedup, encryption and cloud storage.
- iamtew 12y ago
  Right, I was quite sparse with details... I do mainly Linux stuff, and obnam looks nice and feature-packed enough to fit my current needs, I'll dig in to that a bit. Cheers :-)
  - ThatGeoGuy 12y ago
    To be honest, I had the same question last year, and wrote a short guide on how I (eventually) came to do backups [1]. Funny enough, I didn't even notice the topic article until today, which is kind of funny that I see it almost exactly a year later (only off by two days, really). Of course, this is very Linux centric, but it fits my needs and likewise allows me to rsync back anything on the backup at any time, even if I accidentally all of /bin/ or something equally stupid. Though now I've somewhat updated it (I use a fully encrypted system and set cryptsetup to use pass-files on the main disk for the backups in addition to a set password), but the fundamentals are all there. [1] https://thatgeoguy.ca/blog/2013/12/26/encrypted-backups-in-debian/ https://thatgeoguy.ca/blog/2013/12/26/encrypted-backups-in-d...
- pyre 12y ago
  Is there anything about obnam that is Linux-specific? As long as I don't need any HFS-specific features, it should work on OSX, right?
rakoo 12y ago
Simple, Secure Backup That Just Works: https://www.tarsnap.com/ https://www.tarsnap.com/ EDIT: it will require some manual fiddling, especially if have multiple machines, so it's not an option that passes all criterias from the article.
- derekp7 12y ago
  Is it possible to set up a local server for tarsnap, or is the client only for usage with the tarsnap service? And since the client isn't open source (but is distributed as source code), would it be a license violation if someone reverse-engineered the protocol to write an open server component? For cloud-based backups, my preference would be to backup locally first, then sync that to a cloud service, since restoring from a local box is faster than remote. It might be a fun project to create an intermediary server which the tarsnap client talks to, and have that server then forward everything to the cloud service.
  - icebraining 12y ago
    Is it possible to set up a local server for tarsnap, or is the client only for usage with the tarsnap service? It's just for the Tarsnap service. And since the client isn't open source (but is distributed as source code), would it be a license violation if someone reverse-engineered the protocol to write an open server component? It's not clear: https://www.eff.org/issues/coders/reverse-engineering-faq https://www.eff.org/issues/coders/reverse-engineering-faq
derekp7 12y ago
There is my entry, Snebu, which I've been using for the past couple years, at http://www.snebu.com http://www.snebu.com (hosted at Github). There are two things that it does not do, encryption and block-level deduplication. For encryption, my preference is to write it to an encrypted volume. And block-level deduplication would require inventing another proprietary storage format, which I try to avoid. It works a lot like rsnapshot (each backup is a self contained snapshot of a system at that point in time, and only the incremental changes are transferred to the backup server), but with compression and full file level deduplication (not only across backup generations, but across all files including ones from multiple clients). It uses GNU `find` and `tar` on the client to do the heavy lifting of getting the files (so only a simple shell script is needed on the client, and then only if you are doing push backups). On the backend side, it stores the metadata in an SQLite table, and individual files are stored in a vault directory using lzop compatible compression. The advantages over tools like rsnapshot is that it has compression, full file level deduplication, and metadata is stored in a DB file (which avoids a large hardlink-farm for dedup (avoiding problems like https://news.ycombinator.com/item?id=8305283 https://news.ycombinator.com/item?id=8305283), also removes the need for root privileges on the backup server to store file ownership/permissions). The disadvantages compared to other tools, is it doesn't do encryption on the client (this would break dedup if done right, and I'd rather leave encryption to the experts), and it only does file level dedup, so isn't suitable for backing up VM image files (unless you combine it with something like libguestfs, something I'll document in an upcoming release). There are also some nice-to-have features, for example if you have a job to expire all daily backups older than 10 days, and monthly older than 6 months -- and you have a host that hasn't been running / backed up for a year, the expire job will keep a minimum number of backups for each host (default 3 backups, but is tuneable). And the next major release will support multiple repositories, so you can move old backup sets to archival storage (external USB disks, possibly tape library support, etc). Also included is support for storing extended / SElinux attributes, and the current client shell script has a plugin architecture to execute pre/post backup steps for app-specific backups, such as Oracle hot backups. Edit: for Windows backups, I'm currently running it under Cygwin. But I'm not able to preserve Windows specific permissions. To fix this, I need a "tar" implementation for Windows which stores the extended ACL information in PAX headers in the tar file, similar to how Red Hat's patched GNU tar does SElinux attributes. If I can't find something, I may take a crack at it myself.

Over a year old article, but I'm still curious; what options are there for backups these days?

by iamtew 12y ago

Over a year old article, but I'm still curious; what options are there for backups these days?

wazoox 12y ago

Apparently BareOS is making progress, and has now scripting capabilities. Go BareOS! Die, Bacula!

xorcist 12y ago

That depends on what you requirements are. Do you need application specific backups (which?), windows support, mac support, hierarchical storage, tape support? For personal/development use on Linux, you should absolutely take a look at obnam (and the similar attic and bup). They are simple software that do dedup, encryption and cloud storage.

iamtew 12y ago
Right, I was quite sparse with details... I do mainly Linux stuff, and obnam looks nice and feature-packed enough to fit my current needs, I'll dig in to that a bit. Cheers :-)
- ThatGeoGuy 12y ago
  To be honest, I had the same question last year, and wrote a short guide on how I (eventually) came to do backups [1]. Funny enough, I didn't even notice the topic article until today, which is kind of funny that I see it almost exactly a year later (only off by two days, really). Of course, this is very Linux centric, but it fits my needs and likewise allows me to rsync back anything on the backup at any time, even if I accidentally all of /bin/ or something equally stupid. Though now I've somewhat updated it (I use a fully encrypted system and set cryptsetup to use pass-files on the main disk for the backups in addition to a set password), but the fundamentals are all there. [1] https://thatgeoguy.ca/blog/2013/12/26/encrypted-backups-in-debian/ https://thatgeoguy.ca/blog/2013/12/26/encrypted-backups-in-d...
pyre 12y ago
Is there anything about obnam that is Linux-specific? As long as I don't need any HFS-specific features, it should work on OSX, right?

rakoo 12y ago

Simple, Secure Backup That Just Works: https://www.tarsnap.com/ https://www.tarsnap.com/ EDIT: it will require some manual fiddling, especially if have multiple machines, so it's not an option that passes all criterias from the article.

derekp7 12y ago
Is it possible to set up a local server for tarsnap, or is the client only for usage with the tarsnap service? And since the client isn't open source (but is distributed as source code), would it be a license violation if someone reverse-engineered the protocol to write an open server component? For cloud-based backups, my preference would be to backup locally first, then sync that to a cloud service, since restoring from a local box is faster than remote. It might be a fun project to create an intermediary server which the tarsnap client talks to, and have that server then forward everything to the cloud service.
- icebraining 12y ago
  Is it possible to set up a local server for tarsnap, or is the client only for usage with the tarsnap service? It's just for the Tarsnap service. And since the client isn't open source (but is distributed as source code), would it be a license violation if someone reverse-engineered the protocol to write an open server component? It's not clear: https://www.eff.org/issues/coders/reverse-engineering-faq https://www.eff.org/issues/coders/reverse-engineering-faq

derekp7 12y ago

There is my entry, Snebu, which I've been using for the past couple years, at http://www.snebu.com http://www.snebu.com (hosted at Github). There are two things that it does not do, encryption and block-level deduplication. For encryption, my preference is to write it to an encrypted volume. And block-level deduplication would require inventing another proprietary storage format, which I try to avoid. It works a lot like rsnapshot (each backup is a self contained snapshot of a system at that point in time, and only the incremental changes are transferred to the backup server), but with compression and full file level deduplication (not only across backup generations, but across all files including ones from multiple clients). It uses GNU `find` and `tar` on the client to do the heavy lifting of getting the files (so only a simple shell script is needed on the client, and then only if you are doing push backups). On the backend side, it stores the metadata in an SQLite table, and individual files are stored in a vault directory using lzop compatible compression. The advantages over tools like rsnapshot is that it has compression, full file level deduplication, and metadata is stored in a DB file (which avoids a large hardlink-farm for dedup (avoiding problems like https://news.ycombinator.com/item?id=8305283 https://news.ycombinator.com/item?id=8305283), also removes the need for root privileges on the backup server to store file ownership/permissions). The disadvantages compared to other tools, is it doesn't do encryption on the client (this would break dedup if done right, and I'd rather leave encryption to the experts), and it only does file level dedup, so isn't suitable for backing up VM image files (unless you combine it with something like libguestfs, something I'll document in an upcoming release). There are also some nice-to-have features, for example if you have a job to expire all daily backups older than 10 days, and monthly older than 6 months -- and you have a host that hasn't been running / backed up for a year, the expire job will keep a minimum number of backups for each host (default 3 backups, but is tuneable). And the next major release will support multiple repositories, so you can move old backup sets to archival storage (external USB disks, possibly tape library support, etc). Also included is support for storing extended / SElinux attributes, and the current client shell script has a plugin architecture to execute pre/post backup steps for app-specific backups, such as Oracle hot backups. Edit: for Windows backups, I'm currently running it under Cygwin. But I'm not able to preserve Windows specific permissions. To fix this, I need a "tar" implementation for Windows which stores the extended ACL information in PAX headers in the tar file, similar to how Red Hat's patched GNU tar does SElinux attributes. If I can't find something, I may take a crack at it myself.