7 ms·

A shoutout for attic https://attic-backup.org/ https://attic-backup.org/ Attic is one of the new-generation hash-backup tools (like obnam, zbackup, Vembu Hive

by mappu 12y ago

A shoutout for attic https://attic-backup.org/ https://attic-backup.org/ Attic is one of the new-generation hash-backup tools (like obnam, zbackup, Vembu Hive etc). It provides encrypted incremental-forever (unlike duplicity, duplicati, rsnapshot, rdiff-backup, Ahsay etc) with no server-side processing and a convenient CLI interface, and it does let you prune old backups. All other common tools seem to fail on one of the following points - Incremental forever (bandwidth is expensive in a lot of countries) - Untrusted remote storage (so i can hook it up to a dodgy lowendbox VPS) - Optional: No server-side processing needed (so i can hook it up to S3 or Dropbox) If your backup model is based on the old' original + diff(original, v1) + diff(v1, v2).. then you're going to have a slow time restoring. rdiff-backup gets this right by reversing the incremental chain. However, as soon as you need to consolidate incremental images, you lose the possibility of encrypting the data (since encrypt(diff()) is useless from a diff perspective). But with a hash-based backup system? All restore points take constant time to restore. Duplicity, Duplicati 1.x, and Ahsay 5 don't support incremental-forever. Ahsay 6 supports incremental-forever at the expense of requiring trust in the server (server-side decrypt to consolidate images). Duplicati 2 attempted to move to a hash-based system but they chose to use fixed block offsets rather than checksum-based offsets, so the incremental detection is inefficient after an insert point. IMO Attic gets everything right. There's patches for windows support on their github. I wrote a munin plugin for it. Disclaimer: I work in the SMB backup industry.

photoGrant 12y ago
For someone like me who isn't very technically minded (forgive me)... Could you or someone explain why these fantastic sounding tools don't get a developed front-end? Or if they do why am I missing them? The best solution I've found is ChronoSync.
- aidenn0 12y ago
  Because (for the most part) non server-admins don't do backups. It's like flossing; everybody knows you should do it, but nobody actually does it.
- lelutin 12y ago
  as is the case with most 100% volunteer-based free software: lack of time and/or ppl to help. There are currently multiple interfaces available to bup, but each one has some quirks. bup's web interface is still very embryonic and would need the magic touch of some designers / integration specialists to make it fun to work with.
- michaelbuckbee 12y ago
  You might want to checkout Arq from Haystack software - http://www.haystacksoftware.com/arq/ http://www.haystacksoftware.com/arq/ - bring your own storage, GUI client, but with open source client in case they disappear, good support, etc.
- rakoo 12y ago
  For bup, there is something called Kup (http://kde-apps.org/content/show.php/Kup+Backup+System?content=147465 http://kde-apps.org/content/show.php/Kup+Backup+System?conte...), but I haven't tested it. For duplicity, there is a quite good ui in the form of deja dup (see http://www.howtogeek.com/108869/how-to-back-up-ubuntu-the-easy-way-with-dj-dup/ http://www.howtogeek.com/108869/how-to-back-up-ubuntu-the-ea...). It's really nice and easy to use, and if I recall correctly it's installed by default on Ubuntu.
- wvh 12y ago
  Being interested in Linux/Unix, file systems, crypto or whatever other technical topic doesn't necessarily mean you are also interested in developing graphical interfaces. That's an art form and discipline in itself. I suppose the people who make these tools are more interested in the tools' respective technical aspects than in interface or graphical design. Moreover, many of these backup tools run on servers (or as an automated background process) where a graphical interface is more of a handicap than an asset. I think that in open-source software "pretty interface" sounds often like "customer-oriented" which in turn sounds like "getting paid".
sciurus 12y ago
How does attic compare to obnam?
- mappu 12y ago
  I haven't used obnam much, the feature sets are quite similar, perhaps someone else can chip in here. My personal choice of attic over obnam was based on this performance comparison[1] 1. http://librelist.com/browser//attic/2014/1/31/attic-vs-obnam/ http://librelist.com/browser//attic/2014/1/31/attic-vs-obnam...
rsync 12y ago
From: https://attic-backup.org/quickstart.html#quickstart https://attic-backup.org/quickstart.html#quickstart "Attic can initialize and access repositories on remote hosts if the host is accessible using SSH." Fantastic. Will work perfectly here. We[1] are happy to support this just like we've supported duplicity all of these years. EDIT: appears obnam also works over plain old SSH. Can't tell about zbackup, however... As always, email us to discuss the "HN Readers" discount. [1] rsync.net
kijin 12y ago
Sorry, but "Untrusted remote storage" and "No server-side processing" are exactly the opposite of what I need. If the original box is ever compromised, I don't want the attacker to gain any access to the backup. If you use a dumb storage like S3 as your backup server, you need to store your keys on the original box, and anyone who gains control of the original box can destroy your S3 bucket as well. Ditto for any SSH-based backup scheme that requires keys to be stored on the original box. A compromised box could also lie about checksums, silently corrupting your backups. Backups should be pulled from the backup box, not pushed from the original box. Pushing backups is only acceptable for consumer devices, and even then, only because we don't have a reliable way to pull data from them (due to frequently changing IP addresses, NAT, etc). The backup box needs to be even more trustworthy than the original box, not the other way around. I'm willing to live with a significant amount of overhead, both in storage and in bandwidth, in order not to violate this principle. The backup box, of course, could push encrypted data to untrusted storage, such as S3. But only after it has pulled from the original box. In both cases, the connection is initiated from the backup box, not the other way around. The backup box never accepts any incoming connection. Does Attic support this kind of use case? The documentation doesn't seem to have anything to say about backing up remote files to local repositories. I don't see any reason why it won't be supported (since rsync does), but "nominally supported" is different from "optimized for that use case", and I suspect that many of the latest generation of backup tools are optimized for the opposite use case.
- wmf 12y ago
  What about append-only remote storage? This is possible (in a kludgy way) in S3: http://stackoverflow.com/questions/10592541/amazon-s3-acl-for-read-only-and-write-once-access/10592928 http://stackoverflow.com/questions/10592541/amazon-s3-acl-fo...
  - kijin 12y ago
    Better, but it might still expose to the attacker more data than he would otherwise have access to. For example: production box only contains data from the last 3 days, but the backup contains data from the last 12 months. Even stricter access controls (write once, no read) might help with that. Not sure if you can do that with S3 though.
witten 12y ago
If someone is looking at trying attic and wants something a little nicer to configure and run than the very basic shell script from attic's Quick Start[1], check out the wrapper script I wrote to make this a little easier[2]. It's still somewhat "1.0" right now, but it does the basics for me. See the included sample config files for an idea of how the configuration works.[3] [1] https://attic-backup.org/quickstart.html#automating-backups https://attic-backup.org/quickstart.html#automating-backups [2] http://torsion.org/hg/atticmatic/file/tip http://torsion.org/hg/atticmatic/file/tip [3] http://torsion.org/hg/atticmatic/file/tip/sample/config http://torsion.org/hg/atticmatic/file/tip/sample/config
- anarcat 12y ago
  i'm curious: why isn't this part of attic in the first place? i have been working with the bup folks for a while to try to make a similar interface, and i was wondering what was your experience with upstream...
  - witten 12y ago
    Great question. I do think this should be part of what attic provides out of the box, but I still really wanted to use attic despite the fact that it doesn't include this sort of functionality. I'll try contacting the attic devs and see what they say about it.
    - IgorPartola 12y ago
      Your wrapper is only lacking one critical feature I'd love. I am currently using rsnapshot and while its big issue is lack of encryption, it is able to run scripts on remote hosts to pull backups from them. This is a big deal to me since I can then script things like MySQL/Postgres backups, etc. on my master server, rather than having to configure each host individually. It's possible that this is a bad way to run things since my master server is then a SPoF. I do trust this server more since I monitor it much more closely, than I would a dozen random VPS's with a half-dozen different providers.
StavrosK 12y ago
I am also a very happy attic user and can attest to the wonderfulness that is it. Here's a blog post about it: http://www.stavros.io/posts/holy-grail-backups/ http://www.stavros.io/posts/holy-grail-backups/
anarcat 12y ago
from what i can tell here: https://github.com/c4rlo/attic/commit/f4804c07caac3d145f49fc1375c6448111381708 https://github.com/c4rlo/attic/commit/f4804c07caac3d145f49fc... ... attic updates the atime when opening files right now... that patche fixes the problem, but i have to wonder how fast it actually and accurate is. bup has all sorts of helpers and tweaks to make it really fast, but also to ensure that it doesn't touch the filesystem while making backups. i've been amazed to notice how all my files have atimes from july, when i stopped using rsync to make backups and switched to bup. :p
anarcat 12y ago
i was wondering how this would compare with bup. so i did a very quick benchmark: http://anarcat.koumbit.org/2014-11-18-bup-vs-attic-silly-benchmark http://anarcat.koumbit.org/2014-11-18-bup-vs-attic-silly-ben... take it with a grain of salt, but i am surprised to say that attic is at least as fast as bup.
- INTPenis 12y ago
  The only point that stuck in my eye was atime modification from attic but I realized that in my environment I would have to mount all the backup clients remotely on the backup server so I can get around this by mounting sshfs with noatime.
tenfingers 12y ago
Thanks for mentioning attic. Looks like an interesting project which slipped under my radar.

by mappu 12y ago

photoGrant 12y ago

For someone like me who isn't very technically minded (forgive me)... Could you or someone explain why these fantastic sounding tools don't get a developed front-end? Or if they do why am I missing them? The best solution I've found is ChronoSync.

aidenn0 12y ago
Because (for the most part) non server-admins don't do backups. It's like flossing; everybody knows you should do it, but nobody actually does it.
lelutin 12y ago
as is the case with most 100% volunteer-based free software: lack of time and/or ppl to help. There are currently multiple interfaces available to bup, but each one has some quirks. bup's web interface is still very embryonic and would need the magic touch of some designers / integration specialists to make it fun to work with.
michaelbuckbee 12y ago
You might want to checkout Arq from Haystack software - http://www.haystacksoftware.com/arq/ http://www.haystacksoftware.com/arq/ - bring your own storage, GUI client, but with open source client in case they disappear, good support, etc.
rakoo 12y ago
For bup, there is something called Kup (http://kde-apps.org/content/show.php/Kup+Backup+System?content=147465 http://kde-apps.org/content/show.php/Kup+Backup+System?conte...), but I haven't tested it. For duplicity, there is a quite good ui in the form of deja dup (see http://www.howtogeek.com/108869/how-to-back-up-ubuntu-the-easy-way-with-dj-dup/ http://www.howtogeek.com/108869/how-to-back-up-ubuntu-the-ea...). It's really nice and easy to use, and if I recall correctly it's installed by default on Ubuntu.
wvh 12y ago
Being interested in Linux/Unix, file systems, crypto or whatever other technical topic doesn't necessarily mean you are also interested in developing graphical interfaces. That's an art form and discipline in itself. I suppose the people who make these tools are more interested in the tools' respective technical aspects than in interface or graphical design. Moreover, many of these backup tools run on servers (or as an automated background process) where a graphical interface is more of a handicap than an asset. I think that in open-source software "pretty interface" sounds often like "customer-oriented" which in turn sounds like "getting paid".

sciurus 12y ago

How does attic compare to obnam?

mappu 12y ago
I haven't used obnam much, the feature sets are quite similar, perhaps someone else can chip in here. My personal choice of attic over obnam was based on this performance comparison[1] 1. http://librelist.com/browser//attic/2014/1/31/attic-vs-obnam/ http://librelist.com/browser//attic/2014/1/31/attic-vs-obnam...

rsync 12y ago

From: https://attic-backup.org/quickstart.html#quickstart https://attic-backup.org/quickstart.html#quickstart "Attic can initialize and access repositories on remote hosts if the host is accessible using SSH." Fantastic. Will work perfectly here. We[1] are happy to support this just like we've supported duplicity all of these years. EDIT: appears obnam also works over plain old SSH. Can't tell about zbackup, however... As always, email us to discuss the "HN Readers" discount. [1] rsync.net

kijin 12y ago

Sorry, but "Untrusted remote storage" and "No server-side processing" are exactly the opposite of what I need. If the original box is ever compromised, I don't want the attacker to gain any access to the backup. If you use a dumb storage like S3 as your backup server, you need to store your keys on the original box, and anyone who gains control of the original box can destroy your S3 bucket as well. Ditto for any SSH-based backup scheme that requires keys to be stored on the original box. A compromised box could also lie about checksums, silently corrupting your backups. Backups should be pulled from the backup box, not pushed from the original box. Pushing backups is only acceptable for consumer devices, and even then, only because we don't have a reliable way to pull data from them (due to frequently changing IP addresses, NAT, etc). The backup box needs to be even more trustworthy than the original box, not the other way around. I'm willing to live with a significant amount of overhead, both in storage and in bandwidth, in order not to violate this principle. The backup box, of course, could push encrypted data to untrusted storage, such as S3. But only after it has pulled from the original box. In both cases, the connection is initiated from the backup box, not the other way around. The backup box never accepts any incoming connection. Does Attic support this kind of use case? The documentation doesn't seem to have anything to say about backing up remote files to local repositories. I don't see any reason why it won't be supported (since rsync does), but "nominally supported" is different from "optimized for that use case", and I suspect that many of the latest generation of backup tools are optimized for the opposite use case.

wmf 12y ago
What about append-only remote storage? This is possible (in a kludgy way) in S3: http://stackoverflow.com/questions/10592541/amazon-s3-acl-for-read-only-and-write-once-access/10592928 http://stackoverflow.com/questions/10592541/amazon-s3-acl-fo...
- kijin 12y ago
  Better, but it might still expose to the attacker more data than he would otherwise have access to. For example: production box only contains data from the last 3 days, but the backup contains data from the last 12 months. Even stricter access controls (write once, no read) might help with that. Not sure if you can do that with S3 though.

witten 12y ago

If someone is looking at trying attic and wants something a little nicer to configure and run than the very basic shell script from attic's Quick Start[1], check out the wrapper script I wrote to make this a little easier[2]. It's still somewhat "1.0" right now, but it does the basics for me. See the included sample config files for an idea of how the configuration works.[3] [1] https://attic-backup.org/quickstart.html#automating-backups https://attic-backup.org/quickstart.html#automating-backups [2] http://torsion.org/hg/atticmatic/file/tip http://torsion.org/hg/atticmatic/file/tip [3] http://torsion.org/hg/atticmatic/file/tip/sample/config http://torsion.org/hg/atticmatic/file/tip/sample/config

anarcat 12y ago
i'm curious: why isn't this part of attic in the first place? i have been working with the bup folks for a while to try to make a similar interface, and i was wondering what was your experience with upstream...
- witten 12y ago
  Great question. I do think this should be part of what attic provides out of the box, but I still really wanted to use attic despite the fact that it doesn't include this sort of functionality. I'll try contacting the attic devs and see what they say about it.
  - IgorPartola 12y ago
    Your wrapper is only lacking one critical feature I'd love. I am currently using rsnapshot and while its big issue is lack of encryption, it is able to run scripts on remote hosts to pull backups from them. This is a big deal to me since I can then script things like MySQL/Postgres backups, etc. on my master server, rather than having to configure each host individually. It's possible that this is a bad way to run things since my master server is then a SPoF. I do trust this server more since I monitor it much more closely, than I would a dozen random VPS's with a half-dozen different providers.

StavrosK 12y ago

I am also a very happy attic user and can attest to the wonderfulness that is it. Here's a blog post about it: http://www.stavros.io/posts/holy-grail-backups/ http://www.stavros.io/posts/holy-grail-backups/

anarcat 12y ago

from what i can tell here: https://github.com/c4rlo/attic/commit/f4804c07caac3d145f49fc1375c6448111381708 https://github.com/c4rlo/attic/commit/f4804c07caac3d145f49fc... ... attic updates the atime when opening files right now... that patche fixes the problem, but i have to wonder how fast it actually and accurate is. bup has all sorts of helpers and tweaks to make it really fast, but also to ensure that it doesn't touch the filesystem while making backups. i've been amazed to notice how all my files have atimes from july, when i stopped using rsync to make backups and switched to bup. :p

anarcat 12y ago

i was wondering how this would compare with bup. so i did a very quick benchmark: http://anarcat.koumbit.org/2014-11-18-bup-vs-attic-silly-benchmark http://anarcat.koumbit.org/2014-11-18-bup-vs-attic-silly-ben... take it with a grain of salt, but i am surprised to say that attic is at least as fast as bup.

INTPenis 12y ago
The only point that stuck in my eye was atime modification from attic but I realized that in my environment I would have to mount all the backup clients remotely on the backup server so I can get around this by mounting sshfs with noatime.

tenfingers 12y ago

Thanks for mentioning attic. Looks like an interesting project which slipped under my radar.