7 ms·

There's so many backup systems out there these days, but none that are quite perfect. The closest I've found is `rdedup`[1], which may no longer be maintained.

by frio 7y ago

There's so many backup systems out there these days, but none that are quite perfect. The closest I've found is `rdedup`[1], which may no longer be maintained. What I'd like in a backup solution is: - compressed - deduplicated - encrypted using asymmetric encryption, so that the encrypting machine doesn't need to know a password - durable (using par2 or directly supporting repository repair from an independent backup)/signed - composed of standard open source tools so that if the backup software goes away, everything can still be retrieved - supports differing upstream data hosting providers - open source There's lots of things that fill most of those requirements, but none that tick all the boxes. - rdiff-backup does all of the above except deduplication; it's full + incremental which requires a management strategy - restic doesn't compress, doesn't use asymmetric encryption and doesn't use par2 or similar tools for durability - borg doesn't use asymmetric encryption and doesn't use par2/durability stuff; it's also pretty slow I know people call borg the "holy grail", but I think we're still a wee way off that emerging, personally -- even though I do really like (and use!) borg. [1]: https://github.com/dpc/rdedup https://github.com/dpc/rdedup

cookiecaper 7y ago
Yeah, I've been searching for a great general archiver for a long time. I tried borgbackup a couple of years ago and immediately hit a handful of issues that put me off, though I can't really recall the specific technical details clearly (I'm vaguely recollecting something about incompatible block sizes). Whatever the details were, I came away with the distinct impression that the borgbackup developers didn't really respect the gravity of archived records and long-term storage. While that's probably not a totally fair conclusion from a one-shot use, in my defense, I'll offer up the latest version of their changelog [0], where: * the first thing listed is an apparently serious data corruption bug that lived through several stable releases * the second thing listed is an apparently-very-serious security vulnerability ("a flaw in the cryptographic authentication scheme used in Borg allowed an attacker to spoof the manifest ...") * the third thing listed is another data corruption bug titled "Pre-1.0.9 data loss" * the fourth thing listed is another data corruption bug titled "Pre-1.0.4 potential repo corruption" Note these are all post-1.0 versions. To be frank, I dare not scroll further. Users are depending on this software to safeguard important files with the assumption that it will be able to reproduce them bit-for-bit-intact some years down the road. Any long-term storage software requires developers with a fanatical devotion to compatibility, longevity, and integrity if it's ever going to be more than a toy. I hate to pile on open-source devs who're trying their hand at making software that a lot of people appreciate, but overall, it's hard for me to say I regret the choice to pass it up and stick to combinations of ZFS, rsync, and the trusty old tarball. [0] https://github.com/borgbackup/borg/commit/75dcf9356334188276d095d818e22cb4b437a456 https://github.com/borgbackup/borg/commit/75dcf9356334188276...
- viraptor 7y ago
  I still don't know how to really judge software from changelogs, but I think it's really hard to do and this post is not a great example either. This is a changelog backup software which does data storage and encryption mainly. If it ever has bugs worth talking about they're going to be in... data storage and encryption. If we look at rsync for a change which is much older, simpler and mature: https://download.samba.org/pub/rsync/src/rsync-3.1.1-NEWS https://download.samba.org/pub/rsync/src/rsync-3.1.1-NEWS - fixing .. traversal in V3 (how did that survive so long?) - issues with copying attributes (There won't be much data corruption since rsync does 1:1 copies) Time-to-fix could give us a better idea maybe? Either way, software doing X has bugs in X is "normal".
  - cookiecaper 7y ago
    I mean, the issue isn't so much that bugs exist -- like you say, that's par for the course, even among the best developers. It's about a large incidence of severe eat-your-data bugs in such a short span of time (roughly a year) in something that's quote-unquote stable and whose entire purpose is long-term data storage and retrieval. It's even worse because unlike rsync, which synchronizes filetrees from point A to B and can be immediately confirmed to have either worked correctly or not, borg uses a bespoke storage format that's not easily verified or operated upon by standard utilities. You just have to trust it to pull the correct data out when you need it. That's a heavy burden to put on a tool, and the borg devs seem to be struggling under its weight. Consider the standards expected of filesystem or database maintainers. borg repos are not really that different -- they're a big, opaque chunks of bytes that you expect to be able to produce specific data on demand with perfect reliability. These types of systems don't get marked stable when they're still in primordial, eat-your-data development phases, and on the small handful of unfortunate occasions when data loss bugs sneak in, they're a) usually limited to some bizarre corner case; and b) taken extremely seriously, almost to the point of solemnity, and often result in major overhauls to a project's validation and QA routine. If any stable filesystem or database had 4 widely-applicable database corruption bugs within a year, it'd be lights out for that project. All trust lost, reputation irreparably ruined, angry letters to a variety of mailing lists, permanently-increased scrutiny on any new projects or maintainers among the same class of projects, etc. All I'm saying is that based on my admittedly-murky prior experience, the project's observed track record, and the high standard of care that must be met to qualify a project for storing authoritative copies of data, I'm personally not comfortable trusting borg with anything important any time soon. No one is obliged to share that evaluation, of course.
4gotunameagain 7y ago
What about rclone? Not advocating, just want your opinion since you seem like you researched the topic
- frio 7y ago
  rclone is rsync-on-steroids, and you can use it for backups, but it's not the same shape of tool as the above tools. It's effectively "copy these files from $here to $there" where $here and $there are any cloud provider or network filesystem. It's got a built-in encryption layer and some other bits, but you'd still be responsible for ensuring $there is actually $there/backups/(date +%F) or whatever. I'd treat it as a tool in a bigger backup system (which is roughly what rdedup does with it).
- Erlich_Bachman 7y ago
  Rclone is not a backup system, it is a sync system. It does not do any archiving, deduplication, pruning, etc. It just syncs two directories (with possible encryption added on top). It can be efficiently utilized as an extra final step in a backup plan.
bradknowles 7y ago
Don't you want to dedup first, then compress?
- frio 7y ago
  Probably, yes. Sorry, I should’ve said “in no particular order” — this is part of the reason why I haven’t tried to tackle building something myself :).
- cben 7y ago
  Well, with `gzip --rsyncable` and similar (most compression tools have such a mode), it's technically possible to compress first then dedup! To be precise it's "chunk first, then compress", which still admits a 2nd "chunk, then dedup" pass to reuse data, but not as efficiently as a combined "chunk+dedup+compress each chunk"... `gzip --rsyncable` and friends are mostly useful for when you already have compressed files on your filesystem, and want to sync/backup them byte-for-byte. Of course the benefit is lost unless most compressed files in the world are produced with such mode, and sadly most aren't :-( The author of `zsync` did various experiments confirming gzip --rsyncable then sync is sub-optimal, and implemented a somewhat crazy "look inside" approach that can sync compressed files byte-for-byte AND very efficiently: > gzip --rsync does fairly well, with both rsync and zsync transferring about 410kB at the optimum point. zsync with the look-inside method does much better than either of these, with as little as 140K transferred. > -- http://zsync.moria.org.uk/paper/ch03s03.html http://zsync.moria.org.uk/paper/ch03s03.html IIUC though it's more of a "2-files sync" scenario like rsync, not applicable to "chunk everything then dedup" approach of borg and similar tools.
frio 7y ago
Too late to edit now, but I meant "duplicity" rather than "rdiff-backup" in the above post. Woops.
- Erlich_Bachman 7y ago
  Comparing duplicity to something a deduplicating software suite is not really correct, they don't even provide the same functionality (like mounting snapshots, like handling moving around of the files on the hard drive, like space-efficiently backing up files from different machines, like smart and efficient pruning, like speed of access(impossible for duplicity to achieve), etc.).

by frio 7y ago

cookiecaper 7y ago

Yeah, I've been searching for a great general archiver for a long time. I tried borgbackup a couple of years ago and immediately hit a handful of issues that put me off, though I can't really recall the specific technical details clearly (I'm vaguely recollecting something about incompatible block sizes). Whatever the details were, I came away with the distinct impression that the borgbackup developers didn't really respect the gravity of archived records and long-term storage. While that's probably not a totally fair conclusion from a one-shot use, in my defense, I'll offer up the latest version of their changelog [0], where: * the first thing listed is an apparently serious data corruption bug that lived through several stable releases * the second thing listed is an apparently-very-serious security vulnerability ("a flaw in the cryptographic authentication scheme used in Borg allowed an attacker to spoof the manifest ...") * the third thing listed is another data corruption bug titled "Pre-1.0.9 data loss" * the fourth thing listed is another data corruption bug titled "Pre-1.0.4 potential repo corruption" Note these are all post-1.0 versions. To be frank, I dare not scroll further. Users are depending on this software to safeguard important files with the assumption that it will be able to reproduce them bit-for-bit-intact some years down the road. Any long-term storage software requires developers with a fanatical devotion to compatibility, longevity, and integrity if it's ever going to be more than a toy. I hate to pile on open-source devs who're trying their hand at making software that a lot of people appreciate, but overall, it's hard for me to say I regret the choice to pass it up and stick to combinations of ZFS, rsync, and the trusty old tarball. [0] https://github.com/borgbackup/borg/commit/75dcf9356334188276d095d818e22cb4b437a456 https://github.com/borgbackup/borg/commit/75dcf9356334188276...

viraptor 7y ago
I still don't know how to really judge software from changelogs, but I think it's really hard to do and this post is not a great example either. This is a changelog backup software which does data storage and encryption mainly. If it ever has bugs worth talking about they're going to be in... data storage and encryption. If we look at rsync for a change which is much older, simpler and mature: https://download.samba.org/pub/rsync/src/rsync-3.1.1-NEWS https://download.samba.org/pub/rsync/src/rsync-3.1.1-NEWS - fixing .. traversal in V3 (how did that survive so long?) - issues with copying attributes (There won't be much data corruption since rsync does 1:1 copies) Time-to-fix could give us a better idea maybe? Either way, software doing X has bugs in X is "normal".
- cookiecaper 7y ago
  I mean, the issue isn't so much that bugs exist -- like you say, that's par for the course, even among the best developers. It's about a large incidence of severe eat-your-data bugs in such a short span of time (roughly a year) in something that's quote-unquote stable and whose entire purpose is long-term data storage and retrieval. It's even worse because unlike rsync, which synchronizes filetrees from point A to B and can be immediately confirmed to have either worked correctly or not, borg uses a bespoke storage format that's not easily verified or operated upon by standard utilities. You just have to trust it to pull the correct data out when you need it. That's a heavy burden to put on a tool, and the borg devs seem to be struggling under its weight. Consider the standards expected of filesystem or database maintainers. borg repos are not really that different -- they're a big, opaque chunks of bytes that you expect to be able to produce specific data on demand with perfect reliability. These types of systems don't get marked stable when they're still in primordial, eat-your-data development phases, and on the small handful of unfortunate occasions when data loss bugs sneak in, they're a) usually limited to some bizarre corner case; and b) taken extremely seriously, almost to the point of solemnity, and often result in major overhauls to a project's validation and QA routine. If any stable filesystem or database had 4 widely-applicable database corruption bugs within a year, it'd be lights out for that project. All trust lost, reputation irreparably ruined, angry letters to a variety of mailing lists, permanently-increased scrutiny on any new projects or maintainers among the same class of projects, etc. All I'm saying is that based on my admittedly-murky prior experience, the project's observed track record, and the high standard of care that must be met to qualify a project for storing authoritative copies of data, I'm personally not comfortable trusting borg with anything important any time soon. No one is obliged to share that evaluation, of course.

4gotunameagain 7y ago

What about rclone? Not advocating, just want your opinion since you seem like you researched the topic

frio 7y ago
rclone is rsync-on-steroids, and you can use it for backups, but it's not the same shape of tool as the above tools. It's effectively "copy these files from $here to $there" where $here and $there are any cloud provider or network filesystem. It's got a built-in encryption layer and some other bits, but you'd still be responsible for ensuring $there is actually $there/backups/(date +%F) or whatever. I'd treat it as a tool in a bigger backup system (which is roughly what rdedup does with it).
Erlich_Bachman 7y ago
Rclone is not a backup system, it is a sync system. It does not do any archiving, deduplication, pruning, etc. It just syncs two directories (with possible encryption added on top). It can be efficiently utilized as an extra final step in a backup plan.

bradknowles 7y ago

Don't you want to dedup first, then compress?

frio 7y ago
Probably, yes. Sorry, I should’ve said “in no particular order” — this is part of the reason why I haven’t tried to tackle building something myself :).
cben 7y ago
Well, with `gzip --rsyncable` and similar (most compression tools have such a mode), it's technically possible to compress first then dedup! To be precise it's "chunk first, then compress", which still admits a 2nd "chunk, then dedup" pass to reuse data, but not as efficiently as a combined "chunk+dedup+compress each chunk"... `gzip --rsyncable` and friends are mostly useful for when you already have compressed files on your filesystem, and want to sync/backup them byte-for-byte. Of course the benefit is lost unless most compressed files in the world are produced with such mode, and sadly most aren't :-( The author of `zsync` did various experiments confirming gzip --rsyncable then sync is sub-optimal, and implemented a somewhat crazy "look inside" approach that can sync compressed files byte-for-byte AND very efficiently: > gzip --rsync does fairly well, with both rsync and zsync transferring about 410kB at the optimum point. zsync with the look-inside method does much better than either of these, with as little as 140K transferred. > -- http://zsync.moria.org.uk/paper/ch03s03.html http://zsync.moria.org.uk/paper/ch03s03.html IIUC though it's more of a "2-files sync" scenario like rsync, not applicable to "chunk everything then dedup" approach of borg and similar tools.

frio 7y ago

Too late to edit now, but I meant "duplicity" rather than "rdiff-backup" in the above post. Woops.

Erlich_Bachman 7y ago
Comparing duplicity to something a deduplicating software suite is not really correct, they don't even provide the same functionality (like mounting snapshots, like handling moving around of the files on the hard drive, like space-efficiently backing up files from different machines, like smart and efficient pruning, like speed of access(impossible for duplicity to achieve), etc.).