3 ms·

To echo the other comment, please use restic or borg instead of this. For being a "simple" tool, it appears more complicated and with more questionably useful

To echo the other comment, please use restic or borg instead of this. For being a "simple" tool, it appears more complicated and with more questionably useful features than restic or borg. Borg and restic have server components that allow clients to send backups, I believe restic also allows the clients to manage encryption/not trust the server.

derekp7 6y ago
The "simpleness" in Snebu comes from using standardized tools such as find and tar, file storage using LZOP, and the backup catalog in a standard SQLite3 DB, allowing manual recovery of files. However Restic and Borg do have the advantage of writing directly to cloud backups or "dumb" backend data stores such as sftp targets, or API-accessed cloud targets (although Restic doesn't compress and Borg has issues with multiple hosts going to the same repository, they do still have their use cases). Whereas Snebu is more of the use case that would be solved by more complicated to manage tools such as Bacula, Amanda, or commercial backup solutions but with lower setup and administrative requirements.
- deleted 6y ago
  [deleted]
- Aachen 6y ago
  > Restic doesn't compress I didn't actually know that, that's kinda weird. It doesn't seem to be vital to deduplication, looking at https://github.com/restic/restic/pull/2441 https://github.com/restic/restic/pull/2441, they just didn't originally implement it and are now wondering how to best introduce this breaking change (can be forwards compatible, but of course not backwards), trying to get it right in one try, keeping the format "robust", adding other breaking changes while they're at it... wonder if this will happen any time soon. To be honest, though, 98% of the time large files (anything over a few hundred kilobytes) is in a format that compresses to a certain extent. Pictures, audio, even text documents and spreadsheets are compressed these days. One person mentions huge XML files which sounds like they really should have been using a different tool (not a text file for starters, if you're not going to read through those gigabytes of text anyway), but there are other use cases like somewhat sparse database tables or disk images or so. Then again, if you have the space for those original files, doubling the space for a reserve copy is usually not a deal breaker, and Restic also deduplicates within and between files so if something were to be very redundant and compressible it is also likely to be caught by that algorithm. I'm not against increasing the complexity to add it, and breaking compatibility of course---it's a clear downside of Restic to not have this---but I also don't really get anyone who considers it a deal breaker (and only to an extent those that find it a big deal). Either way, thanks for pointing this out, I didn't know (and made an incorrect assumption since it seems so standard) and it's good to know about.
ajsnigrutin 6y ago
For a simple tool, use rsnapshot ( https://rsnapshot.org/ https://rsnapshot.org/ ). Pull system, which rsyncs (over ssh) the data from machones to folders on a server, then, on the next day, creates a hard-link copy of the backup folders, to get a separate "yesterdays" version, and rsyncs over the "todays", to get the new "todays". ...plus some scripts to do a weekly,and monthly versions and to keep the number of backups at a desired number (config). Only downside is, that it needs root access to rsync whole filesystems, and that you need to manually set the excludes for each machine.
- derekp7 6y ago
  That was actually the original inspiration for Snebu (not rsnapshot specifically, but a custom script implementing similar functionality based on the original rsync snapshot paper). Snapshots are so much better than level-based backups (full/differential/incremental) when using disk-backed storage. I ended up running into this problem: https://news.ycombinator.com/item?id=8305283 https://news.ycombinator.com/item?id=8305283 when I got to 20 hosts and a year's worth of backups (14 daily, 6 weekly, and 12 monthly snapshots, so 32 snapshots per hosts). It was made worse in my case because I was using ZFS with compression. So the original implementation of Snebu used shell scripts, flat files for the catalog, and a small utility to burst apart tar files.
- rsync 6y ago
  "For a simple tool, use rsnapshot ..." Even simpler is to do a dumb, 1:1 mirror, with rsync and then rotate zfs snapshots on the server side. Two benefits here over rsnapshot (or rsync snapshots, with hardlinks) are: 1) rsync snapshots diff on a file granularity - if you change one bit of the file, your penalty is the storage space for that entire file. ZFS, on the other hand, diffs on a bits level so your penalty is just the changed portions of that paticular file. 2) Depending on your implementation, ZFS snapshots are immutable/read-only ... so neither a misconfiguration nor ransomware/mallory can destroy those backups if you lose your credentials. Historically, people would create and maintain "rsync snapshots" inside their rsync.net accounts using 'cp -al' and other unix commands run over SSH.[1] After transitioning to ZFS, however, we encourage all customers to just do a "dumb" 1:1 rsync to us and let us maintain and rotate their preferred snapshot schedule (days/weeks/months/quarters/years). [1] https://www.rsync.net/resources/howto/remote_commands.html https://www.rsync.net/resources/howto/remote_commands.html

derekp7 6y ago

The "simpleness" in Snebu comes from using standardized tools such as find and tar, file storage using LZOP, and the backup catalog in a standard SQLite3 DB, allowing manual recovery of files. However Restic and Borg do have the advantage of writing directly to cloud backups or "dumb" backend data stores such as sftp targets, or API-accessed cloud targets (although Restic doesn't compress and Borg has issues with multiple hosts going to the same repository, they do still have their use cases). Whereas Snebu is more of the use case that would be solved by more complicated to manage tools such as Bacula, Amanda, or commercial backup solutions but with lower setup and administrative requirements.

deleted 6y ago
[deleted]
Aachen 6y ago
> Restic doesn't compress I didn't actually know that, that's kinda weird. It doesn't seem to be vital to deduplication, looking at https://github.com/restic/restic/pull/2441 https://github.com/restic/restic/pull/2441, they just didn't originally implement it and are now wondering how to best introduce this breaking change (can be forwards compatible, but of course not backwards), trying to get it right in one try, keeping the format "robust", adding other breaking changes while they're at it... wonder if this will happen any time soon. To be honest, though, 98% of the time large files (anything over a few hundred kilobytes) is in a format that compresses to a certain extent. Pictures, audio, even text documents and spreadsheets are compressed these days. One person mentions huge XML files which sounds like they really should have been using a different tool (not a text file for starters, if you're not going to read through those gigabytes of text anyway), but there are other use cases like somewhat sparse database tables or disk images or so. Then again, if you have the space for those original files, doubling the space for a reserve copy is usually not a deal breaker, and Restic also deduplicates within and between files so if something were to be very redundant and compressible it is also likely to be caught by that algorithm. I'm not against increasing the complexity to add it, and breaking compatibility of course---it's a clear downside of Restic to not have this---but I also don't really get anyone who considers it a deal breaker (and only to an extent those that find it a big deal). Either way, thanks for pointing this out, I didn't know (and made an incorrect assumption since it seems so standard) and it's good to know about.

ajsnigrutin 6y ago

For a simple tool, use rsnapshot ( https://rsnapshot.org/ https://rsnapshot.org/ ). Pull system, which rsyncs (over ssh) the data from machones to folders on a server, then, on the next day, creates a hard-link copy of the backup folders, to get a separate "yesterdays" version, and rsyncs over the "todays", to get the new "todays". ...plus some scripts to do a weekly,and monthly versions and to keep the number of backups at a desired number (config). Only downside is, that it needs root access to rsync whole filesystems, and that you need to manually set the excludes for each machine.

derekp7 6y ago
That was actually the original inspiration for Snebu (not rsnapshot specifically, but a custom script implementing similar functionality based on the original rsync snapshot paper). Snapshots are so much better than level-based backups (full/differential/incremental) when using disk-backed storage. I ended up running into this problem: https://news.ycombinator.com/item?id=8305283 https://news.ycombinator.com/item?id=8305283 when I got to 20 hosts and a year's worth of backups (14 daily, 6 weekly, and 12 monthly snapshots, so 32 snapshots per hosts). It was made worse in my case because I was using ZFS with compression. So the original implementation of Snebu used shell scripts, flat files for the catalog, and a small utility to burst apart tar files.
rsync 6y ago
"For a simple tool, use rsnapshot ..." Even simpler is to do a dumb, 1:1 mirror, with rsync and then rotate zfs snapshots on the server side. Two benefits here over rsnapshot (or rsync snapshots, with hardlinks) are: 1) rsync snapshots diff on a file granularity - if you change one bit of the file, your penalty is the storage space for that entire file. ZFS, on the other hand, diffs on a bits level so your penalty is just the changed portions of that paticular file. 2) Depending on your implementation, ZFS snapshots are immutable/read-only ... so neither a misconfiguration nor ransomware/mallory can destroy those backups if you lose your credentials. Historically, people would create and maintain "rsync snapshots" inside their rsync.net accounts using 'cp -al' and other unix commands run over SSH.[1] After transitioning to ZFS, however, we encourage all customers to just do a "dumb" 1:1 rsync to us and let us maintain and rotate their preferred snapshot schedule (days/weeks/months/quarters/years). [1] https://www.rsync.net/resources/howto/remote_commands.html https://www.rsync.net/resources/howto/remote_commands.html