3 ms·
In my testing (early 2019), NONE of restic, rclone, borg, duplicity, or tarsnap was able to handle anything other than hobbyist workloads. restic fell over har
by dmd 7y ago
In my testing (early 2019), NONE of restic, rclone, borg, duplicity, or tarsnap was able to handle anything other than hobbyist workloads.
restic fell over hard at around 100 terabytes; the others at around 500.
I'm backing up a little over 3 petabytes. I use Bacula. It's awful, but I haven't found anything else that can deal with that kind of volume.
- zmix 7y ago> I use Bacula. Did you test 'Burp'[1]? [1] https://burp.grke.org/why.html https://burp.grke.org/why.html
- JeremyNT 7y agoWhen I was backing up several 100 tb, I tried every open source option at the time and found them all lacking. What ultimately worked? Plain old rsync over ssh, to a zfs pool with snapshots and compression. As far as I could figure, the only notable downside of this is that the storage device must be trusted, since it has access to all of the data, and that you effectively needed root permissions on the storage you're copying to for filesystem permissions which makes a multi tenant backup server cumbersome (you could chroot or use containers or something but these solutions can become fiddly, e.g. running multiple instances of ssh on nonstandard ports to enable multitenancy).
- jmiserez 7y agoDoesn't rsync also slow to a crawl if there are too many files, due to memory requirements? Is there a way around that? EDIT: Ah I see rsync 3.0.0 (released in 2008) fixed this issue. Maybe I was on an old version.
- dmd 7y agoI've been very tempted to dump bacula and do that, especially since my main disk-based storage is an Oracle ZFS appliance. But I also do backups to tape (LTO-8, in a Storagetek library), and I do like that Bacula handles that for me.
- aleph- 7y agoSo restic has at least gotten somewhat better with several upstream PRs getting us past 100TB usability.