5 ms·

Does anyone have a recommendation for a backup client that handles millions of tiny files? I'm using rsnapshot right now, which works but backing up to an NFS s

by a2tech 7y ago

Does anyone have a recommendation for a backup client that handles millions of tiny files? I'm using rsnapshot right now, which works but backing up to an NFS share is incredibly slow (most of the time is spent in iterating over the filesystem to get a list of changed files, then running the hardlink process from the previous snapshot).

dsign 7y ago
Over the years, I've got burn so many times with Linux network filesystems (including SMB/CIFFS, which is not all Linux but still) that I would start by recommending not to touch a network filesystem in the same host where anything important happens. The issue is with how network failures (which always eventually happen) interact with the "uninterruptible" Linux process state. Hell breaks loose, and the failure is anything but obvious.
rsync 7y ago
You're going to have to walk all those inodes no matter what you do. rsync is as good as anything at that task. A better way would be to unmount and send the filesystem with 'dd' or something like that, or, to use 'zfs send' but I have a suspicion that neither of those options are available to you ... I will say that splitting the rsync job (rsnapshot runs rsync underneath) into multiple, smaller jobs, could save you some time if you're running into any resource limits while you walk that big set of inodes... so if you're lucky and you have 4 or 5 or 8 top level dirs that are all roughly the same size, you could do a handful of smaller jobs, one after the other, instead of one huge one ...
- module0000 7y ago
  >> A better way would be to unmount and send the filesystem with 'dd' or something like that To add to that.. to avoid having to unmount your filesystem; use LVM. Then you can call `sync`, snapshot your main volume, and `dd` the clean snapshot. Once you're done, remove the snapshot. This strategy avoids downtime while backing up your volume.
realusername 7y ago
I use duplicity for that personally, I backup my whole home with it (and there's so much small files like this) it works very well.
- rebelpixel 7y ago
  I'm currently using restic and used duplicity before for a few years, sending backups to backblaze b2. What I don't like about duplicity is how it spews weird error messages that I gathered were related to Python versions. It was easier to just start over with restic and haven't had any problems since.
kijin 7y ago
Run backups to and from the hosts where the files actually reside. Running rsync over ssh is probably many times faster than running it over NFS.
deleted 7y ago
[deleted]

by a2tech 7y ago

dsign 7y ago

Over the years, I've got burn so many times with Linux network filesystems (including SMB/CIFFS, which is not all Linux but still) that I would start by recommending not to touch a network filesystem in the same host where anything important happens. The issue is with how network failures (which always eventually happen) interact with the "uninterruptible" Linux process state. Hell breaks loose, and the failure is anything but obvious.

rsync 7y ago

You're going to have to walk all those inodes no matter what you do. rsync is as good as anything at that task. A better way would be to unmount and send the filesystem with 'dd' or something like that, or, to use 'zfs send' but I have a suspicion that neither of those options are available to you ... I will say that splitting the rsync job (rsnapshot runs rsync underneath) into multiple, smaller jobs, could save you some time if you're running into any resource limits while you walk that big set of inodes... so if you're lucky and you have 4 or 5 or 8 top level dirs that are all roughly the same size, you could do a handful of smaller jobs, one after the other, instead of one huge one ...

module0000 7y ago
>> A better way would be to unmount and send the filesystem with 'dd' or something like that To add to that.. to avoid having to unmount your filesystem; use LVM. Then you can call `sync`, snapshot your main volume, and `dd` the clean snapshot. Once you're done, remove the snapshot. This strategy avoids downtime while backing up your volume.

realusername 7y ago

I use duplicity for that personally, I backup my whole home with it (and there's so much small files like this) it works very well.

rebelpixel 7y ago
I'm currently using restic and used duplicity before for a few years, sending backups to backblaze b2. What I don't like about duplicity is how it spews weird error messages that I gathered were related to Python versions. It was easier to just start over with restic and haven't had any problems since.

kijin 7y ago

Run backups to and from the hosts where the files actually reside. Running rsync over ssh is probably many times faster than running it over NFS.

deleted 7y ago

[deleted]