2 ms·

Hmm, anyone know if they fixed some of the exponential time issues it was having? I was really excited about some of the features and tried it a couple years ag

Hmm, anyone know if they fixed some of the exponential time issues it was having? I was really excited about some of the features and tried it a couple years ago, and it died before getting very far sync'ing my NAS to a usb3 jbod I plugged directly in to prime it for a rpi4 network attached backup. For something like that I would expect the backup speed to be a few hundred MB/sec and it quickly was just running in the 10's of MB/sec and getting slower. I've noted this before that my NAS has a tendency to kill a lot of backup applications that haven't really been stressed. Its about 50 something TB of data, made up of a fair number of compressed video files, family movie kinds of things and about ~7T of source files. The combination of a crapton of tiny files and 50T of data kills most of these more recent open source backup applications, which seem to be released with the "it can backup my laptop" testing. Also, I see someone mentions it sill doesn't have a compressed block option, which I really want because most of those source files compress to about 1/3rd of their space, which is important since my upload speeds are crummy (thanks spectrum!) I wonder how much of that deficiency is just lack of good go compression routines that can run at a few hundred MB/sec.

johntash 5y ago
My main dataset that I back up with restic is around 11 million files or so, and fluctuates between 8-15TB of source data. I think I've run into similar issues testing out various backup tools too, but have settled on restic for now since it's been the most reliable. I'd definitely recommend trying restic again if it's been a couple years. Somewhat recently they made some nice speed improvements. It used to take me several days to do a forget+purge on my restic repo. Now it takes only a few hours (less than an actual backup takes). How many files are you backing up? I found the biggest issue for me was that almost every tool tries to keep the list of files in memory, so once you get into millions of files - it starts to require a lot more memory and can crash on low-resource machines.

johntash 5y ago

My main dataset that I back up with restic is around 11 million files or so, and fluctuates between 8-15TB of source data. I think I've run into similar issues testing out various backup tools too, but have settled on restic for now since it's been the most reliable. I'd definitely recommend trying restic again if it's been a couple years. Somewhat recently they made some nice speed improvements. It used to take me several days to do a forget+purge on my restic repo. Now it takes only a few hours (less than an actual backup takes). How many files are you backing up? I found the biggest issue for me was that almost every tool tries to keep the list of files in memory, so once you get into millions of files - it starts to require a lot more memory and can crash on low-resource machines.