3 ms·
> Sadly, the only thing I've found so far that works at all at those scales is Bacula, and that is file-based -- i.e., if you have a gigabyte file that changes
by wolfhumble 8y ago
> Sadly, the only thing I've found so far that works at all at those scales is Bacula, and that is file-based -- i.e., if you have a gigabyte file that changes by one byte, it backs up the whole gigabyte again. Not ideal.
Would it possible to split those bigger files before doing backup (log files with log splitters, sql databases with incremental specialised backup tools like e.g. xtrabackup), or are these e.g. image files with different versions?
I really like the concept of byte level deduplication, but have often thought the price you pay for the space reduction might not be worth it considering today's network speed as well as storage sizes and prices.
Would be interesting to hear your experience concerning this!
- dmd 8y agoI'm definitely looking into what I can do as far as getting logs rotated (and thus split) better, but a lot of what I back up are legacy medical systems where making any kind of change is extremely difficult both technically and politically.
- wolfhumble 8y agoOkay, I see. Thanks for the feedback!
- dspillett 8y agoA slightly hacky method might be an overlay filesystem (using FUSE or similar) that mirrors the underlying filesystem for small files but represents the larger ones as smaller units (so bigfile becomes bigfile.block0000, bigfile.block0001, ...). That way only the changed block would get transferred if you one-byte change is a modify or append rather than on insert or delete. If the backup service is using file timestamps as the only key to refresh then it would have to store the last modification date and a good hash of each block, and when the date of the file being mirrored is updated scan each block to see if there is a change there and update the stored hash and timestamps accordingly. This would need to be orchestrated to reduce the risk of temporary corruption if there are several updates and the rechecking process coincides with a backup sweep (i.e. make sure you don't present updated dates for any block until you can present them for all needed). For restoration, you either manually concatenate the parts or have an overlay filesystem that operates in reverse: showing the smaller block files as a single large unit. You'd have to very thoroughly test the overlays and their interaction with the backup service before risking it on important data, so it might not be something you would genuinely consider...
- dmd 8y agoDude. That’s more than just “slightly” hacky. Points for the idea though.
- dspillett 8y agoManic me has some wonderful ideas that are simultaneously excellent and terrible. I have to keep a careful eye on him in the day job! One of these days I intend to implement and collect a few of his musings under the banner "the ministry of silly code".