5 ms·

I have an 8TB hard drive in my desktop and an 8TB hard drive in a low power server. Btrfs to presumably detect bitrot at the filesystem level. They run syncthin

I have an 8TB hard drive in my desktop and an 8TB hard drive in a low power server. Btrfs to presumably detect bitrot at the filesystem level. They run syncthing to keep in sync. Both machines have external hard disk drive bays, and every so often I'll manually copy all my files to a 3rd disk as a cold backup. It seems like a good enough solution for me to not worry or think too hard about it. Syncthing isn't the most user friendly software though. When I upgraded from 2TB to 8TB drives, I accidentally a setting and ended up with about 1TB of duplicated "sync conflict" files. If I wasn't a salty software engineer comfortable writing arbitrarily complex scripts with free reign to delete files, it would have taken more than a weekend afternoon to clean that up...

edwinyzh 6y ago
What low power server do you mean here? I use a similar setup, but I'm on Windows with a folder-to-folder sync tool to sync to an external HDD which is connected through USB. I do have a 3rd HDD do make occasionally cold standalone backups.
- sgtnoodle 6y ago
  It's an Intel baytrail nano-ITX motherboard, with what I think is called a pico PSU, and an SSD for the operating system. It runs off a 12V 2A power brick. No fans or anything. The only component that isnt't solid state is the hard drive. I used various arm single board computers previously, with the hard drive connected via USB. When I started using syncthing, though, it was literally taking days to hash through all the files, only going 1-2 MB/sec. The UI wouldn't work until hours after restarting the daemon because it took so long just to build its data structures in RAM. The baytrail CPU has enough oomph to tear through the hashing, and it's just nice to have expandable RAM and SATA.
ahnick 6y ago
When you say you "copy all my files to a 3rd disk as a cold backup", do you mean you are overwriting any existing files on the 3rd disk? If so, I would recommend considering using something like Duplicacy (https://github.com/gilbertchen/duplicacy https://github.com/gilbertchen/duplicacy) to perform your backups with snapshots. The reason being is you can decide how long to keep older copies of files, just in case something changes on a file, but you don't happen to realize the incorrect update for a couple of months and want to go back to an earlier version. To give you an idea of what is possible, here is the snapshot configuration I use: Keep no snapshots older than 450 days Keep 1 snapshot every 90 day(s) if older than 180 day(s) Keep 1 snapshot every 30 day(s) if older than 30 day(s) Keep 1 snapshot every 1 day(s) if older than 1 day(s)
- sgtnoodle 6y ago
  I wipe the 3rd disk and then dumbly copy all my data to it again. My reasoning is that I want the complete volume of data actually copied as part of the backup activity. The source filesystem, being btrfs, should raise an error if there was bit rot, but that requires actually reading the file contents. If there is, I can grab the copy from the syncthing peer. The destination hard drive has to actually write all the file contents, so the magnetic fields should be all fresh. The 3rd hard disk is a cold backup in case syncthing has a common mode failure that takes out my redundant drives. I want its filesystem to be as boring as possible. For me that currently means ext4. I don't want to have to manage and worry about another tool that optimizes away data transfers, when there's value in the data transfer and plenty of copy bandwidth going SATA to SATA.
  - ahnick 6y ago
    Your current approach sounds fine from the perspective of bitrot. The hole in your approach right now is protecting from incorrect updates made either by yourself or by an application to your data that you don't notice immediately. For example, say you are using a financial application like GnuCash on your desktop and it has bug in the software that gets triggered causing a bad write to the file. (power goes out, OOM, whatever..) syncthing will happily propagate that changed file that contains the bad write to your server and when you copy to the 3rd disk you will also propagate that changed file. You deleted what was on the third disk, so now you no longer have a "good copy" of the file. If you add some sort of snapshotting into your backup routine, then you would be protected from this, because even though you would still propagate the change to the most recent backups, you could still go back to an older snapshot (maybe a week or a month ago or whatever) and pull back a working version of the file.
    - sgtnoodle 6y ago
      That certainly makes sense. In my case, I don't run any applications like that directly on my storage filesystem by default. Those sorts of files and directories, including my linux home directory, are on a non-redundant SSD. If there's a failure, I accept the risk of losing that data. 90% of the time the data either doesn't matter, or is redundantly stored somewhere else, such as a git repo. A typical use case is digital photos; I may dump them into my desktop directory at first so that I may view and sort them, and they'll live there for an arbitrary length of time. I simply won't delete them from their SD card until they're copied into the storage filesystem. Also, I have enough disks laying around that I usually have a 2nd cold backup to wipe. So, I have some inadvertent temporal redundancy going back a year or two. When I make a conscious decision to copy files over to my storage, I have piece of mind that I'll get physical redundancy within hours, and will get picked up by my cold backup eventually.

edwinyzh 6y ago

What low power server do you mean here? I use a similar setup, but I'm on Windows with a folder-to-folder sync tool to sync to an external HDD which is connected through USB. I do have a 3rd HDD do make occasionally cold standalone backups.

sgtnoodle 6y ago
It's an Intel baytrail nano-ITX motherboard, with what I think is called a pico PSU, and an SSD for the operating system. It runs off a 12V 2A power brick. No fans or anything. The only component that isnt't solid state is the hard drive. I used various arm single board computers previously, with the hard drive connected via USB. When I started using syncthing, though, it was literally taking days to hash through all the files, only going 1-2 MB/sec. The UI wouldn't work until hours after restarting the daemon because it took so long just to build its data structures in RAM. The baytrail CPU has enough oomph to tear through the hashing, and it's just nice to have expandable RAM and SATA.

ahnick 6y ago

When you say you "copy all my files to a 3rd disk as a cold backup", do you mean you are overwriting any existing files on the 3rd disk? If so, I would recommend considering using something like Duplicacy (https://github.com/gilbertchen/duplicacy https://github.com/gilbertchen/duplicacy) to perform your backups with snapshots. The reason being is you can decide how long to keep older copies of files, just in case something changes on a file, but you don't happen to realize the incorrect update for a couple of months and want to go back to an earlier version. To give you an idea of what is possible, here is the snapshot configuration I use: Keep no snapshots older than 450 days Keep 1 snapshot every 90 day(s) if older than 180 day(s) Keep 1 snapshot every 30 day(s) if older than 30 day(s) Keep 1 snapshot every 1 day(s) if older than 1 day(s)

sgtnoodle 6y ago
I wipe the 3rd disk and then dumbly copy all my data to it again. My reasoning is that I want the complete volume of data actually copied as part of the backup activity. The source filesystem, being btrfs, should raise an error if there was bit rot, but that requires actually reading the file contents. If there is, I can grab the copy from the syncthing peer. The destination hard drive has to actually write all the file contents, so the magnetic fields should be all fresh. The 3rd hard disk is a cold backup in case syncthing has a common mode failure that takes out my redundant drives. I want its filesystem to be as boring as possible. For me that currently means ext4. I don't want to have to manage and worry about another tool that optimizes away data transfers, when there's value in the data transfer and plenty of copy bandwidth going SATA to SATA.
- ahnick 6y ago
  Your current approach sounds fine from the perspective of bitrot. The hole in your approach right now is protecting from incorrect updates made either by yourself or by an application to your data that you don't notice immediately. For example, say you are using a financial application like GnuCash on your desktop and it has bug in the software that gets triggered causing a bad write to the file. (power goes out, OOM, whatever..) syncthing will happily propagate that changed file that contains the bad write to your server and when you copy to the 3rd disk you will also propagate that changed file. You deleted what was on the third disk, so now you no longer have a "good copy" of the file. If you add some sort of snapshotting into your backup routine, then you would be protected from this, because even though you would still propagate the change to the most recent backups, you could still go back to an older snapshot (maybe a week or a month ago or whatever) and pull back a working version of the file.
  - sgtnoodle 6y ago
    That certainly makes sense. In my case, I don't run any applications like that directly on my storage filesystem by default. Those sorts of files and directories, including my linux home directory, are on a non-redundant SSD. If there's a failure, I accept the risk of losing that data. 90% of the time the data either doesn't matter, or is redundantly stored somewhere else, such as a git repo. A typical use case is digital photos; I may dump them into my desktop directory at first so that I may view and sort them, and they'll live there for an arbitrary length of time. I simply won't delete them from their SD card until they're copied into the storage filesystem. Also, I have enough disks laying around that I usually have a 2nd cold backup to wipe. So, I have some inadvertent temporal redundancy going back a year or two. When I make a conscious decision to copy files over to my storage, I have piece of mind that I'll get physical redundancy within hours, and will get picked up by my cold backup eventually.