3 ms·

When you say you "copy all my files to a 3rd disk as a cold backup", do you mean you are overwriting any existing files on the 3rd disk? If so, I would recomme

by ahnick 6y ago

When you say you "copy all my files to a 3rd disk as a cold backup", do you mean you are overwriting any existing files on the 3rd disk? If so, I would recommend considering using something like Duplicacy (https://github.com/gilbertchen/duplicacy https://github.com/gilbertchen/duplicacy) to perform your backups with snapshots. The reason being is you can decide how long to keep older copies of files, just in case something changes on a file, but you don't happen to realize the incorrect update for a couple of months and want to go back to an earlier version. To give you an idea of what is possible, here is the snapshot configuration I use: Keep no snapshots older than 450 days Keep 1 snapshot every 90 day(s) if older than 180 day(s) Keep 1 snapshot every 30 day(s) if older than 30 day(s) Keep 1 snapshot every 1 day(s) if older than 1 day(s)

sgtnoodle 6y ago
I wipe the 3rd disk and then dumbly copy all my data to it again. My reasoning is that I want the complete volume of data actually copied as part of the backup activity. The source filesystem, being btrfs, should raise an error if there was bit rot, but that requires actually reading the file contents. If there is, I can grab the copy from the syncthing peer. The destination hard drive has to actually write all the file contents, so the magnetic fields should be all fresh. The 3rd hard disk is a cold backup in case syncthing has a common mode failure that takes out my redundant drives. I want its filesystem to be as boring as possible. For me that currently means ext4. I don't want to have to manage and worry about another tool that optimizes away data transfers, when there's value in the data transfer and plenty of copy bandwidth going SATA to SATA.
- ahnick 6y ago
  Your current approach sounds fine from the perspective of bitrot. The hole in your approach right now is protecting from incorrect updates made either by yourself or by an application to your data that you don't notice immediately. For example, say you are using a financial application like GnuCash on your desktop and it has bug in the software that gets triggered causing a bad write to the file. (power goes out, OOM, whatever..) syncthing will happily propagate that changed file that contains the bad write to your server and when you copy to the 3rd disk you will also propagate that changed file. You deleted what was on the third disk, so now you no longer have a "good copy" of the file. If you add some sort of snapshotting into your backup routine, then you would be protected from this, because even though you would still propagate the change to the most recent backups, you could still go back to an older snapshot (maybe a week or a month ago or whatever) and pull back a working version of the file.
  - sgtnoodle 6y ago
    That certainly makes sense. In my case, I don't run any applications like that directly on my storage filesystem by default. Those sorts of files and directories, including my linux home directory, are on a non-redundant SSD. If there's a failure, I accept the risk of losing that data. 90% of the time the data either doesn't matter, or is redundantly stored somewhere else, such as a git repo. A typical use case is digital photos; I may dump them into my desktop directory at first so that I may view and sort them, and they'll live there for an arbitrary length of time. I simply won't delete them from their SD card until they're copied into the storage filesystem. Also, I have enough disks laying around that I usually have a 2nd cold backup to wipe. So, I have some inadvertent temporal redundancy going back a year or two. When I make a conscious decision to copy files over to my storage, I have piece of mind that I'll get physical redundancy within hours, and will get picked up by my cold backup eventually.

by ahnick 6y ago

sgtnoodle 6y ago

I wipe the 3rd disk and then dumbly copy all my data to it again. My reasoning is that I want the complete volume of data actually copied as part of the backup activity. The source filesystem, being btrfs, should raise an error if there was bit rot, but that requires actually reading the file contents. If there is, I can grab the copy from the syncthing peer. The destination hard drive has to actually write all the file contents, so the magnetic fields should be all fresh. The 3rd hard disk is a cold backup in case syncthing has a common mode failure that takes out my redundant drives. I want its filesystem to be as boring as possible. For me that currently means ext4. I don't want to have to manage and worry about another tool that optimizes away data transfers, when there's value in the data transfer and plenty of copy bandwidth going SATA to SATA.

ahnick 6y ago
Your current approach sounds fine from the perspective of bitrot. The hole in your approach right now is protecting from incorrect updates made either by yourself or by an application to your data that you don't notice immediately. For example, say you are using a financial application like GnuCash on your desktop and it has bug in the software that gets triggered causing a bad write to the file. (power goes out, OOM, whatever..) syncthing will happily propagate that changed file that contains the bad write to your server and when you copy to the 3rd disk you will also propagate that changed file. You deleted what was on the third disk, so now you no longer have a "good copy" of the file. If you add some sort of snapshotting into your backup routine, then you would be protected from this, because even though you would still propagate the change to the most recent backups, you could still go back to an older snapshot (maybe a week or a month ago or whatever) and pull back a working version of the file.
- sgtnoodle 6y ago
  That certainly makes sense. In my case, I don't run any applications like that directly on my storage filesystem by default. Those sorts of files and directories, including my linux home directory, are on a non-redundant SSD. If there's a failure, I accept the risk of losing that data. 90% of the time the data either doesn't matter, or is redundantly stored somewhere else, such as a git repo. A typical use case is digital photos; I may dump them into my desktop directory at first so that I may view and sort them, and they'll live there for an arbitrary length of time. I simply won't delete them from their SD card until they're copied into the storage filesystem. Also, I have enough disks laying around that I usually have a 2nd cold backup to wipe. So, I have some inadvertent temporal redundancy going back a year or two. When I make a conscious decision to copy files over to my storage, I have piece of mind that I'll get physical redundancy within hours, and will get picked up by my cold backup eventually.