3 ms·

Make sure your borg repos are copying properly and in full! I had a horrible realisation that my borg backups were timing out on the offiste copy, meaning the

by corobo 5y ago

Make sure your borg repos are copying properly and in full! I had a horrible realisation that my borg backups were timing out on the offiste copy, meaning the resulting offsite backup I had was non-existent. The heartbreaking error message Inconsistency detected. Please run "borg check [repository]" - although likely this is "beyond repair" Course following the 3-2-1 rule you're probably good, but aye I'm treating borg repos as delicately as I would a striped raid now. I am also monitoring that `borg check` actually comes back successfully before considering the backup complete too :) Also if you're in a small company, start up, etc.. Set yourself a 3-2-1+ backup system. + is the miracle backup that makes you look good.

klysm 5y ago
This is why you gotta test your backups too
- pkulak 5y ago
  Can Borg do this? Like, something you can set up to run once a month or so?
  - throwaway81523 5y ago
    Borg has a command ("borg check") to test backup metadata. I'm not sure exactly what it does, but it is pretty slow, about 80 minutes to check a 1.6TB backup depending on how busy the backup server is. Another approach might be to mount the backup as a FUSE filesystem and monitor it with something like tripwire. I just thought of that a minute ago and haven't looked into it, so idk if it is workable. At the end of the day you have to occasionally do a full restore to another system, and test everything.
- corobo 5y ago
  Indeed. You only get burned once .. and then apparently once again a decade later when you let your guard down
dspillett 5y ago
> I had a horrible realisation that my borg backups were timing out ... meaning the ... backup I had was ... A backup that isn't tested is not a true backup, it is a disappointment waiting to be found! This can happy with any backup tool, my hand-crafted¹ rsync based scripts included. Testing isn't hard to setup if you don't mind the final step being manual. Snapshots have a checksum file, and daily one is picked and rechecked, any difference is an indication of bit-rot on the storage medium or something accidentally getting deleted/modified otherwise. After the newest copy is created by the main backup script a list of files not touched since the backup started is made and pushed up, the backup site checks those files and sends the result back so it can be compared. Any difference in these checksums results in an email to an account that makes my phone shout in a distressed manner. The manual part here is occasionally checking the results manually because not getting an email could either mean all is well or that something has broken to the point that the checks aren't running at all². For specific systems like my mail server I have a replica VM, not visible to the Internet at large, that wipes itself and restores from the latest off-site backup. I look at that occasionally to see that it is running and has the messages I've recently sent and received. As a bonus this VM could quickly be made to be publicly available and take over with a few firewall DNS changes, should the main mail server or its host physically die, and even if it doesn't take over its existence proves that the restore method is reliable should I need to restore the one in the primary location. Some extra automated checks could be added to this too, but again there comes a point where writing the checks takes more time than just doing that manually³ and I'd still do it manually out of paranoia anyway. [1] If I'm honest, “string together” would be much more accurate than “crafted” [2] I could automate that a bit too, but then that automation still needs to be verified occasionally, it quickly gets to the point where it is double-checks all the way down and making sure you check manually occasionally is far far more maintainable a system. [3] If these were a business thing rather than personal services, then the automated procedure vs manual checks desirability balance might change somewhat.
- corobo 5y ago
  > A backup that isn't tested is not a true backup, it is a disappointment waiting to be found Yup. 500 gigs of disk seatfiller (a full backup in this instance would have been somewhere between 1-2T) Live and learn.. and learn again later when you let yourself slip :) My backup process is a LOT noisier if any of the commands fail/timeout/dirs-arent-same-size-after now!

by corobo 5y ago

klysm 5y ago

This is why you gotta test your backups too

pkulak 5y ago
Can Borg do this? Like, something you can set up to run once a month or so?
- throwaway81523 5y ago
  Borg has a command ("borg check") to test backup metadata. I'm not sure exactly what it does, but it is pretty slow, about 80 minutes to check a 1.6TB backup depending on how busy the backup server is. Another approach might be to mount the backup as a FUSE filesystem and monitor it with something like tripwire. I just thought of that a minute ago and haven't looked into it, so idk if it is workable. At the end of the day you have to occasionally do a full restore to another system, and test everything.
corobo 5y ago
Indeed. You only get burned once .. and then apparently once again a decade later when you let your guard down

dspillett 5y ago

> I had a horrible realisation that my borg backups were timing out ... meaning the ... backup I had was ... A backup that isn't tested is not a true backup, it is a disappointment waiting to be found! This can happy with any backup tool, my hand-crafted¹ rsync based scripts included. Testing isn't hard to setup if you don't mind the final step being manual. Snapshots have a checksum file, and daily one is picked and rechecked, any difference is an indication of bit-rot on the storage medium or something accidentally getting deleted/modified otherwise. After the newest copy is created by the main backup script a list of files not touched since the backup started is made and pushed up, the backup site checks those files and sends the result back so it can be compared. Any difference in these checksums results in an email to an account that makes my phone shout in a distressed manner. The manual part here is occasionally checking the results manually because not getting an email could either mean all is well or that something has broken to the point that the checks aren't running at all². For specific systems like my mail server I have a replica VM, not visible to the Internet at large, that wipes itself and restores from the latest off-site backup. I look at that occasionally to see that it is running and has the messages I've recently sent and received. As a bonus this VM could quickly be made to be publicly available and take over with a few firewall DNS changes, should the main mail server or its host physically die, and even if it doesn't take over its existence proves that the restore method is reliable should I need to restore the one in the primary location. Some extra automated checks could be added to this too, but again there comes a point where writing the checks takes more time than just doing that manually³ and I'd still do it manually out of paranoia anyway. [1] If I'm honest, “string together” would be much more accurate than “crafted” [2] I could automate that a bit too, but then that automation still needs to be verified occasionally, it quickly gets to the point where it is double-checks all the way down and making sure you check manually occasionally is far far more maintainable a system. [3] If these were a business thing rather than personal services, then the automated procedure vs manual checks desirability balance might change somewhat.

corobo 5y ago
> A backup that isn't tested is not a true backup, it is a disappointment waiting to be found Yup. 500 gigs of disk seatfiller (a full backup in this instance would have been somewhere between 1-2T) Live and learn.. and learn again later when you let yourself slip :) My backup process is a LOT noisier if any of the commands fail/timeout/dirs-arent-same-size-after now!