3 ms·

I say “you are only as good as your last restore”, a phrase I stole from the Theatre industry where your performance “is only as good as your last rehearsal”. O

I say “you are only as good as your last restore”, a phrase I stole from the Theatre industry where your performance “is only as good as your last rehearsal”. Of course this is most useful in Disaster Recovery scenarios. Rehearsal? Restore? Oh we’ve never tested it.

dspillett 5y ago
For a couple of VMs I have an automated restore to another VM. The test copies rsync from the latest backup daily and restart themselves, and once a week do a full wipe and resync. A script touches the copy mail store (I currently use Zimbra for a mail server) to check it is up, and that the last item “received” is no older than 24 hours, and emails me to say all is well. If I don't get that daily mail, the restore has failed somehow. I keep meaning to add an extra layer that checks for the daily mail and sends me an SMS if it doesn't arrive, just in case I don't notice. The copy VMs don't need to be high spec, though in theory if the main machines die I could just switch other to them in minutes by DNS+firewall updates (and by giving them a bit more CPU+RAM allocation), just enough that the services start and run fast enough for the occasional manual paranoia check, and they aren't accessible to the outside world (but could be, if I needed to make that switch in a DR situation). Similar for the little web servers I still run and key parts of general data backup (the parts that can't be reobtained at all if both originals and backups are lost): restore from backup after the update of the backups is due to finish, checksum everything older a few hours on both sides, and send a mail if the checksums match. That sometimes give false negatives, if a file is updated between backup and checksum, I have a couple of possible fixes for that in mind but it only happens if I'm working silly late due to when the checks run so I've not been bothered enough to implement one.

dspillett 5y ago

For a couple of VMs I have an automated restore to another VM. The test copies rsync from the latest backup daily and restart themselves, and once a week do a full wipe and resync. A script touches the copy mail store (I currently use Zimbra for a mail server) to check it is up, and that the last item “received” is no older than 24 hours, and emails me to say all is well. If I don't get that daily mail, the restore has failed somehow. I keep meaning to add an extra layer that checks for the daily mail and sends me an SMS if it doesn't arrive, just in case I don't notice. The copy VMs don't need to be high spec, though in theory if the main machines die I could just switch other to them in minutes by DNS+firewall updates (and by giving them a bit more CPU+RAM allocation), just enough that the services start and run fast enough for the occasional manual paranoia check, and they aren't accessible to the outside world (but could be, if I needed to make that switch in a DR situation). Similar for the little web servers I still run and key parts of general data backup (the parts that can't be reobtained at all if both originals and backups are lost): restore from backup after the update of the backups is due to finish, checksum everything older a few hours on both sides, and send a mail if the checksums match. That sometimes give false negatives, if a file is updated between backup and checksum, I have a couple of possible fixes for that in mind but it only happens if I'm working silly late due to when the checks run so I've not been bothered enough to implement one.