an openbsd vm in a... secret location... runs rsnapshot (rsync over ssh) to all of my servers every few hours and keeps separate hourly, daily, weekly, monthly,

an openbsd vm in a... secret location... runs rsnapshot (rsync over ssh) to all of my servers every few hours and keeps separate hourly, daily, weekly, monthly, and yearly "full" backups. they are stored on a usb hard drive encrypted with openbsd's softraid. when the server boots up, i have to enter in the softraid passphrase, it mounts the drive, then i launch screen and start up the script that spawns the rsnapshot processes at the appropriate times. when starting it, i enter in the common ssh key passphrase (which is restricted on all the servers to only allow logging in from that network) and common mysql backup user password. these are stored in memory and then the script runs rsnapshot every few hours. once a day, the script does a full mysql dump of all databases on all servers using a ruby script with net::ssh to tunnel mysql access over ssh. it does this to keep "proper" mysql backups in case the rsync'd binary mysql files are transferred in an odd state, and keep daily backups of everything. it also lets me quickly get at data with standard text tools instead of having to restore the raw mysql files somewhere temporarily, start up mysqld, and extract it. so once the server is booted, it's completely automated. i've been doing backups this way for a couple years and have recovered files and databases from them a number of times. no complaints.