5 ms·
Check your backups daily. The whole process is automated and shouldn't cost much.
by scurvy 10y ago
Check your backups daily. The whole process is automated and shouldn't cost much.
- colechristensen 10y agoThen we'll have a check your check your backups day where you make sure your automation isn't making sure the sky is blue.
- scurvy 10y agoIt's in there now. Row counts per table and checksums are written out with the backup for every table. If things don't match up, alarm in a loud and noisy way.
- riffraff 10y agoThe point is that you could have a bug in the verification process that returns "all good" when it really isn't.
- Andys 10y agoI have a SELECT of the most recent updated_at fields for a few tables emailed to me so I can eyeball the values
- noir_lord 10y agoThats a neat idea, I might look at that, thank you.
- scurvy 10y agoMy point is that there are sanity checks built into the process. It's not difficult. There doesn't need to be a human verifying things there. Is the row count increasing over the previous backup? Check. Is the checksum different? Check. Is it non-zero? Check. And on and on. Stop making excuses and automate testing your backups.
- riffraff 10y agoso, when your check process does not report because it failed to run, how do you know? Do you have a monitor for your monitor? Does that monitor have a monitor? There is no reason not to automate your backup tests, but there is no reason not to eyeball the check is actually working from time to time.
- scurvy 10y agoYes, the restore process checks in daily; alarms are thrown if it doesn't check in. MySQL is also pretty good about not starting if the data is corrupted for whatever reason. The process not starting is a pretty obvious alarm, too. Checks are cheap. Better to automate your logic and invest in good monitoring and automation.