4 ms·
> This then lead to the discovery that we didn't have any backups as a result of the system not working for a long time, as well as the system meant to notify u
by hnarn 3y ago
> This then lead to the discovery that we didn't have any backups as a result of the system not working for a long time, as well as the system meant to notify us of any backup errors not working either.
Reminder that validating backups for critical systems is everyones[1] job in organizations where there’s not a literal backup team working on it full time.
This is probably one of the worst experiences a developer or sysadmin can have, but in no situation can it just be one persons fault.
If multiple lines of defense have failed (backup validation, monitoring etc.) and nobody noticed, it’s simply a question of when.
[1]: all technical personnel that can be remotely considered stakeholders in the backups not failing
- Aeolun 3y agoI really enjoy RDS for not having to worry about this. Point-in-time rollbacks for the past 24 hours, and if we accidentally drop the whole database we can restore it from the periodical snapshots.
- hnarn 3y agoZFS snapshots are really nice too, if ZFS is an option.
- andrewaylett 3y agoYou should probably still try it occasionally :). For one of my team's services, we don't just back up the data: we then also restore it into a parallel environment. If the backup fails, the restore fails, and then the environment fails, and while our monitoring might miss some instances of spurious success, it's less likely that the whole chain will look like it's working if the actual backup is not actually working.
- wlll 3y agoI noticed that we weren't validating backups at a company I used to work at so I did a /really/ simple hack where I checked that the backups were within a certain number of bytes in size of the previous dump. Literally a couple of weeks later after putting this live (I think it was a Nagios check) it alerted that the backups had gone down from many 10s of GB to a few k in size. I can't remember what had gone wrong now, it was years ago, but we /did/ catch the issue. Obviously this isn't a comprehensive check, I later wrote a tool that pulled backups down, restored them, and did some sanity checks on the data, but the quick hack worked as an interim.
- hnarn 3y agoWhile you have reason to be proud about setting up the monitoring, your most unique and valuable skill in that moment was that you gave a shit.
- wlll 3y agoHonestly, thinking about how contracts have gone in the past, I think a fair amount of the value I provide is in giving a shit about stuff.