3 ms·

Not referring to Gitlab's incident but the general problem with this stuff is too often people treat something like "make a backup" too literally. As in, they

by borplk 10y ago

Not referring to Gitlab's incident but the general problem with this stuff is too often people treat something like "make a backup" too literally. As in, they think the act of generating a backup file is the last stage of the process and they are done with it. Maybe they go the extra mile and throw it in a cronfile too. What you have to do is to consider any and all backups non-existent until you have a complete backup strategy. In other words you have to appreciate that "having a backup" is a means to an end and another way of saying "being able to successfully recover from data loss". So just generating a file is not sufficient. You have to complete a successful test restore before you can call it a backup. You have to have heart-beat measures in place to make sure you will not be impacted by a silent failure (example: check $last_successful_backup >= last X hours). You have to periodically manually check that your automated checks work ("simulate" backups not being generated and wait for alert). Far too often people don't appreciate the depth and the weight that a phrase like "backup plan" carries. So you may ask them to "take care of backups" they will go run pg_dump and mysqldump and say "It's done". No goddamn it, it's not done.

bluGill 10y ago
There are levels. Obviously the best test is 3 backups on 3 different continents (in the future one of the backups will be on Mars), with regular tests that each site can restore from scratch. Even the minimum level: an untested backup to the same disk is better than what most people have. That untested backup is there and protects against accidental file deletion - which will eventually become a test for most people. If the disk fails it is also something to work with. If you tell a disaster recovery service that there is a backup they can use that redundancy to their advantage: odds are physical damage isn't to both the backup and the read data and they only have to recover one. Even if the damage is to both there is something to work with. As we move up the ladder there is more and more. An untested backup has data - give me the team a few years and we can recover it. We might have to recreate the restore from scratch, but there is something. Remember though, the farther down the ladder you are the more expensive recovery can be. If you have tested backups on 3 continents recovery only costs a couple hours downtime - an actuary can put an exact dollar cost on this. If the price is too high you can invest in redundancy on the live system. If you have an untested backup it might be years before your team can recover it: millions of dollars in labor to recover the data and sever years of no/reduced business while they recover it. (hint: the company will go out of business because they cannot afford to recover the data)

by borplk 10y ago

bluGill 10y ago

There are levels. Obviously the best test is 3 backups on 3 different continents (in the future one of the backups will be on Mars), with regular tests that each site can restore from scratch. Even the minimum level: an untested backup to the same disk is better than what most people have. That untested backup is there and protects against accidental file deletion - which will eventually become a test for most people. If the disk fails it is also something to work with. If you tell a disaster recovery service that there is a backup they can use that redundancy to their advantage: odds are physical damage isn't to both the backup and the read data and they only have to recover one. Even if the damage is to both there is something to work with. As we move up the ladder there is more and more. An untested backup has data - give me the team a few years and we can recover it. We might have to recreate the restore from scratch, but there is something. Remember though, the farther down the ladder you are the more expensive recovery can be. If you have tested backups on 3 continents recovery only costs a couple hours downtime - an actuary can put an exact dollar cost on this. If the price is too high you can invest in redundancy on the live system. If you have an untested backup it might be years before your team can recover it: millions of dollars in labor to recover the data and sever years of no/reduced business while they recover it. (hint: the company will go out of business because they cannot afford to recover the data)