3 ms·

#4: Taking backups for granted. Everywhere I've ever worked I've encountered some variation of this. One recent customer had some MySQL problems which took do

#4: Taking backups for granted. Everywhere I've ever worked I've encountered some variation of this. One recent customer had some MySQL problems which took down the sole MediaWiki server holding all their IT documentation. Not only did they not have a standby, not a single backup or dump was ever done. These days what I find even more astonishing is how even experienced professionals make the mistake of thinking they don't need separate backups because their system has some integral disk redundancy (e.g. http://www.joyent.com/joyeurblog/2008/01/22/bingodisk-and-strongspace-what-happened/ http://www.joyent.com/joyeurblog/2008/01/22/bingodisk-and-st... ).

gaius 17y ago
Another true story: A while ago I was DBA on a very important database (as in, the company would likely go out of business if it was lost). The way it was backed up was to quiesce the standby, take a snapshot of its filesystems on the SAN, then send those to tape while the standby was reenabled. Sounds good. But what had happened was that the sysadmin, to speed it up, was running several parallel backups, each one of which had a hard-coded list of files. So as soon as one new datafile was added to the database (I estimate, a week after that regime was put in place) all subsequent backups were invalid. We were in that situation for over a year and everyone believed that the databases was being properly backed up. We would even occasionally restore a file (picked from the backupset, hmm) and block-verify it. I only discovered it when I was upgrading the system and I fixed it real quick (dynamically generating n equally-sized lists of files is trivial!). Hard coding the list of files was stupid, sure, and betrayed a fundamental lack of understanding about how the system worked - but it was also stupid that no-one ever checked, and that betrayed a lack of understanding of how organizations work. The lesson I took away from that was, you have to have people who have visibility of things end-to-end.
- sbhat7 17y ago
  This is why we blacklist databases/tables that are not to be backed up and backup everything else. This makes sure any new database is added to the backup.
billswift 17y ago
About 12 years ago, when I was still fairly new to linux, I screwed up enough to require a reformat of the HD. I discovered afterward that my backups couldn't be read, the last 3 MONTHS worth of backups. I still don't know what was wrong, I was just making tarballs of my home directory. Always check occasionally to make sure your backups are actually readable.

gaius 17y ago

Another true story: A while ago I was DBA on a very important database (as in, the company would likely go out of business if it was lost). The way it was backed up was to quiesce the standby, take a snapshot of its filesystems on the SAN, then send those to tape while the standby was reenabled. Sounds good. But what had happened was that the sysadmin, to speed it up, was running several parallel backups, each one of which had a hard-coded list of files. So as soon as one new datafile was added to the database (I estimate, a week after that regime was put in place) all subsequent backups were invalid. We were in that situation for over a year and everyone believed that the databases was being properly backed up. We would even occasionally restore a file (picked from the backupset, hmm) and block-verify it. I only discovered it when I was upgrading the system and I fixed it real quick (dynamically generating n equally-sized lists of files is trivial!). Hard coding the list of files was stupid, sure, and betrayed a fundamental lack of understanding about how the system worked - but it was also stupid that no-one ever checked, and that betrayed a lack of understanding of how organizations work. The lesson I took away from that was, you have to have people who have visibility of things end-to-end.

sbhat7 17y ago
This is why we blacklist databases/tables that are not to be backed up and backup everything else. This makes sure any new database is added to the backup.

billswift 17y ago

About 12 years ago, when I was still fairly new to linux, I screwed up enough to require a reformat of the HD. I discovered afterward that my backups couldn't be read, the last 3 MONTHS worth of backups. I still don't know what was wrong, I was just making tarballs of my home directory. Always check occasionally to make sure your backups are actually readable.