3 ms·

>The only two unforgivable sins are “no working backups” and being hacked. I always change that sentence to "no working restore" because that's often forgotten

by nix23 6y ago

>The only two unforgivable sins are “no working backups” and being hacked. I always change that sentence to "no working restore" because that's often forgotten, but yeah it's the same as a "working backup" but changes the focus what's the important part with backups.

Ensorceled 6y ago
I have, on two occasions, discovered that my new place of employment had never tested the restore process and had broken daily backups. One of those companies had a whole failover site ready to go in case of emergency and had never tested that the database replication was operating correctly.
- nix23 6y ago
  >I have, on two occasions, discovered that my new place of employment had never tested the restore process and had broken daily backups. Same same, it was with backupexec, once the daily backup-report said everything fine, so i made a restore on a vm (just to check) then a dry-run rsync with the prod Machine (Fileserver) and it wanted to transfer about 1TB of files, the last "real" backup was 2 year ago, turned out the backupexec-agent on the Fileserver was never updated and reported that the last backup was successful (the 2 yo that was).
  - Ensorceled 6y ago
    I checked the backups and they were all something like 10Mb, I asked if that was a reasonable size and everybody freaked out. It's near the top of my CTO/VP first week checklist ... "what is our recovery strategy" along with "who are my team"
    - nix23 6y ago
      + Where is the coffeemaker and do we have any Oracle or Microsoft Products, and do we have all the licenses and CAL's :)
iso1210 6y ago
About 15 years ago was very happy with his source code backups, the code was on an IRIX server and a cronjob tarred up the backups and he kept multiple copies of the tars offsite, and even offline. The tarballs even extracted fine and at first glance the directory structure and file lengths were ok. Trouble is a bug in the file system meant that somehow the files had been overwritten with nulls. The program continued to run, as it sat in memory. Until one day the box was rebooted (power test or something). It didn't come back. The files were great, containing multiple files all of the right looking length. Everything was null. "main.c" or whatever it was was 8 kilobytes of null characters. This had been happening for some months so even the oldest source backups were wiped, and the actual binaries were out too. The system was called ELVIS, which made for a great status report. As it was on an ancient irix machine and he was rewriting in PHP at the time he just accelerated that and we were back up in a few days (the program was a front end to a database and file stores on other machines, so it was just the web part)
ocdtrekkie 6y ago
Good ole' Schrödinger's Backups. Backups both exist and don't exist until you try to restore them. :) I have a few frequent flyers at work for "I accidentally deleted this file, corrupted this database, etc." that I end up appreciating for making me somewhat regularly pull files from backups.
- nix23 6y ago
  >Good ole' Schrödinger's Backups. Backups both exist and don't exist until you try to restore them. That sentence i steal from you next time i have to tell someone what a backup/restore is. Thanks :)

by nix23 6y ago

Ensorceled 6y ago

I have, on two occasions, discovered that my new place of employment had never tested the restore process and had broken daily backups. One of those companies had a whole failover site ready to go in case of emergency and had never tested that the database replication was operating correctly.

nix23 6y ago
>I have, on two occasions, discovered that my new place of employment had never tested the restore process and had broken daily backups. Same same, it was with backupexec, once the daily backup-report said everything fine, so i made a restore on a vm (just to check) then a dry-run rsync with the prod Machine (Fileserver) and it wanted to transfer about 1TB of files, the last "real" backup was 2 year ago, turned out the backupexec-agent on the Fileserver was never updated and reported that the last backup was successful (the 2 yo that was).
- Ensorceled 6y ago
  I checked the backups and they were all something like 10Mb, I asked if that was a reasonable size and everybody freaked out. It's near the top of my CTO/VP first week checklist ... "what is our recovery strategy" along with "who are my team"
  - nix23 6y ago
    + Where is the coffeemaker and do we have any Oracle or Microsoft Products, and do we have all the licenses and CAL's :)

iso1210 6y ago

About 15 years ago was very happy with his source code backups, the code was on an IRIX server and a cronjob tarred up the backups and he kept multiple copies of the tars offsite, and even offline. The tarballs even extracted fine and at first glance the directory structure and file lengths were ok. Trouble is a bug in the file system meant that somehow the files had been overwritten with nulls. The program continued to run, as it sat in memory. Until one day the box was rebooted (power test or something). It didn't come back. The files were great, containing multiple files all of the right looking length. Everything was null. "main.c" or whatever it was was 8 kilobytes of null characters. This had been happening for some months so even the oldest source backups were wiped, and the actual binaries were out too. The system was called ELVIS, which made for a great status report. As it was on an ancient irix machine and he was rewriting in PHP at the time he just accelerated that and we were back up in a few days (the program was a front end to a database and file stores on other machines, so it was just the web part)

ocdtrekkie 6y ago

Good ole' Schrödinger's Backups. Backups both exist and don't exist until you try to restore them. :) I have a few frequent flyers at work for "I accidentally deleted this file, corrupted this database, etc." that I end up appreciating for making me somewhat regularly pull files from backups.

nix23 6y ago
>Good ole' Schrödinger's Backups. Backups both exist and don't exist until you try to restore them. That sentence i steal from you next time i have to tell someone what a backup/restore is. Thanks :)