3 ms·

Been there Done that :) I was once on-call working for one the leading organizations. I got a call in the middle of the night that some critical job had failed

Been there Done that :) I was once on-call working for one the leading organizations. I got a call in the middle of the night that some critical job had failed and due to the significant data load, it was imperative to restart the processing. I login to the system with a privileged account. Restart the job with new parameters and since I wanted not to see the ugly logs, I wanted to redirect the output to /dev/null. I run the following command ./jobname 1>./db-file-name and there is -THE DISASTER- For some reason this kept popping in my head - "Bad things happen to Good people" We recovered the data but there was some data loss still as the mirror backup had not run. Of course, we have come long way since then. Now, there are constant sync between Prod/DR and multitude of offline backups and recovery is possible for last 7 days, the month or any month during the year and the year before.

hobs 11y ago
I was doing a favor to a friend and on a refferal talked to a guy who didnt have anything but weekly backups and had a corrupt database due to some drive failures. I was able to determine that the corrupt data was repairable if we had a copy of the old db, and since it was a tiny machine system I asked "Would you mind restoring the backup side by side with production and I can do what I need?" "Sure thing!" I wait for a minute, and then my connection to the production database dies. I refresh the client, and now the one database available is restoring from a backup... I called him and asked if he meant to overwrite his production copy with his backup instead of do it side by side, and he says petulantly, "I didnt do that!" I ask him to check again, and he responds with "I will call you right back!" Five minutes later I get the call, "How do I roll back my restore partially through the restore process?" Oops.

hobs 11y ago

I was doing a favor to a friend and on a refferal talked to a guy who didnt have anything but weekly backups and had a corrupt database due to some drive failures. I was able to determine that the corrupt data was repairable if we had a copy of the old db, and since it was a tiny machine system I asked "Would you mind restoring the backup side by side with production and I can do what I need?" "Sure thing!" I wait for a minute, and then my connection to the production database dies. I refresh the client, and now the one database available is restoring from a backup... I called him and asked if he meant to overwrite his production copy with his backup instead of do it side by side, and he says petulantly, "I didnt do that!" I ask him to check again, and he responds with "I will call you right back!" Five minutes later I get the call, "How do I roll back my restore partially through the restore process?" Oops.