3 ms·

yeah, long time ago I synced the CRM database of the austria press agency to my sony clie https://en.wikipedia.org/wiki/CLI%C3%89 https://en.wikipedia.org/wiki/

by franze 6y ago

yeah, long time ago I synced the CRM database of the austria press agency to my sony clie https://en.wikipedia.org/wiki/CLI%C3%89 https://en.wikipedia.org/wiki/CLI%C3%89 and then my sony clie synced it back. completely deleting all data other then the email and name on the CRM server. gladly there was a backup, but the backup could not get used to get the database back-up. the discussion with IT were fabolous: "yes, we have it all backed up, so the backup is there, but we cant use it to bring the CRM back-up. but its backed up and we even have backups from the backup. they can also not get used to bringt it back up." since then it is my standard procedure to make sure that backups can actually be used to bring projects back up. usually the first test "use backup to bring stuff back up" fails.

KMag 6y ago
If you haven't tested your backups, you don't really have backups. If you aren't allowed to verify the backups yourself, find a safe "mistake" to make to force the responsible parties to run through restoring from backups. Bonus points for using the backup testing exercise as an excuse to convince your manager to let you do a bit of refactoring to remove some particularly ugly module you've been itching to get rid of. Factor out the ugly module, then "accidentally" delete it. My fist job out of school was writing software models of pre-prototype communications equipment for a defense contractor. There was lots of government red tape, and crazy prices being charged everywhere. We couldn't use CVS or SVN (this predated git) to store our code, because they weren't government-approved tools. We could use PVCS to store source code, but only for the build artefacts and corresponding source code of official production versions delivered to the government. (And only people that had taken a multi-hour course to be certified on PVCS could touch it. Every release involved a couple government employees coming out to the defense contractor and physically watching the one guy on the team check in the release to PVCS and burn a pair of DVDs for the government. So, we were forced to do source control by "last one out on Friday has to zip up the entire shared drive and name it with the date". But, at least we were required to pay some crazy price to have the sysadmins backup the shared network drive to tape every week (or was it every day?). Once a month, the admins would take the current tape and send it via courier to a bomb-proof, fire-proof bunker out in the mountains in Pennsylvania or some such. They'd then put the spare tape in the tape drive to start making backups. As soon as the bunker got the latest tape, they'd send back the oldest tape via courier to become the spare tape. So, we always had at least one tape physically present in the bunker. It required 3 tapes to always have at least one tape in the bunker and one tape in the drive. I was less than a year out of school, but based on my interactions with the system admins, I figured there was at most a 50/50 chance they got the backups correct. I asked them once how the backups were going, and got back a very low key, "yea, yea, don't worry about it, we got it" answer. I convinced my boss to let us fake a source code loss: zip up two copies of the network drive, and store the zip files on two different systems. Check the integrity of both zip files, and then "accidentally" delete a directory containing some source we were in the midst of removing anyway. The sysadmins were unable to get our "accidentally" deleted source code back for us. They had neglected to erase the spare tape when putting it into the drive, so they were always only appending to our backups. They just ignored the errors when the tape got full.

by franze 6y ago

KMag 6y ago

If you haven't tested your backups, you don't really have backups. If you aren't allowed to verify the backups yourself, find a safe "mistake" to make to force the responsible parties to run through restoring from backups. Bonus points for using the backup testing exercise as an excuse to convince your manager to let you do a bit of refactoring to remove some particularly ugly module you've been itching to get rid of. Factor out the ugly module, then "accidentally" delete it. My fist job out of school was writing software models of pre-prototype communications equipment for a defense contractor. There was lots of government red tape, and crazy prices being charged everywhere. We couldn't use CVS or SVN (this predated git) to store our code, because they weren't government-approved tools. We could use PVCS to store source code, but only for the build artefacts and corresponding source code of official production versions delivered to the government. (And only people that had taken a multi-hour course to be certified on PVCS could touch it. Every release involved a couple government employees coming out to the defense contractor and physically watching the one guy on the team check in the release to PVCS and burn a pair of DVDs for the government. So, we were forced to do source control by "last one out on Friday has to zip up the entire shared drive and name it with the date". But, at least we were required to pay some crazy price to have the sysadmins backup the shared network drive to tape every week (or was it every day?). Once a month, the admins would take the current tape and send it via courier to a bomb-proof, fire-proof bunker out in the mountains in Pennsylvania or some such. They'd then put the spare tape in the tape drive to start making backups. As soon as the bunker got the latest tape, they'd send back the oldest tape via courier to become the spare tape. So, we always had at least one tape physically present in the bunker. It required 3 tapes to always have at least one tape in the bunker and one tape in the drive. I was less than a year out of school, but based on my interactions with the system admins, I figured there was at most a 50/50 chance they got the backups correct. I asked them once how the backups were going, and got back a very low key, "yea, yea, don't worry about it, we got it" answer. I convinced my boss to let us fake a source code loss: zip up two copies of the network drive, and store the zip files on two different systems. Check the integrity of both zip files, and then "accidentally" delete a directory containing some source we were in the midst of removing anyway. The sysadmins were unable to get our "accidentally" deleted source code back for us. They had neglected to erase the spare tape when putting it into the drive, so they were always only appending to our backups. They just ignored the errors when the tape got full.