4 ms·

You can make pretty much any sort of tech fuckup and be forgiven. The only two unforgivable sins are “no working backups” and being hacked. Even being hacked i

You can make pretty much any sort of tech fuckup and be forgiven. The only two unforgivable sins are “no working backups” and being hacked. Even being hacked is kinda forgivable cause it’s so hard to avoid. No working backups is totally unforgivable. Any tech person who walks into any tech job at any level up from junior to CTO has a duty as one of their first tasks to find out if working backups are being done of their work.

franze 6y ago
yeah, long time ago I synced the CRM database of the austria press agency to my sony clie https://en.wikipedia.org/wiki/CLI%C3%89 https://en.wikipedia.org/wiki/CLI%C3%89 and then my sony clie synced it back. completely deleting all data other then the email and name on the CRM server. gladly there was a backup, but the backup could not get used to get the database back-up. the discussion with IT were fabolous: "yes, we have it all backed up, so the backup is there, but we cant use it to bring the CRM back-up. but its backed up and we even have backups from the backup. they can also not get used to bringt it back up." since then it is my standard procedure to make sure that backups can actually be used to bring projects back up. usually the first test "use backup to bring stuff back up" fails.
- KMag 6y ago
  If you haven't tested your backups, you don't really have backups. If you aren't allowed to verify the backups yourself, find a safe "mistake" to make to force the responsible parties to run through restoring from backups. Bonus points for using the backup testing exercise as an excuse to convince your manager to let you do a bit of refactoring to remove some particularly ugly module you've been itching to get rid of. Factor out the ugly module, then "accidentally" delete it. My fist job out of school was writing software models of pre-prototype communications equipment for a defense contractor. There was lots of government red tape, and crazy prices being charged everywhere. We couldn't use CVS or SVN (this predated git) to store our code, because they weren't government-approved tools. We could use PVCS to store source code, but only for the build artefacts and corresponding source code of official production versions delivered to the government. (And only people that had taken a multi-hour course to be certified on PVCS could touch it. Every release involved a couple government employees coming out to the defense contractor and physically watching the one guy on the team check in the release to PVCS and burn a pair of DVDs for the government. So, we were forced to do source control by "last one out on Friday has to zip up the entire shared drive and name it with the date". But, at least we were required to pay some crazy price to have the sysadmins backup the shared network drive to tape every week (or was it every day?). Once a month, the admins would take the current tape and send it via courier to a bomb-proof, fire-proof bunker out in the mountains in Pennsylvania or some such. They'd then put the spare tape in the tape drive to start making backups. As soon as the bunker got the latest tape, they'd send back the oldest tape via courier to become the spare tape. So, we always had at least one tape physically present in the bunker. It required 3 tapes to always have at least one tape in the bunker and one tape in the drive. I was less than a year out of school, but based on my interactions with the system admins, I figured there was at most a 50/50 chance they got the backups correct. I asked them once how the backups were going, and got back a very low key, "yea, yea, don't worry about it, we got it" answer. I convinced my boss to let us fake a source code loss: zip up two copies of the network drive, and store the zip files on two different systems. Check the integrity of both zip files, and then "accidentally" delete a directory containing some source we were in the midst of removing anyway. The sysadmins were unable to get our "accidentally" deleted source code back for us. They had neglected to erase the spare tape when putting it into the drive, so they were always only appending to our backups. They just ignored the errors when the tape got full.
foolmeonce 6y ago
> No working backups is totally unforgivable. It sounds to me like a "purge for legal reasons" function was applied to more records. Beyond some possible time limit to undo, it should not be possible to retrieve such a purge from backups. If they can retrieve this, then that is a larger problem.
- J-dawg 6y ago
  The PNC has been challenged in the past for holding on to records that legally should have been deleted. If I put my tin foil hat on for a moment, I wonder if this "mistake" was a way of getting rid of some records that should have already been deleted. Doing it this way, and publicly admitting it, might have been deemed safer than clandestinely deleting the records and then getting caught in a security audit later.
nix23 6y ago
>The only two unforgivable sins are “no working backups” and being hacked. I always change that sentence to "no working restore" because that's often forgotten, but yeah it's the same as a "working backup" but changes the focus what's the important part with backups.
- Ensorceled 6y ago
  I have, on two occasions, discovered that my new place of employment had never tested the restore process and had broken daily backups. One of those companies had a whole failover site ready to go in case of emergency and had never tested that the database replication was operating correctly.
  - nix23 6y ago
    >I have, on two occasions, discovered that my new place of employment had never tested the restore process and had broken daily backups. Same same, it was with backupexec, once the daily backup-report said everything fine, so i made a restore on a vm (just to check) then a dry-run rsync with the prod Machine (Fileserver) and it wanted to transfer about 1TB of files, the last "real" backup was 2 year ago, turned out the backupexec-agent on the Fileserver was never updated and reported that the last backup was successful (the 2 yo that was).
    - Ensorceled 6y ago
      I checked the backups and they were all something like 10Mb, I asked if that was a reasonable size and everybody freaked out. It's near the top of my CTO/VP first week checklist ... "what is our recovery strategy" along with "who are my team"
      - nix23 6y ago
        + Where is the coffeemaker and do we have any Oracle or Microsoft Products, and do we have all the licenses and CAL's :)
- 6y ago
kace91 6y ago
>The only two unforgivable sins are “no working backups” and being hacked. Storing user passwords instead of hashes for the second, maybe. Bonus points if they're in plaintext.

franze 6y ago

yeah, long time ago I synced the CRM database of the austria press agency to my sony clie https://en.wikipedia.org/wiki/CLI%C3%89 https://en.wikipedia.org/wiki/CLI%C3%89 and then my sony clie synced it back. completely deleting all data other then the email and name on the CRM server. gladly there was a backup, but the backup could not get used to get the database back-up. the discussion with IT were fabolous: "yes, we have it all backed up, so the backup is there, but we cant use it to bring the CRM back-up. but its backed up and we even have backups from the backup. they can also not get used to bringt it back up." since then it is my standard procedure to make sure that backups can actually be used to bring projects back up. usually the first test "use backup to bring stuff back up" fails.

KMag 6y ago
If you haven't tested your backups, you don't really have backups. If you aren't allowed to verify the backups yourself, find a safe "mistake" to make to force the responsible parties to run through restoring from backups. Bonus points for using the backup testing exercise as an excuse to convince your manager to let you do a bit of refactoring to remove some particularly ugly module you've been itching to get rid of. Factor out the ugly module, then "accidentally" delete it. My fist job out of school was writing software models of pre-prototype communications equipment for a defense contractor. There was lots of government red tape, and crazy prices being charged everywhere. We couldn't use CVS or SVN (this predated git) to store our code, because they weren't government-approved tools. We could use PVCS to store source code, but only for the build artefacts and corresponding source code of official production versions delivered to the government. (And only people that had taken a multi-hour course to be certified on PVCS could touch it. Every release involved a couple government employees coming out to the defense contractor and physically watching the one guy on the team check in the release to PVCS and burn a pair of DVDs for the government. So, we were forced to do source control by "last one out on Friday has to zip up the entire shared drive and name it with the date". But, at least we were required to pay some crazy price to have the sysadmins backup the shared network drive to tape every week (or was it every day?). Once a month, the admins would take the current tape and send it via courier to a bomb-proof, fire-proof bunker out in the mountains in Pennsylvania or some such. They'd then put the spare tape in the tape drive to start making backups. As soon as the bunker got the latest tape, they'd send back the oldest tape via courier to become the spare tape. So, we always had at least one tape physically present in the bunker. It required 3 tapes to always have at least one tape in the bunker and one tape in the drive. I was less than a year out of school, but based on my interactions with the system admins, I figured there was at most a 50/50 chance they got the backups correct. I asked them once how the backups were going, and got back a very low key, "yea, yea, don't worry about it, we got it" answer. I convinced my boss to let us fake a source code loss: zip up two copies of the network drive, and store the zip files on two different systems. Check the integrity of both zip files, and then "accidentally" delete a directory containing some source we were in the midst of removing anyway. The sysadmins were unable to get our "accidentally" deleted source code back for us. They had neglected to erase the spare tape when putting it into the drive, so they were always only appending to our backups. They just ignored the errors when the tape got full.

foolmeonce 6y ago

> No working backups is totally unforgivable. It sounds to me like a "purge for legal reasons" function was applied to more records. Beyond some possible time limit to undo, it should not be possible to retrieve such a purge from backups. If they can retrieve this, then that is a larger problem.

J-dawg 6y ago
The PNC has been challenged in the past for holding on to records that legally should have been deleted. If I put my tin foil hat on for a moment, I wonder if this "mistake" was a way of getting rid of some records that should have already been deleted. Doing it this way, and publicly admitting it, might have been deemed safer than clandestinely deleting the records and then getting caught in a security audit later.

nix23 6y ago

>The only two unforgivable sins are “no working backups” and being hacked. I always change that sentence to "no working restore" because that's often forgotten, but yeah it's the same as a "working backup" but changes the focus what's the important part with backups.

Ensorceled 6y ago
I have, on two occasions, discovered that my new place of employment had never tested the restore process and had broken daily backups. One of those companies had a whole failover site ready to go in case of emergency and had never tested that the database replication was operating correctly.
- nix23 6y ago
  >I have, on two occasions, discovered that my new place of employment had never tested the restore process and had broken daily backups. Same same, it was with backupexec, once the daily backup-report said everything fine, so i made a restore on a vm (just to check) then a dry-run rsync with the prod Machine (Fileserver) and it wanted to transfer about 1TB of files, the last "real" backup was 2 year ago, turned out the backupexec-agent on the Fileserver was never updated and reported that the last backup was successful (the 2 yo that was).
  - Ensorceled 6y ago
    I checked the backups and they were all something like 10Mb, I asked if that was a reasonable size and everybody freaked out. It's near the top of my CTO/VP first week checklist ... "what is our recovery strategy" along with "who are my team"
    - nix23 6y ago
      + Where is the coffeemaker and do we have any Oracle or Microsoft Products, and do we have all the licenses and CAL's :)
6y ago

kace91 6y ago

>The only two unforgivable sins are “no working backups” and being hacked. Storing user passwords instead of hashes for the second, maybe. Bonus points if they're in plaintext.