4 ms·

Yeah the fire drill to make sure everything works is crazy important. I just wish that storing things like keys on paper was easier. At CoreOS we put some key

Yeah the fire drill to make sure everything works is crazy important. I just wish that storing things like keys on paper was easier. At CoreOS we put some keys on printed QR codes and scanned them with an airgapped laptop every 90 days to confirm the keys were safe.

denton-scratch 5y ago
> Yeah the fire drill I agree. But I've never been allowed to run a fire drill. Rebuilding a network from bare tin is obviously expensive, but not as expensive as losing the business. And then there's the sheer stress of being responsible for the backups, but not being able to test bare-metal recovery. I'm interested in backup ("what kind of weirdo is this!"), but that wasn't a fun responsibility.
- bigiain 5y ago
  > But I've never been allowed to run a fire drill. Rebuilding a network from bare tin is obviously expensive, but not as expensive as losing the business. Sometime your job is not to be able to take the backups, but to take the blame when a restore can't be done. If you think that might be you, time to polish up that resume... Especially for cloud-based businesses, refusing to let you spend ~8 hours of production platform costs to run a full platform rebuild fire drill is insane, that's about 0.1% of your annual prod AWS budget, and 0.5% of you (or your team's) annual time.
  - denton-scratch 5y ago
    We weren't cloud-based; "bare-metal restore" would have meant acquiring and housing new hardware. And we'd have needed test-scripts and so on - more than 8 hours, more like a few days to do it repeatably (but once that was done, it could have been automated). The boss owned the company; I think it was shortsighted of him. Being able to blame me wouldn't have done him much good, once his firm had gone down the tubes, and I'd moved on. But it was his business to lose, and he knew what his margins were - I didn't. He didn't like me much, mainly because he was a control freak, and I knew his systems much better than he did - I had the control that he craved. Control freaks shouldn't hire people that know more than they do.

denton-scratch 5y ago

> Yeah the fire drill I agree. But I've never been allowed to run a fire drill. Rebuilding a network from bare tin is obviously expensive, but not as expensive as losing the business. And then there's the sheer stress of being responsible for the backups, but not being able to test bare-metal recovery. I'm interested in backup ("what kind of weirdo is this!"), but that wasn't a fun responsibility.

bigiain 5y ago
> But I've never been allowed to run a fire drill. Rebuilding a network from bare tin is obviously expensive, but not as expensive as losing the business. Sometime your job is not to be able to take the backups, but to take the blame when a restore can't be done. If you think that might be you, time to polish up that resume... Especially for cloud-based businesses, refusing to let you spend ~8 hours of production platform costs to run a full platform rebuild fire drill is insane, that's about 0.1% of your annual prod AWS budget, and 0.5% of you (or your team's) annual time.
- denton-scratch 5y ago
  We weren't cloud-based; "bare-metal restore" would have meant acquiring and housing new hardware. And we'd have needed test-scripts and so on - more than 8 hours, more like a few days to do it repeatably (but once that was done, it could have been automated). The boss owned the company; I think it was shortsighted of him. Being able to blame me wouldn't have done him much good, once his firm had gone down the tubes, and I'd moved on. But it was his business to lose, and he knew what his margins were - I didn't. He didn't like me much, mainly because he was a control freak, and I knew his systems much better than he did - I had the control that he craved. Control freaks shouldn't hire people that know more than they do.