4 ms·
People will screw up, so you have to do simple things to make screwing up hard. The production credentials should never have been in the document. Letting a jun
by matwood 9y ago
People will screw up, so you have to do simple things to make screwing up hard. The production credentials should never have been in the document. Letting a junior have prod level access is not that far out of the normal in a small startup environment, but don't make them part of the setup guide. Sounds like they also have backup issues, which points to overall poor devops knowledge.
Not part of this story, but another pet peeve of mine is when I see scripts checking strings like "if env = 'test' else <runs against prod>". This sets up another WTF situation if someone typos 'test' now the script hits prod.
- samstave 9y agoHeh, or take a Netflix Chaos Monkey approach and have a new employee attempt to take down the whole system on their first day and fire any engineers who built whatever the new employee is actually able to break!
- karlkatzke 9y agoWhy fire them? It's valuable experience that you are paying a lot for them to gain. Better: hold a postmortem, figure out what broke, and make the people who screwed it up originally fix it. Keep people who screw things up, as long as they also fix it.
- samstave 9y agoI wasnt serious about "firing" -- but was just maintaining the spirit of what happened to the OP on reddit... but yeah - I agree with you...
- CodeWriter23 9y agoSounds like a technique taught at The Pirate School of Management.
- falsedan 9y ago> so you have to do simple things to make screwing up hard No one goes out of their way to screw up; I'd recommend making it easier to recognize when you've made a mistake, and recover from it. Except for critical business stuff, that needs severe "you cannot fuck this up" safeguards.
- foobarian 9y agoWe've been calling that the "Volkswagen pattern"