3 ms·
I lost a PM job once because a new-hire junior admin deleted a symlink and took a major site down... The CEO who fired me was upset that I couldn't restore the
by winternett 5y ago
I lost a PM job once because a new-hire junior admin deleted a symlink and took a major site down... The CEO who fired me was upset that I couldn't restore the massive database and run a diff on it versus a local backup I made out of being overly cautious the day prior.
I was supposed to just be a PM, not even supposed to be a Dev... Served me right for acting like I might have been able to fix the issue. No more volunteering outside of my role... Possibly also why job descriptions include so much out of normal scope "responsibility" now.
The company had a bad habit of underpaying roles, and regularly hired inexperienced admins to do key PROD work for clients.
I was also blacklisted for rehire from the company and took the fall I guess when the issue was explained to the clients.
Oh well.. Spilled milk... The company kept failing after I left and lost all it's major contracts weeks after I got cut anyway. I got a decent severance payout and still won a 25% raise on my next gig... Moved forward without scratches.
Lessons -
Back everything up well, no matter how much time it takes, before touching PROD or even before others touch PROD.
Don't work for places that are regularly dedicated to underpaying talent to do mission critical work.
When a door closes, another opens, learn from the past, but don't carry it forward, and don't fight ignorant and tech-blind arrogance in leadership, it only hinders your career growth if you crusade against bad leadership. Karma does the real work for you eventually if you remain calm & composed.
Quickly identify if tech talent and problem solving stops at your level, and if so, protect yourself from being a scapegoat for leadership failures.
Don't be the one that screws things up, be the one that both anticipates and quickly and fixes all varieties of screw-ups from everyone else.
- thaumasiotes 5y ago> I lost a PM job once because a new-hire junior admin deleted a symlink and took a major site down... The CEO who fired me was upset that I couldn't restore the massive database and run a diff on it versus a local backup I made out of being overly cautious the day prior. But... the database wasn't gone? If you delete a symlink, you can just put it back.
- winternett 5y agoThe (devops) admin deleted the link to an assets folder for all the file uploads for a pretty large web site and didn't even know how to explain what he did. We all didn't know it was a symlink that was deleted until later after the failure, but the disconnection caused the entire site's db to go corrupt... I restored the link but that didn't solve the resulting corruption, so a DB recovery/diff was in process from my desktop backup when they turned to blame on me...The DB was really FUBARED... Actually quite glad I didn't have to finish the fix on my own after all that drama TBH. :/