4 ms·
I've done similar things, I think most people working long enough have. I've made it a point in teams I've run to reassure the junior staff with tales of how al
by hnbear 3y ago
I've done similar things, I think most people working long enough have. I've made it a point in teams I've run to reassure the junior staff with tales of how all the seniors have made egregious mistakes and survived to tell the tale.
There's a part of one of the Netflix DevOps books about how only the really senior, really important staff screw up the most. It generally takes access, ability, and doing challenging things that the most senior and impactful staff are trying to accomplish to end up in a position where a mistake will cause big damage. That said, yes, teams should also build and deploy resilient systems with well practiced back-ups and restores, and make that common place enough that everyone can restore quickly.
I think in my case the closest to this was a very similar case - fixing a customer record in the customer DB. I'd written the sql quickly, was in a GUI SQL client, and far too fast selected the query from the many I was working on, and hit the shortcut to run it. Then realized a split second later I'd mis-selected just the UPDATE and SET statements, and missed the WHERE. Every customer was name called John Smith. Whoops.
I know he called out that he's in a start-up, but in my case I was at a big financial. The other lesson I learnt is to make very good friends with the infra teams, and have sufficient mutual respect and support that you (as front-office) do as they ask, and go out of your way to help with whatever projects they have (eg. upgrades, patching, migrations), and at the same time, don't bother them will silly questions, and in return be clear that when you do need something and say it's urgent, that it's genuinely urgent and needs immediate attention so that DB restores get handled ASAP. Be a great customer to them, and they'll be great service providers back.