4 ms·

I can only imagine the way the person who pulled the trigger on the deletion script felt the moment they realized what had happened. I’ve been there with much

I can only imagine the way the person who pulled the trigger on the deletion script felt the moment they realized what had happened. I’ve been there with much less significant incidents when a “routine” change turned into a potentially resume generating event. It’s not fun. Ultimately the responsibility is with the organization that made it possible for an event of that scale to happen rather than the individual person who happened to trigger it but that doesn’t make it feel any better.

llamaLord 4y ago
As someone who observed this particular incident from the inside (holy shit-balls have the last 3 weeks not been fun), one of the few positive elements of it has been the universal and effectively instinctual agreement internally that it was a massive screw-up in the system that we all have to own, rather than one or several individual screw-ups that need to be put at the feet of individuals.
- trebligdivad 4y ago
  Well, the one constant of IT is 'shit happens'; mark this one down as something interesting you've seen.
SnowHill9902 4y ago
That moment when you see DELETE 18388272773 0
- tetha 4y ago
  Reasons why I don't enter writing queries without starting a transaction first. ABORT, good sir, ABORT.
Kwpolska 4y ago
Thomas J. Watson famously said: Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?
- freddie_mercury 4y ago
  This quip never made much sense. Mistakes almost certainly follow some kind of pareto distribution instead of it being evenly distributed. That's why 2% of doctors are responsible for 39% of malpractice lawsuits. If your doctor cut off the wrong leg and you sued (and won) $2 million, would go back to the same doctor and just chalk it up as "well, now he's got $2 million of training"?
  - hatware 4y ago
    > This quip never made much sense. I see it as: Your employee just learned a valuable lesson, and you definitely don't want to hire another new employee to make that same mistake again. Ultimately, it's the owner taking responsibility for the fuck up because they hired the employee, but still standing behind their decision to hire the employee. If your employee is an idiot and makes up 39% of mistakes, of course you won't keep them around when they make a big one. But most employees are not fuck-ups, as you alluded to with 2% of doctors making a majority of malpractice suits.
- ramraj07 4y ago
  Lol do you should keep an idiot who shot a (looks at notes for what’s as cheap as 600k in the army) a javelin missile while drunk? It’s prudent to apply the concept of nuance. If some intern deleted all these accounts you should fire the guy responsible for letting things free enough for the intern to do it but you should fire them. Whatever lesson learned could be learned from a wiki page detailing the incident.
originalvichy 4y ago
I think in this instance it’s the person who sent the IDs that feels worse. The deleters were provided IDs that had 30 correct app IDs and the rest were site IDs. Like they mentioned they had a delete script that worked for all types of unique IDs so that also can dilute the feeling of ”it’s all my fault” hopefully.
doctor_eval 4y ago
I was on call “back in the day” when a customer rang to tell me that she’d meant to type: # find /tmp -exec rm {} /; But instead she’d typed # find / tmp -exec rm {} /; I learned a lot about Unix (and, as it happens, X windows) file systems that week. Note that this was long before tools like sudo existed, where cleaning up tmp was often a manual process. There is no way we could blame a person for a single keystroke error - it’s crazy that the command worked - but I now look at the find command with an enormous amount of fear.
- BonoboIO 4y ago
  Good old days deleting an entire server ... learned that the hard way.
- Bender 4y ago
  I ran into a similar one a few times from the same team that kept doing things in their scripts like rm -rf ${basedir}/* without setting set -u to exit if there are undefined variables. I suggested a few other sanity checks but was told I was adding friction.

llamaLord 4y ago

As someone who observed this particular incident from the inside (holy shit-balls have the last 3 weeks not been fun), one of the few positive elements of it has been the universal and effectively instinctual agreement internally that it was a massive screw-up in the system that we all have to own, rather than one or several individual screw-ups that need to be put at the feet of individuals.

trebligdivad 4y ago
Well, the one constant of IT is 'shit happens'; mark this one down as something interesting you've seen.

SnowHill9902 4y ago

That moment when you see DELETE 18388272773 0

tetha 4y ago
Reasons why I don't enter writing queries without starting a transaction first. ABORT, good sir, ABORT.

Kwpolska 4y ago

Thomas J. Watson famously said: Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?

freddie_mercury 4y ago
This quip never made much sense. Mistakes almost certainly follow some kind of pareto distribution instead of it being evenly distributed. That's why 2% of doctors are responsible for 39% of malpractice lawsuits. If your doctor cut off the wrong leg and you sued (and won) $2 million, would go back to the same doctor and just chalk it up as "well, now he's got $2 million of training"?
- hatware 4y ago
  > This quip never made much sense. I see it as: Your employee just learned a valuable lesson, and you definitely don't want to hire another new employee to make that same mistake again. Ultimately, it's the owner taking responsibility for the fuck up because they hired the employee, but still standing behind their decision to hire the employee. If your employee is an idiot and makes up 39% of mistakes, of course you won't keep them around when they make a big one. But most employees are not fuck-ups, as you alluded to with 2% of doctors making a majority of malpractice suits.
ramraj07 4y ago
Lol do you should keep an idiot who shot a (looks at notes for what’s as cheap as 600k in the army) a javelin missile while drunk? It’s prudent to apply the concept of nuance. If some intern deleted all these accounts you should fire the guy responsible for letting things free enough for the intern to do it but you should fire them. Whatever lesson learned could be learned from a wiki page detailing the incident.

originalvichy 4y ago

I think in this instance it’s the person who sent the IDs that feels worse. The deleters were provided IDs that had 30 correct app IDs and the rest were site IDs. Like they mentioned they had a delete script that worked for all types of unique IDs so that also can dilute the feeling of ”it’s all my fault” hopefully.

doctor_eval 4y ago

I was on call “back in the day” when a customer rang to tell me that she’d meant to type: # find /tmp -exec rm {} /; But instead she’d typed # find / tmp -exec rm {} /; I learned a lot about Unix (and, as it happens, X windows) file systems that week. Note that this was long before tools like sudo existed, where cleaning up tmp was often a manual process. There is no way we could blame a person for a single keystroke error - it’s crazy that the command worked - but I now look at the find command with an enormous amount of fear.

BonoboIO 4y ago
Good old days deleting an entire server ... learned that the hard way.
Bender 4y ago
I ran into a similar one a few times from the same team that kept doing things in their scripts like rm -rf ${basedir}/* without setting set -u to exit if there are undefined variables. I suggested a few other sanity checks but was told I was adding friction.