9 ms·

A lot of people are calling for the person who did this to be instantly fired, but I've always been of the opinion that the guy that pushed that button would be

• verelo 9y ago
  Also, it’s rarely one persons fault, and the next human is likely going to fail in similar (or worse, different!) ways to the previous human. Process controls variables like this, there is obviously an issue that we can learn from here and adjust to improve.
• packetslave 9y ago
  BINGO. If someone makes a mistake like this, unless it was deliberate or negligent (e.g. he/she didn't follow established procedures), you don't fire them! You fix the technical or procedural failures that allowed them to make the mistake in the first place. Heck, you should probably THANK them for making the system better by uncovering a bug! When I was still mostly a Noogler, I took a large portion of Google completely down worldwide for 6 minutes. Guess what? We did a postmortem, fixed the procedural and technical problems that allowed it to happen, and I'm still here!
  • digi_owl 9y ago
    Makes me think of the stories of the early days of school computing, and how the admin would give the kids that hacked said computers part time jobs as they had demonstrated themselves both interested and talented.
  • nodesocket 9y ago
    Love to hear the story how you took down Google for six minutes? Was this a long time ago?
    • packetslave 9y ago
      This was in 2012. Let's just say that a router config only getting partially applied (because of a software bug) can cause a LOT of traffic to suddenly go places it's not supposed to go.
      • ronald_raygun 9y ago
        Oh wow, I was a noogler there 4ish years ago, and they still tell that story as an example of "its okay to break things, we care about fixing them after so they dont break in the future"
        • grivescorbett 9y ago
          Was a noogler about a year ago. Story still told.
          • aalleavitch 9y ago
            It appears we have a living legend here!
            • ianai 9y ago
              They made him a slave of the packets though.
        • packetslave 9y ago
          There's a couple of good stories about massive outages and good incident response. Mine is just one of them (and at some level, I was very lucky). There's also the one where all the frontend servers worldwide went into a crash loop from a bad configuration push. The SRE doing the push noticed some "weirdness" and rolled back even before the full scope of the issue was known. That one's in the SRE book.
          • mynewtb 9y ago
  • someonewhocar3s 9y ago
    You thank them for preventing the system from going down. Not for taking it down. But yes, if you performed well and the mistake was not to blame on you, that's not your fault. Still, an easy-mistake (user error caused by software (UX) defect) is a tad harder to place. Of course procedure shouldn't allow easy-errors, but that doesn't quite excuse making one. Also, as a company Google failed for a good 6 minutes. Since the one executing the flawed procedure was the one best positioned to notice, they carry a tad more responsibility than the rest of the org. But really, mostly the executives look bad after such an event, as they have the supreme responsibility.
  • scoggs 9y ago
    Oof! I'm not envious of the way you must have felt when you realized what had happened. We've all (probably) had the same sort of feeling on a much, much smaller scale and I've never been one to enjoy that heart-sinking-into-my-shoes feeling!
  • simonebrunozzi 9y ago
    This was (probably still is) the best approach at AWS in the early days (source: I worked there from 2008 to 2014). Never blame the person, find out why he/she did the mistake and make sure it doesn't happen again.
  • sa46 9y ago
    An interesting variation to this thought is: When is appropriate to fire someone for making a mistake? Some ideas: - gross negligence / malice - ethics - publicity damage / internet pitchforks Additionally, it doesn't seem like the forgive mistakes approach meshes well with the startup advice of "it's never too early to fire."
    • Renaud 9y ago
      Incompetence should probably be number one. There is a big difference between someone making a mistake due to unclear or incomplete process and someone doing that mistake because they are not competent to perform that task. This then raises the issue that hiring and leaving incompetent people in place is itself proof of an organisational incompetence...
  • nfread 9y ago
    They say worker hit real button rather than test button. Some folks don’t believe this b/c test button is hit on 1st of month. It is not the 1st of month so why was any button hit, especially one that is so major, how could be accident?
• throwacide 9y ago
  Misguided. Circumstances are that he either is or is not qualified for a single point of failure job like that. If not it’s management failure. If the human is “qualified” for a single point of failure responsibility like that, he’s now fired.
  • macintux 9y ago
    I suspect that if your criteria is “you can never make a mistake” that no one is qualified for a single point of failure responsibility.
    • throwacide 9y ago
      So, gross management failure. Or, more likely, a hack. Although I don’t see how they would lie that one person really has such power, knowing the truth would likely leak. This isn’t a newbie getting set up on his dev db who was given too much access to production. They’re pointing the finger so that mechanism that facilitates single point of failure should be independently audited
    • bilbo0s 9y ago
      Also, at a more fundamental level... when you're dealing with processes as critical as this one... there should be no single point of failure. Even our President has a backup, a failsafe, and a safeguard for the purposes we're discussing here. And that backup has a backup, a failsafe, and a safeguard. Etc etc etc.
      • Feniks 9y ago
        Not really true. Thanks to advances in technology POTUS had 10 minutes to launch by the 1970s before Washington was toast. It was an open secret that nuclear retaliation could be instigated without White House or even Pentagon approval if the Soviets tried to decapitate Nato.
        • lsc 9y ago
          "Of course, the whole point of a Doomsday Machine is lost if you KEEP IT A SECRET!" A rational player would only put effort into a retaliatory strike capability to the extent that actually having a retaliatory strike capability strengthened the enemy's perception that they would be obliterated if they struck first.
• ghaff 9y ago
  I'm always a bit shocked at the number of people here calling for someone's head in this sort of situation. I imagine the same people would downvote to oblivion anyone who came on here and wrote something like: "The only way we're going to get better software is by firing people who write bugs and make sure they never work in the industry again."
  • ImSkeptical 9y ago
    I'd say we just don't have enough information. It's possible someone deserves to be fired for this - e.g. if the alarm was intentional or negligent. It's also possible it was a misclick, confusing UI, etc, in which the person or people who should be fired are those who designed the system such that a simple mistake could set it off. There is a third possibility too - that some unpredictable alignment of factors means that ultimately nobody is too badly at fault, in which case the system should just be adapted and improved. I generally agree with the sentiments of this thread - you shouldn't fire people for the kinds of mistakes that all humans frequently make. However, I also think that this was a severe and costly incident - a large number of people thought for a while they were in significant danger, and investigating who is responsible, and making corrections with regards to their employment, may be reasonable depending on the exact circumstances.
    • ghaff 9y ago
      That's fair and I'm not suggesting that every screw up should just be shrugged off with an "to err is human" and it must be a systems problem. But for a one-off error, the person responsible probably already feels terrible and if they didn't grossly violate established protocols or did something as a joke or whatever, the result should be more along the lines of "Don't do this again!" and making it harder for it to happen.
    • ben_jones 9y ago
      To add to this, I've read more then a few stories already of people mass texting and calling their loved ones because they thought they might never see them again. This is a tragic human toll that needs to be acknowledged and appreciated for what it really is. It wasn't just a smoke alarm going of accidentally.
• nradov 9y ago
  Just to play devil's advocate, in some cases the point of firing an employee in a situation like this isn't about that one employee but more pour encourager les autres. So the remaining employees will see that management takes the issue seriously and they need to be diligent about following procedures. In this particular case no real harm was done and firing would probably be excessive.
  • dzhiurgis 9y ago
    > no real harm was done I wish to believe, but probably there was real harm, probably someone will come up with some million figures. Statistically, probably someone died during those minutes of terror and someone will try to attribute to this incident. What is interesting that employee probably meant no harm, but compare this to that SWAT murder a month earlier. The prankster meant no harm either, but he's going to spend years in jail. Both for sending a wrong alert.
    • CapacitorSet 9y ago
      >What is interesting that employee probably meant no harm, but compare this to that SWAT murder a month earlier. Swatting is intentional, triggering the alert wasn't (at least, so it seems). It's not like the operator pressed the button knowing that it would trigger an alert and expected it not to be taken seriously; he didn't mean to trigger the alert at all.
      • dzhiurgis 9y ago
        What I am pointing at is the paradox of intention. Ongoing car slams into you and kills your whole family while you were driving drunk; you go to jail for manslaughter and get sued by other driver for all you've got.
    • rrix2 9y ago
      > The prankster meant no harm either, Are you kidding? The people who were involved in this didn’t do it to introduce the swattee to new friends. They did it to terrorize someone who made them mad.
• drdeadringer 9y ago
  A chemistry lab instructor I had in university had the same sentiments. Her story was regarding an employee adding water to acid instead of the reverse, thereby creating an acidic cloud initiating an evacuation of the building [we're talking high volumes of acid and water here; think pumping in the water using a garden hose]. "You'll never have me doing this again," the person managed through tears. "No, I'll have you do this from now on as my go-to person," was the response, "because I know you'll never do this again." This was our "Add Acid" lesson.
• nickysielicki 9y ago
  That's one way to look at it. The other perspective is that most people would not need to learn that lesson in the first place. This employee is unlikely to make this type of mistake again, but broadly speaking, is he less likely to make mistakes than someone who wouldn't have made this mistake in the first place? Doubtful.
• deleted 9y ago
  [deleted]
• IncRnd 9y ago
  That may be for some cases, but there should be an extreme amount of due care on the part of employees who can hit a button and cause parents to put their children in storm drains. The particular person who did this may not be fired, but their responsibilities will somehow change.
  • IncRnd 9y ago
    I see my comment was downvoted. Yet, today's news, just a day later, shows that the comment was absolutely correct. The person has been reassigned.
• m3kw9 9y ago
  If the Ui was so deliberate easy to send an alert and it was allowed to be implemented, mistakes will happen, he shouldn’t be fired for that
• hollander 9y ago
  It depends. Much has to do with the way management handles this. If they work with fear, it might happen again. But of course this one man or woman is not responsible for this. The procedure needs to be reviewed of course. Why is such an important event the responsibility of one person? At least two persons should have to press two buttons and it should be made such that one person cannot press them both. People saying nothing happened - I don't agree. This could cause serious problems for individuals, heart attacks, accidents. This is not without risk.
• cm2187 9y ago
  Or perhaps he is careless and should be not be trusted with such a critical duty.
• userbinator 9y ago
  It reminds me of this story from Thomas Watson of IBM: A young executive had made some bad decisions that cost the company several million dollars. He was summoned to Watson’s office, fully expecting to be dismissed. As he entered the office, the young executive said, "I suppose after that set of mistakes you will want to fire me." Watson was said to have replied, "Not at all, young man, we have just spent a couple of million dollars educating you."
  • josephjrobison 9y ago
    More than reminds, it was a direct paraphrase!
  • kregasaurusrex 9y ago
    I'd spoken with Rob Hall of the Watson project during my freshman year of college at a technical talk having seen it Jeopardy! but not knowing much about the technical aspects of what goes into generating an answer based on natural language processing. He took the time to explain to me and a small group of students who stayed for about an hour afterwards about how to go about attacking problems that require both expertise and parallelization on a scale such that confidence levels could be established for each logical answer generated from a tree of possibilities. A great man and team who know the value of educating potential future employees and sharing the knowledge that went into solving a hard problem. I'd typed this up before realizing you were talking about the former CEO rather than the project so I'd thought to post it anyway.
• jrs235 9y ago
  Reminds me of the legend of the newly hired sales guy that managed to screw up and lose a multi million dollar deal that was supposed to be a slam dunk. After the sale feel through the VP of sales scheduled an appointment with the sales guy. The sales guy began cleaning up his desk and boxing his things anticipating what the outcome of the meeting would be. Upon meeting the VP the sales person profusely apologized and accepted responsibility and said he understood why he was being let go. The VP interrupted him and paused for a moment before telling him "I did not invite you here to fire you. I wanted to see how you were doing and ensure that you learned a valuable lesson. The last thing I thought about doing was letting you go after spending millions of dollars on your education and training."
• wdr1 9y ago
  If he's learned. Therein is a key responsibility of the manager: if they've learned, you're 100% right. If they haven't, don't care, don't get it or whatever, then you also need to take responsibility & replace that person.
• dingo_bat 9y ago
  Also, if he was able to do something in error, that means the system is designed wrong. It's a process problem. You cannot fire this one guy and expect this to not happen again.
  • thaumasiotes 9y ago
    > Also, if he was able to do something in error, that means the system is designed wrong. It's a process problem. This doesn't work in all contexts, and I suspect it fails for a large majority of contexts. If the system always prevents you from doing the wrong thing, it will also often prevent you from doing the right thing. For example, I submit that the philosophy "ability to make a mistake means that the system is designed wrong" completely falls down in the case of going to a new restaurant for lunch and finding out you hate the food there. I wish this claim, which is very common in discussions like these, came with some analysis of why this odd philosophy which is a terrible idea in so many contexts is instead a good idea in the context at issue.