10 ms·

Just watched an interview on CNN with the Governor of Hawaii and their director of emergency management. Pretty impressed that the director didn't throw anybody

Just watched an interview on CNN with the Governor of Hawaii and their director of emergency management. Pretty impressed that the director didn't throw anybody under the bus. He basically said "An employee pushed the wrong button. It's my responsibility, so this is my fault. We're going to make procedure and technical changes to make sure this doesn't happen again."

blantonl 9y ago
A lot of people are calling for the person who did this to be instantly fired, but I've always been of the opinion that the guy that pushed that button would be an outstanding hire - because he'll never, ever do something like that again and he's learned an important lesson.
- verelo 9y ago
  Also, it’s rarely one persons fault, and the next human is likely going to fail in similar (or worse, different!) ways to the previous human. Process controls variables like this, there is obviously an issue that we can learn from here and adjust to improve.
- packetslave 9y ago
  BINGO. If someone makes a mistake like this, unless it was deliberate or negligent (e.g. he/she didn't follow established procedures), you don't fire them! You fix the technical or procedural failures that allowed them to make the mistake in the first place. Heck, you should probably THANK them for making the system better by uncovering a bug! When I was still mostly a Noogler, I took a large portion of Google completely down worldwide for 6 minutes. Guess what? We did a postmortem, fixed the procedural and technical problems that allowed it to happen, and I'm still here!
  - digi_owl 9y ago
    Makes me think of the stories of the early days of school computing, and how the admin would give the kids that hacked said computers part time jobs as they had demonstrated themselves both interested and talented.
  - nodesocket 9y ago
    Love to hear the story how you took down Google for six minutes? Was this a long time ago?
    - packetslave 9y ago
      This was in 2012. Let's just say that a router config only getting partially applied (because of a software bug) can cause a LOT of traffic to suddenly go places it's not supposed to go.
      - ronald_raygun 9y ago
        Oh wow, I was a noogler there 4ish years ago, and they still tell that story as an example of "its okay to break things, we care about fixing them after so they dont break in the future"
        grivescorbett 9y ago
        Was a noogler about a year ago. Story still told.
        aalleavitch 9y ago
        It appears we have a living legend here!
        ianai 9y ago
        They made him a slave of the packets though.
        packetslave 9y ago
        There's a couple of good stories about massive outages and good incident response. Mine is just one of them (and at some level, I was very lucky). There's also the one where all the frontend servers worldwide went into a crash loop from a bad configuration push. The SRE doing the push noticed some "weirdness" and rolled back even before the full scope of the issue was known. That one's in the SRE book.
  - someonewhocar3s 9y ago
    You thank them for preventing the system from going down. Not for taking it down. But yes, if you performed well and the mistake was not to blame on you, that's not your fault. Still, an easy-mistake (user error caused by software (UX) defect) is a tad harder to place. Of course procedure shouldn't allow easy-errors, but that doesn't quite excuse making one. Also, as a company Google failed for a good 6 minutes. Since the one executing the flawed procedure was the one best positioned to notice, they carry a tad more responsibility than the rest of the org. But really, mostly the executives look bad after such an event, as they have the supreme responsibility.
  - scoggs 9y ago
    Oof! I'm not envious of the way you must have felt when you realized what had happened. We've all (probably) had the same sort of feeling on a much, much smaller scale and I've never been one to enjoy that heart-sinking-into-my-shoes feeling!
  - simonebrunozzi 9y ago
    This was (probably still is) the best approach at AWS in the early days (source: I worked there from 2008 to 2014). Never blame the person, find out why he/she did the mistake and make sure it doesn't happen again.
  - sa46 9y ago
    An interesting variation to this thought is: When is appropriate to fire someone for making a mistake? Some ideas: - gross negligence / malice - ethics - publicity damage / internet pitchforks Additionally, it doesn't seem like the forgive mistakes approach meshes well with the startup advice of "it's never too early to fire."
    - Renaud 9y ago
      Incompetence should probably be number one. There is a big difference between someone making a mistake due to unclear or incomplete process and someone doing that mistake because they are not competent to perform that task. This then raises the issue that hiring and leaving incompetent people in place is itself proof of an organisational incompetence...
  - nfread 9y ago
    They say worker hit real button rather than test button. Some folks don’t believe this b/c test button is hit on 1st of month. It is not the 1st of month so why was any button hit, especially one that is so major, how could be accident?
- throwacide 9y ago
  Misguided. Circumstances are that he either is or is not qualified for a single point of failure job like that. If not it’s management failure. If the human is “qualified” for a single point of failure responsibility like that, he’s now fired.
  - macintux 9y ago
    I suspect that if your criteria is “you can never make a mistake” that no one is qualified for a single point of failure responsibility.
    - throwacide 9y ago
      So, gross management failure. Or, more likely, a hack. Although I don’t see how they would lie that one person really has such power, knowing the truth would likely leak. This isn’t a newbie getting set up on his dev db who was given too much access to production. They’re pointing the finger so that mechanism that facilitates single point of failure should be independently audited
    - bilbo0s 9y ago
      Also, at a more fundamental level... when you're dealing with processes as critical as this one... there should be no single point of failure. Even our President has a backup, a failsafe, and a safeguard for the purposes we're discussing here. And that backup has a backup, a failsafe, and a safeguard. Etc etc etc.
      - Feniks 9y ago
        Not really true. Thanks to advances in technology POTUS had 10 minutes to launch by the 1970s before Washington was toast. It was an open secret that nuclear retaliation could be instigated without White House or even Pentagon approval if the Soviets tried to decapitate Nato.
        lsc 9y ago
        "Of course, the whole point of a Doomsday Machine is lost if you KEEP IT A SECRET!" A rational player would only put effort into a retaliatory strike capability to the extent that actually having a retaliatory strike capability strengthened the enemy's perception that they would be obliterated if they struck first.
- ghaff 9y ago
  I'm always a bit shocked at the number of people here calling for someone's head in this sort of situation. I imagine the same people would downvote to oblivion anyone who came on here and wrote something like: "The only way we're going to get better software is by firing people who write bugs and make sure they never work in the industry again."
  - ImSkeptical 9y ago
    I'd say we just don't have enough information. It's possible someone deserves to be fired for this - e.g. if the alarm was intentional or negligent. It's also possible it was a misclick, confusing UI, etc, in which the person or people who should be fired are those who designed the system such that a simple mistake could set it off. There is a third possibility too - that some unpredictable alignment of factors means that ultimately nobody is too badly at fault, in which case the system should just be adapted and improved. I generally agree with the sentiments of this thread - you shouldn't fire people for the kinds of mistakes that all humans frequently make. However, I also think that this was a severe and costly incident - a large number of people thought for a while they were in significant danger, and investigating who is responsible, and making corrections with regards to their employment, may be reasonable depending on the exact circumstances.
    - ghaff 9y ago
      That's fair and I'm not suggesting that every screw up should just be shrugged off with an "to err is human" and it must be a systems problem. But for a one-off error, the person responsible probably already feels terrible and if they didn't grossly violate established protocols or did something as a joke or whatever, the result should be more along the lines of "Don't do this again!" and making it harder for it to happen.
    - ben_jones 9y ago
      To add to this, I've read more then a few stories already of people mass texting and calling their loved ones because they thought they might never see them again. This is a tragic human toll that needs to be acknowledged and appreciated for what it really is. It wasn't just a smoke alarm going of accidentally.
- nradov 9y ago
  Just to play devil's advocate, in some cases the point of firing an employee in a situation like this isn't about that one employee but more pour encourager les autres. So the remaining employees will see that management takes the issue seriously and they need to be diligent about following procedures. In this particular case no real harm was done and firing would probably be excessive.
  - dzhiurgis 9y ago
    > no real harm was done I wish to believe, but probably there was real harm, probably someone will come up with some million figures. Statistically, probably someone died during those minutes of terror and someone will try to attribute to this incident. What is interesting that employee probably meant no harm, but compare this to that SWAT murder a month earlier. The prankster meant no harm either, but he's going to spend years in jail. Both for sending a wrong alert.
    - CapacitorSet 9y ago
      >What is interesting that employee probably meant no harm, but compare this to that SWAT murder a month earlier. Swatting is intentional, triggering the alert wasn't (at least, so it seems). It's not like the operator pressed the button knowing that it would trigger an alert and expected it not to be taken seriously; he didn't mean to trigger the alert at all.
      - dzhiurgis 9y ago
        What I am pointing at is the paradox of intention. Ongoing car slams into you and kills your whole family while you were driving drunk; you go to jail for manslaughter and get sued by other driver for all you've got.
    - rrix2 9y ago
      > The prankster meant no harm either, Are you kidding? The people who were involved in this didn’t do it to introduce the swattee to new friends. They did it to terrorize someone who made them mad.
- drdeadringer 9y ago
  A chemistry lab instructor I had in university had the same sentiments. Her story was regarding an employee adding water to acid instead of the reverse, thereby creating an acidic cloud initiating an evacuation of the building [we're talking high volumes of acid and water here; think pumping in the water using a garden hose]. "You'll never have me doing this again," the person managed through tears. "No, I'll have you do this from now on as my go-to person," was the response, "because I know you'll never do this again." This was our "Add Acid" lesson.
- nickysielicki 9y ago
  That's one way to look at it. The other perspective is that most people would not need to learn that lesson in the first place. This employee is unlikely to make this type of mistake again, but broadly speaking, is he less likely to make mistakes than someone who wouldn't have made this mistake in the first place? Doubtful.
- deleted 9y ago
  [deleted]
- IncRnd 9y ago
  That may be for some cases, but there should be an extreme amount of due care on the part of employees who can hit a button and cause parents to put their children in storm drains. The particular person who did this may not be fired, but their responsibilities will somehow change.
  - IncRnd 9y ago
    I see my comment was downvoted. Yet, today's news, just a day later, shows that the comment was absolutely correct. The person has been reassigned.
- m3kw9 9y ago
  If the Ui was so deliberate easy to send an alert and it was allowed to be implemented, mistakes will happen, he shouldn’t be fired for that
- hollander 9y ago
  It depends. Much has to do with the way management handles this. If they work with fear, it might happen again. But of course this one man or woman is not responsible for this. The procedure needs to be reviewed of course. Why is such an important event the responsibility of one person? At least two persons should have to press two buttons and it should be made such that one person cannot press them both. People saying nothing happened - I don't agree. This could cause serious problems for individuals, heart attacks, accidents. This is not without risk.
- cm2187 9y ago
  Or perhaps he is careless and should be not be trusted with such a critical duty.
- userbinator 9y ago
  It reminds me of this story from Thomas Watson of IBM: A young executive had made some bad decisions that cost the company several million dollars. He was summoned to Watson’s office, fully expecting to be dismissed. As he entered the office, the young executive said, "I suppose after that set of mistakes you will want to fire me." Watson was said to have replied, "Not at all, young man, we have just spent a couple of million dollars educating you."
  - josephjrobison 9y ago
    More than reminds, it was a direct paraphrase!
  - kregasaurusrex 9y ago
    I'd spoken with Rob Hall of the Watson project during my freshman year of college at a technical talk having seen it Jeopardy! but not knowing much about the technical aspects of what goes into generating an answer based on natural language processing. He took the time to explain to me and a small group of students who stayed for about an hour afterwards about how to go about attacking problems that require both expertise and parallelization on a scale such that confidence levels could be established for each logical answer generated from a tree of possibilities. A great man and team who know the value of educating potential future employees and sharing the knowledge that went into solving a hard problem. I'd typed this up before realizing you were talking about the former CEO rather than the project so I'd thought to post it anyway.
- jrs235 9y ago
  Reminds me of the legend of the newly hired sales guy that managed to screw up and lose a multi million dollar deal that was supposed to be a slam dunk. After the sale feel through the VP of sales scheduled an appointment with the sales guy. The sales guy began cleaning up his desk and boxing his things anticipating what the outcome of the meeting would be. Upon meeting the VP the sales person profusely apologized and accepted responsibility and said he understood why he was being let go. The VP interrupted him and paused for a moment before telling him "I did not invite you here to fire you. I wanted to see how you were doing and ensure that you learned a valuable lesson. The last thing I thought about doing was letting you go after spending millions of dollars on your education and training."
- wdr1 9y ago
  If he's learned. Therein is a key responsibility of the manager: if they've learned, you're 100% right. If they haven't, don't care, don't get it or whatever, then you also need to take responsibility & replace that person.
- dingo_bat 9y ago
  Also, if he was able to do something in error, that means the system is designed wrong. It's a process problem. You cannot fire this one guy and expect this to not happen again.
  - thaumasiotes 9y ago
    > Also, if he was able to do something in error, that means the system is designed wrong. It's a process problem. This doesn't work in all contexts, and I suspect it fails for a large majority of contexts. If the system always prevents you from doing the wrong thing, it will also often prevent you from doing the right thing. For example, I submit that the philosophy "ability to make a mistake means that the system is designed wrong" completely falls down in the case of going to a new restaurant for lunch and finding out you hate the food there. I wish this claim, which is very common in discussions like these, came with some analysis of why this odd philosophy which is a terrible idea in so many contexts is instead a good idea in the context at issue.
tetha 9y ago
That's how it should be. As I maintain - mistakes are places where the systems and automation are too weak for our fat fingers, and mistakes of my team are my mistakes.
justicezyx 9y ago
Now people are impressed with what should be all along. I think a small group of people feel "mission accomplished".
djhworld 9y ago
I wonder what those procedural changes could be Could they do it so say, two people have to input a code or something from different machines, or something more physical like turning two keys, hitting a button etc
- zwily 9y ago
  You mean like https://youtu.be/pnBVIhiUd_g https://youtu.be/pnBVIhiUd_g?
- roywiggins 9y ago
  A big fat "Are you really sure?" confirmation box would seem to be the absolute minimum requirement here...
  - deleted 9y ago
    [deleted]
  - gaius 9y ago
    A previous employer had an “are you sure?” dialogue on Outlook’s reply-all button, you can script it up in VBA. Didn’t stop people from doing it.
    - loco5niner 9y ago
      If it that dialog fired every time I clicked reply-all, I would start ignoring that dialog, especially if it was annoying.
- astura 9y ago
  Putting the button behind glass.
  - chiph 9y ago
    At a military communications center I once worked at we had the EPO button positioned near the door so you knew where it was, and you could hit it on the way out as you evacuated. One day someone had a toolbag in their hand as they were leaving and it swung and hit that big red button ... and it did what it was supposed to do - cutting all the power to the room. Which ended up being a big hassle - they had to reload crypto keys and so on. That very afternoon a plastic guard was installed over the button so you had to lift it first.
nodesocket 9y ago
Kudos. That is how you manage. However my concern is that it took over 30 minutes to send another message stating it was false. Seems like they should have been able to spot their mistake almost instantly. As I commented below, if this happened when the stock market was open, this could have had a huge impact financially. Over 30 minutes with no updates is an eternity for news to hit Wall Street and algo trades start kicking in.
- roywiggins 9y ago
  It could be that there were no "oopsie" messages coded into the system and they didn't have an easy way to send a newly written message out. If it's designed to be used with canned messages, it might take some time to work out how to put a new one in...
  - ithkuil 9y ago
    Or you just don't rush and ensure your mitigation doesn't make things even worse
- loooooooooooool 9y ago
  > Over 30 minutes with no updates is an eternity for news to hit Wall Street and algo trades start kicking in Those poor HFTraders... Who cares?
musicandscones 9y ago
It's sad that a beautiful paradise like Hawaii is still threatened by the scums of the earth (North Korea), propped up by Russia and China. We really need to step up the sactions against Russia and North Korea, and start to impose tariffs and sactions against China. Otherwise a nuclear strike will likely head to hawaii, silicon valley, new york, or tokyo.
- pjc50 9y ago
  There is nowhere else to go with the sanctions against North Korea. What the situation needs is de-escalation.
neurotech1 9y ago
This is known as the "Asoh Defense". Capt. Kohei Asoh accidentally flew a DC-8 airliner into the water while on approach to SFO. No major injuries and the aircraft was repaired. The incident was noted for Capt. Asoh frankly accepting responsibility. [0] https://en.wikipedia.org/wiki/Japan_Airlines_Flight_2#The_"Asoh_defense" https://en.wikipedia.org/wiki/Japan_Airlines_Flight_2#The_"A...
- mistersquid 9y ago
  While Governor Ige did accept full responsibility for the incident, he did not directly execute the technical steps that caused the alert to be sent to HI residents. This is different than Capt. Asoh's defense because Asoh was directly responsible for > attempt[ing] an automatic-coupled Instrument Landing System (ILS) approach, something neither [he nor Capt. Hazen] had done before on a recorded DC-8 flight. In any case, your reference to the defense is fantastic. In the profane spirit of what Asoh told the National Transportation Safety Board investigators--"As you Americans say, I fucked up."--I think it's not too off the mark to observe "What an Asoh". [0] [0] I tried resisting this but it's just sitting there, y'know. Also, I'm sure my American eyes are mispronouncing Captain Asoh's name. For what it's worth, we should all be so forthright to own up to the mistakes we've made.
- totalZero 9y ago
  > Asoh mistakenly landed the plane near Coyote Point in the shallow waters of San Francisco Bay, two and a half miles short of the runway. I was shocked at the 2.5mi deviation until I realized that this is about 51 seconds of flight at his airspeed of 177mi/hr. Still, that's a pretty astonishing distance from the runway.
yorby 9y ago
CNN are not very objective on government issues... hopefully this was a video interview and not a "quote" ... (because videos are a bit harder to fake).
abalone 9y ago
The problem is not the employee who pushed the button. It's whomever designed a system with no checks and balances against a single human operator's error, given the severity of the outcome. No way we should accept errors like this as some sort of blameless par for course in software engineering. Especially as SV grows more and more into life-critical systems like autonomous vehicles. If there was deep incompetence in architecting a system like this then yes, potentially the employees responsible for that architecture could be disciplined. If the responsible employees were too junior to design appropriate safeguards correctly, then the finger points at the managers who set up the team that way. It sounds like the parties to blame here lie somewhere in the management chain between the poor unfortunate button-pusher and the director of all emergency management for Hawaii.
tomjen3 9y ago
Is it going to have any consequences for the governor? Otherwise taking responsibility is just cheap points.
fapjacks 9y ago
So, people on the mainland are reacting differently than people in Hawaii. But this is not unexpected. When you live in such an isolated and vulnerable place, you are prepared for events which take a turn for the worst. I sort of figured people on the mainland would be clutching pearls and calling for the firing of the employee. But we have "false alarms" in Hawaii occasionally (e.g. with tsunami warnings which fizzle out). Most people on the island take the more reasonable position that we seem to have our ducks in a row for alerting, and I'll take an occasional misfired false positive over a false negative. It won't lessen my community's readiness if the real thing happens. Call it a really true-to-life drill. If you don't live in Hawaii and are having this kind of weird panicked reaction to a mistake, perhaps give yourself a minute to really take stock about your perspective on life. It was a mistake, and maybe even helped us become better prepared.
- throwanem 9y ago
  > If you don't live in Hawaii and are having this kind of weird panicked reaction to a mistake There is a lot of fearmongering just lately around the idea that there's going to be a nuclear war with North Korea because the president likes to talk shit on Twitter. I've had so much practice, with my explanation of all the reasons that won't happen, that by now I could do it as an elevator pitch if I needed to. That's why so many people are having this "weird panicked reaction" - panicked, yes, but not weird, when so much effort has been spent to insist that an arrantly implausible counterfactual is not just possible but likely. (On that note, I expect the next wave of articles and op-eds to revolve around Hawaii's recent "close call", and why it means everyone should be much more scared than everyone is already. Who cares about externalities like the health of the republic when there's attention to be monetized?)
JBlue42 9y ago
>We're going to make procedure and technical changes to make sure this doesn't happen again. Must be reading some DevOps blogs about blameless post-mortems.
trumpsamerica 9y ago
This was Trump he needs to be in peached

blantonl 9y ago

A lot of people are calling for the person who did this to be instantly fired, but I've always been of the opinion that the guy that pushed that button would be an outstanding hire - because he'll never, ever do something like that again and he's learned an important lesson.

verelo 9y ago
Also, it’s rarely one persons fault, and the next human is likely going to fail in similar (or worse, different!) ways to the previous human. Process controls variables like this, there is obviously an issue that we can learn from here and adjust to improve.
packetslave 9y ago
BINGO. If someone makes a mistake like this, unless it was deliberate or negligent (e.g. he/she didn't follow established procedures), you don't fire them! You fix the technical or procedural failures that allowed them to make the mistake in the first place. Heck, you should probably THANK them for making the system better by uncovering a bug! When I was still mostly a Noogler, I took a large portion of Google completely down worldwide for 6 minutes. Guess what? We did a postmortem, fixed the procedural and technical problems that allowed it to happen, and I'm still here!
- digi_owl 9y ago
  Makes me think of the stories of the early days of school computing, and how the admin would give the kids that hacked said computers part time jobs as they had demonstrated themselves both interested and talented.
- nodesocket 9y ago
  Love to hear the story how you took down Google for six minutes? Was this a long time ago?
  - packetslave 9y ago
    This was in 2012. Let's just say that a router config only getting partially applied (because of a software bug) can cause a LOT of traffic to suddenly go places it's not supposed to go.
    - ronald_raygun 9y ago
      Oh wow, I was a noogler there 4ish years ago, and they still tell that story as an example of "its okay to break things, we care about fixing them after so they dont break in the future"
      - grivescorbett 9y ago
        Was a noogler about a year ago. Story still told.
        aalleavitch 9y ago
        It appears we have a living legend here!
        ianai 9y ago
        They made him a slave of the packets though.
      - packetslave 9y ago
        There's a couple of good stories about massive outages and good incident response. Mine is just one of them (and at some level, I was very lucky). There's also the one where all the frontend servers worldwide went into a crash loop from a bad configuration push. The SRE doing the push noticed some "weirdness" and rolled back even before the full scope of the issue was known. That one's in the SRE book.
- someonewhocar3s 9y ago
  You thank them for preventing the system from going down. Not for taking it down. But yes, if you performed well and the mistake was not to blame on you, that's not your fault. Still, an easy-mistake (user error caused by software (UX) defect) is a tad harder to place. Of course procedure shouldn't allow easy-errors, but that doesn't quite excuse making one. Also, as a company Google failed for a good 6 minutes. Since the one executing the flawed procedure was the one best positioned to notice, they carry a tad more responsibility than the rest of the org. But really, mostly the executives look bad after such an event, as they have the supreme responsibility.
- scoggs 9y ago
  Oof! I'm not envious of the way you must have felt when you realized what had happened. We've all (probably) had the same sort of feeling on a much, much smaller scale and I've never been one to enjoy that heart-sinking-into-my-shoes feeling!
- simonebrunozzi 9y ago
  This was (probably still is) the best approach at AWS in the early days (source: I worked there from 2008 to 2014). Never blame the person, find out why he/she did the mistake and make sure it doesn't happen again.
- sa46 9y ago
  An interesting variation to this thought is: When is appropriate to fire someone for making a mistake? Some ideas: - gross negligence / malice - ethics - publicity damage / internet pitchforks Additionally, it doesn't seem like the forgive mistakes approach meshes well with the startup advice of "it's never too early to fire."
  - Renaud 9y ago
    Incompetence should probably be number one. There is a big difference between someone making a mistake due to unclear or incomplete process and someone doing that mistake because they are not competent to perform that task. This then raises the issue that hiring and leaving incompetent people in place is itself proof of an organisational incompetence...
- nfread 9y ago
  They say worker hit real button rather than test button. Some folks don’t believe this b/c test button is hit on 1st of month. It is not the 1st of month so why was any button hit, especially one that is so major, how could be accident?
throwacide 9y ago
Misguided. Circumstances are that he either is or is not qualified for a single point of failure job like that. If not it’s management failure. If the human is “qualified” for a single point of failure responsibility like that, he’s now fired.
- macintux 9y ago
  I suspect that if your criteria is “you can never make a mistake” that no one is qualified for a single point of failure responsibility.
  - throwacide 9y ago
    So, gross management failure. Or, more likely, a hack. Although I don’t see how they would lie that one person really has such power, knowing the truth would likely leak. This isn’t a newbie getting set up on his dev db who was given too much access to production. They’re pointing the finger so that mechanism that facilitates single point of failure should be independently audited
  - bilbo0s 9y ago
    Also, at a more fundamental level... when you're dealing with processes as critical as this one... there should be no single point of failure. Even our President has a backup, a failsafe, and a safeguard for the purposes we're discussing here. And that backup has a backup, a failsafe, and a safeguard. Etc etc etc.
    - Feniks 9y ago
      Not really true. Thanks to advances in technology POTUS had 10 minutes to launch by the 1970s before Washington was toast. It was an open secret that nuclear retaliation could be instigated without White House or even Pentagon approval if the Soviets tried to decapitate Nato.
      - lsc 9y ago
        "Of course, the whole point of a Doomsday Machine is lost if you KEEP IT A SECRET!" A rational player would only put effort into a retaliatory strike capability to the extent that actually having a retaliatory strike capability strengthened the enemy's perception that they would be obliterated if they struck first.
ghaff 9y ago
I'm always a bit shocked at the number of people here calling for someone's head in this sort of situation. I imagine the same people would downvote to oblivion anyone who came on here and wrote something like: "The only way we're going to get better software is by firing people who write bugs and make sure they never work in the industry again."
- ImSkeptical 9y ago
  I'd say we just don't have enough information. It's possible someone deserves to be fired for this - e.g. if the alarm was intentional or negligent. It's also possible it was a misclick, confusing UI, etc, in which the person or people who should be fired are those who designed the system such that a simple mistake could set it off. There is a third possibility too - that some unpredictable alignment of factors means that ultimately nobody is too badly at fault, in which case the system should just be adapted and improved. I generally agree with the sentiments of this thread - you shouldn't fire people for the kinds of mistakes that all humans frequently make. However, I also think that this was a severe and costly incident - a large number of people thought for a while they were in significant danger, and investigating who is responsible, and making corrections with regards to their employment, may be reasonable depending on the exact circumstances.
  - ghaff 9y ago
    That's fair and I'm not suggesting that every screw up should just be shrugged off with an "to err is human" and it must be a systems problem. But for a one-off error, the person responsible probably already feels terrible and if they didn't grossly violate established protocols or did something as a joke or whatever, the result should be more along the lines of "Don't do this again!" and making it harder for it to happen.
  - ben_jones 9y ago
    To add to this, I've read more then a few stories already of people mass texting and calling their loved ones because they thought they might never see them again. This is a tragic human toll that needs to be acknowledged and appreciated for what it really is. It wasn't just a smoke alarm going of accidentally.
nradov 9y ago
Just to play devil's advocate, in some cases the point of firing an employee in a situation like this isn't about that one employee but more pour encourager les autres. So the remaining employees will see that management takes the issue seriously and they need to be diligent about following procedures. In this particular case no real harm was done and firing would probably be excessive.
- dzhiurgis 9y ago
  > no real harm was done I wish to believe, but probably there was real harm, probably someone will come up with some million figures. Statistically, probably someone died during those minutes of terror and someone will try to attribute to this incident. What is interesting that employee probably meant no harm, but compare this to that SWAT murder a month earlier. The prankster meant no harm either, but he's going to spend years in jail. Both for sending a wrong alert.
  - CapacitorSet 9y ago
    >What is interesting that employee probably meant no harm, but compare this to that SWAT murder a month earlier. Swatting is intentional, triggering the alert wasn't (at least, so it seems). It's not like the operator pressed the button knowing that it would trigger an alert and expected it not to be taken seriously; he didn't mean to trigger the alert at all.
    - dzhiurgis 9y ago
      What I am pointing at is the paradox of intention. Ongoing car slams into you and kills your whole family while you were driving drunk; you go to jail for manslaughter and get sued by other driver for all you've got.
  - rrix2 9y ago
    > The prankster meant no harm either, Are you kidding? The people who were involved in this didn’t do it to introduce the swattee to new friends. They did it to terrorize someone who made them mad.
drdeadringer 9y ago
A chemistry lab instructor I had in university had the same sentiments. Her story was regarding an employee adding water to acid instead of the reverse, thereby creating an acidic cloud initiating an evacuation of the building [we're talking high volumes of acid and water here; think pumping in the water using a garden hose]. "You'll never have me doing this again," the person managed through tears. "No, I'll have you do this from now on as my go-to person," was the response, "because I know you'll never do this again." This was our "Add Acid" lesson.
nickysielicki 9y ago
That's one way to look at it. The other perspective is that most people would not need to learn that lesson in the first place. This employee is unlikely to make this type of mistake again, but broadly speaking, is he less likely to make mistakes than someone who wouldn't have made this mistake in the first place? Doubtful.
deleted 9y ago
[deleted]
IncRnd 9y ago
That may be for some cases, but there should be an extreme amount of due care on the part of employees who can hit a button and cause parents to put their children in storm drains. The particular person who did this may not be fired, but their responsibilities will somehow change.
- IncRnd 9y ago
  I see my comment was downvoted. Yet, today's news, just a day later, shows that the comment was absolutely correct. The person has been reassigned.
m3kw9 9y ago
If the Ui was so deliberate easy to send an alert and it was allowed to be implemented, mistakes will happen, he shouldn’t be fired for that
hollander 9y ago
It depends. Much has to do with the way management handles this. If they work with fear, it might happen again. But of course this one man or woman is not responsible for this. The procedure needs to be reviewed of course. Why is such an important event the responsibility of one person? At least two persons should have to press two buttons and it should be made such that one person cannot press them both. People saying nothing happened - I don't agree. This could cause serious problems for individuals, heart attacks, accidents. This is not without risk.
cm2187 9y ago
Or perhaps he is careless and should be not be trusted with such a critical duty.
userbinator 9y ago
It reminds me of this story from Thomas Watson of IBM: A young executive had made some bad decisions that cost the company several million dollars. He was summoned to Watson’s office, fully expecting to be dismissed. As he entered the office, the young executive said, "I suppose after that set of mistakes you will want to fire me." Watson was said to have replied, "Not at all, young man, we have just spent a couple of million dollars educating you."
- josephjrobison 9y ago
  More than reminds, it was a direct paraphrase!
- kregasaurusrex 9y ago
  I'd spoken with Rob Hall of the Watson project during my freshman year of college at a technical talk having seen it Jeopardy! but not knowing much about the technical aspects of what goes into generating an answer based on natural language processing. He took the time to explain to me and a small group of students who stayed for about an hour afterwards about how to go about attacking problems that require both expertise and parallelization on a scale such that confidence levels could be established for each logical answer generated from a tree of possibilities. A great man and team who know the value of educating potential future employees and sharing the knowledge that went into solving a hard problem. I'd typed this up before realizing you were talking about the former CEO rather than the project so I'd thought to post it anyway.
jrs235 9y ago
Reminds me of the legend of the newly hired sales guy that managed to screw up and lose a multi million dollar deal that was supposed to be a slam dunk. After the sale feel through the VP of sales scheduled an appointment with the sales guy. The sales guy began cleaning up his desk and boxing his things anticipating what the outcome of the meeting would be. Upon meeting the VP the sales person profusely apologized and accepted responsibility and said he understood why he was being let go. The VP interrupted him and paused for a moment before telling him "I did not invite you here to fire you. I wanted to see how you were doing and ensure that you learned a valuable lesson. The last thing I thought about doing was letting you go after spending millions of dollars on your education and training."
wdr1 9y ago
If he's learned. Therein is a key responsibility of the manager: if they've learned, you're 100% right. If they haven't, don't care, don't get it or whatever, then you also need to take responsibility & replace that person.
dingo_bat 9y ago
Also, if he was able to do something in error, that means the system is designed wrong. It's a process problem. You cannot fire this one guy and expect this to not happen again.
- thaumasiotes 9y ago
  > Also, if he was able to do something in error, that means the system is designed wrong. It's a process problem. This doesn't work in all contexts, and I suspect it fails for a large majority of contexts. If the system always prevents you from doing the wrong thing, it will also often prevent you from doing the right thing. For example, I submit that the philosophy "ability to make a mistake means that the system is designed wrong" completely falls down in the case of going to a new restaurant for lunch and finding out you hate the food there. I wish this claim, which is very common in discussions like these, came with some analysis of why this odd philosophy which is a terrible idea in so many contexts is instead a good idea in the context at issue.

tetha 9y ago

That's how it should be. As I maintain - mistakes are places where the systems and automation are too weak for our fat fingers, and mistakes of my team are my mistakes.

justicezyx 9y ago

Now people are impressed with what should be all along. I think a small group of people feel "mission accomplished".

djhworld 9y ago

I wonder what those procedural changes could be Could they do it so say, two people have to input a code or something from different machines, or something more physical like turning two keys, hitting a button etc

zwily 9y ago
You mean like https://youtu.be/pnBVIhiUd_g https://youtu.be/pnBVIhiUd_g?
roywiggins 9y ago
A big fat "Are you really sure?" confirmation box would seem to be the absolute minimum requirement here...
- deleted 9y ago
  [deleted]
- gaius 9y ago
  A previous employer had an “are you sure?” dialogue on Outlook’s reply-all button, you can script it up in VBA. Didn’t stop people from doing it.
  - loco5niner 9y ago
    If it that dialog fired every time I clicked reply-all, I would start ignoring that dialog, especially if it was annoying.
astura 9y ago
Putting the button behind glass.
- chiph 9y ago
  At a military communications center I once worked at we had the EPO button positioned near the door so you knew where it was, and you could hit it on the way out as you evacuated. One day someone had a toolbag in their hand as they were leaving and it swung and hit that big red button ... and it did what it was supposed to do - cutting all the power to the room. Which ended up being a big hassle - they had to reload crypto keys and so on. That very afternoon a plastic guard was installed over the button so you had to lift it first.

nodesocket 9y ago

Kudos. That is how you manage. However my concern is that it took over 30 minutes to send another message stating it was false. Seems like they should have been able to spot their mistake almost instantly. As I commented below, if this happened when the stock market was open, this could have had a huge impact financially. Over 30 minutes with no updates is an eternity for news to hit Wall Street and algo trades start kicking in.

roywiggins 9y ago
It could be that there were no "oopsie" messages coded into the system and they didn't have an easy way to send a newly written message out. If it's designed to be used with canned messages, it might take some time to work out how to put a new one in...
- ithkuil 9y ago
  Or you just don't rush and ensure your mitigation doesn't make things even worse
loooooooooooool 9y ago
> Over 30 minutes with no updates is an eternity for news to hit Wall Street and algo trades start kicking in Those poor HFTraders... Who cares?

musicandscones 9y ago

It's sad that a beautiful paradise like Hawaii is still threatened by the scums of the earth (North Korea), propped up by Russia and China. We really need to step up the sactions against Russia and North Korea, and start to impose tariffs and sactions against China. Otherwise a nuclear strike will likely head to hawaii, silicon valley, new york, or tokyo.

pjc50 9y ago
There is nowhere else to go with the sanctions against North Korea. What the situation needs is de-escalation.

neurotech1 9y ago

This is known as the "Asoh Defense". Capt. Kohei Asoh accidentally flew a DC-8 airliner into the water while on approach to SFO. No major injuries and the aircraft was repaired. The incident was noted for Capt. Asoh frankly accepting responsibility. [0] https://en.wikipedia.org/wiki/Japan_Airlines_Flight_2#The_"Asoh_defense" https://en.wikipedia.org/wiki/Japan_Airlines_Flight_2#The_"A...

mistersquid 9y ago
While Governor Ige did accept full responsibility for the incident, he did not directly execute the technical steps that caused the alert to be sent to HI residents. This is different than Capt. Asoh's defense because Asoh was directly responsible for > attempt[ing] an automatic-coupled Instrument Landing System (ILS) approach, something neither [he nor Capt. Hazen] had done before on a recorded DC-8 flight. In any case, your reference to the defense is fantastic. In the profane spirit of what Asoh told the National Transportation Safety Board investigators--"As you Americans say, I fucked up."--I think it's not too off the mark to observe "What an Asoh". [0] [0] I tried resisting this but it's just sitting there, y'know. Also, I'm sure my American eyes are mispronouncing Captain Asoh's name. For what it's worth, we should all be so forthright to own up to the mistakes we've made.
totalZero 9y ago
> Asoh mistakenly landed the plane near Coyote Point in the shallow waters of San Francisco Bay, two and a half miles short of the runway. I was shocked at the 2.5mi deviation until I realized that this is about 51 seconds of flight at his airspeed of 177mi/hr. Still, that's a pretty astonishing distance from the runway.

yorby 9y ago

CNN are not very objective on government issues... hopefully this was a video interview and not a "quote" ... (because videos are a bit harder to fake).

abalone 9y ago

The problem is not the employee who pushed the button. It's whomever designed a system with no checks and balances against a single human operator's error, given the severity of the outcome. No way we should accept errors like this as some sort of blameless par for course in software engineering. Especially as SV grows more and more into life-critical systems like autonomous vehicles. If there was deep incompetence in architecting a system like this then yes, potentially the employees responsible for that architecture could be disciplined. If the responsible employees were too junior to design appropriate safeguards correctly, then the finger points at the managers who set up the team that way. It sounds like the parties to blame here lie somewhere in the management chain between the poor unfortunate button-pusher and the director of all emergency management for Hawaii.

tomjen3 9y ago

Is it going to have any consequences for the governor? Otherwise taking responsibility is just cheap points.

fapjacks 9y ago

So, people on the mainland are reacting differently than people in Hawaii. But this is not unexpected. When you live in such an isolated and vulnerable place, you are prepared for events which take a turn for the worst. I sort of figured people on the mainland would be clutching pearls and calling for the firing of the employee. But we have "false alarms" in Hawaii occasionally (e.g. with tsunami warnings which fizzle out). Most people on the island take the more reasonable position that we seem to have our ducks in a row for alerting, and I'll take an occasional misfired false positive over a false negative. It won't lessen my community's readiness if the real thing happens. Call it a really true-to-life drill. If you don't live in Hawaii and are having this kind of weird panicked reaction to a mistake, perhaps give yourself a minute to really take stock about your perspective on life. It was a mistake, and maybe even helped us become better prepared.

throwanem 9y ago
> If you don't live in Hawaii and are having this kind of weird panicked reaction to a mistake There is a lot of fearmongering just lately around the idea that there's going to be a nuclear war with North Korea because the president likes to talk shit on Twitter. I've had so much practice, with my explanation of all the reasons that won't happen, that by now I could do it as an elevator pitch if I needed to. That's why so many people are having this "weird panicked reaction" - panicked, yes, but not weird, when so much effort has been spent to insist that an arrantly implausible counterfactual is not just possible but likely. (On that note, I expect the next wave of articles and op-eds to revolve around Hawaii's recent "close call", and why it means everyone should be much more scared than everyone is already. Who cares about externalities like the health of the republic when there's attention to be monetized?)

JBlue42 9y ago

>We're going to make procedure and technical changes to make sure this doesn't happen again. Must be reading some DevOps blogs about blameless post-mortems.

trumpsamerica 9y ago

This was Trump he needs to be in peached