3 ms·

BINGO. If someone makes a mistake like this, unless it was deliberate or negligent (e.g. he/she didn't follow established procedures), you don't fire them! You

BINGO. If someone makes a mistake like this, unless it was deliberate or negligent (e.g. he/she didn't follow established procedures), you don't fire them! You fix the technical or procedural failures that allowed them to make the mistake in the first place. Heck, you should probably THANK them for making the system better by uncovering a bug! When I was still mostly a Noogler, I took a large portion of Google completely down worldwide for 6 minutes. Guess what? We did a postmortem, fixed the procedural and technical problems that allowed it to happen, and I'm still here!

digi_owl 9y ago
Makes me think of the stories of the early days of school computing, and how the admin would give the kids that hacked said computers part time jobs as they had demonstrated themselves both interested and talented.
nodesocket 9y ago
Love to hear the story how you took down Google for six minutes? Was this a long time ago?
- packetslave 9y ago
  This was in 2012. Let's just say that a router config only getting partially applied (because of a software bug) can cause a LOT of traffic to suddenly go places it's not supposed to go.
  - ronald_raygun 9y ago
    Oh wow, I was a noogler there 4ish years ago, and they still tell that story as an example of "its okay to break things, we care about fixing them after so they dont break in the future"
    - grivescorbett 9y ago
      Was a noogler about a year ago. Story still told.
      - aalleavitch 9y ago
        It appears we have a living legend here!
        ianai 9y ago
        They made him a slave of the packets though.
    - packetslave 9y ago
      There's a couple of good stories about massive outages and good incident response. Mine is just one of them (and at some level, I was very lucky). There's also the one where all the frontend servers worldwide went into a crash loop from a bad configuration push. The SRE doing the push noticed some "weirdness" and rolled back even before the full scope of the issue was known. That one's in the SRE book.
      - mynewtb 9y ago
        GFE? SRE?
        mikejb 9y ago
        SRE is Site Reliability Engineer; GFE is the "Google Front End"
      - Bluestrike2 9y ago
        Site Reliability Engineering.[0] Google's SRE book is a pretty interesting read. 0. https://landing.google.com/sre/interview/ben-treynor.html https://landing.google.com/sre/interview/ben-treynor.html
    - packetslave 9y ago
      I think my favorite quote about my outage was "boy, it's a good thing you didn't decide to go take a crap right about then..."
  - ceph_ 9y ago
    db31.lpp01 lol
    - packetslave 9y ago
      When they decommed it, I asked if I could have db31 to go Office Space on. They said no :-(
someonewhocar3s 9y ago
You thank them for preventing the system from going down. Not for taking it down. But yes, if you performed well and the mistake was not to blame on you, that's not your fault. Still, an easy-mistake (user error caused by software (UX) defect) is a tad harder to place. Of course procedure shouldn't allow easy-errors, but that doesn't quite excuse making one. Also, as a company Google failed for a good 6 minutes. Since the one executing the flawed procedure was the one best positioned to notice, they carry a tad more responsibility than the rest of the org. But really, mostly the executives look bad after such an event, as they have the supreme responsibility.
scoggs 9y ago
Oof! I'm not envious of the way you must have felt when you realized what had happened. We've all (probably) had the same sort of feeling on a much, much smaller scale and I've never been one to enjoy that heart-sinking-into-my-shoes feeling!
simonebrunozzi 9y ago
This was (probably still is) the best approach at AWS in the early days (source: I worked there from 2008 to 2014). Never blame the person, find out why he/she did the mistake and make sure it doesn't happen again.
sa46 9y ago
An interesting variation to this thought is: When is appropriate to fire someone for making a mistake? Some ideas: - gross negligence / malice - ethics - publicity damage / internet pitchforks Additionally, it doesn't seem like the forgive mistakes approach meshes well with the startup advice of "it's never too early to fire."
- Renaud 9y ago
  Incompetence should probably be number one. There is a big difference between someone making a mistake due to unclear or incomplete process and someone doing that mistake because they are not competent to perform that task. This then raises the issue that hiring and leaving incompetent people in place is itself proof of an organisational incompetence...
nfread 9y ago
They say worker hit real button rather than test button. Some folks don’t believe this b/c test button is hit on 1st of month. It is not the 1st of month so why was any button hit, especially one that is so major, how could be accident?

digi_owl 9y ago

Makes me think of the stories of the early days of school computing, and how the admin would give the kids that hacked said computers part time jobs as they had demonstrated themselves both interested and talented.

nodesocket 9y ago

Love to hear the story how you took down Google for six minutes? Was this a long time ago?

packetslave 9y ago
This was in 2012. Let's just say that a router config only getting partially applied (because of a software bug) can cause a LOT of traffic to suddenly go places it's not supposed to go.
- ronald_raygun 9y ago
  Oh wow, I was a noogler there 4ish years ago, and they still tell that story as an example of "its okay to break things, we care about fixing them after so they dont break in the future"
  - grivescorbett 9y ago
    Was a noogler about a year ago. Story still told.
    - aalleavitch 9y ago
      It appears we have a living legend here!
      - ianai 9y ago
        They made him a slave of the packets though.
  - packetslave 9y ago
    There's a couple of good stories about massive outages and good incident response. Mine is just one of them (and at some level, I was very lucky). There's also the one where all the frontend servers worldwide went into a crash loop from a bad configuration push. The SRE doing the push noticed some "weirdness" and rolled back even before the full scope of the issue was known. That one's in the SRE book.
    - mynewtb 9y ago
      GFE? SRE?
      - mikejb 9y ago
        SRE is Site Reliability Engineer; GFE is the "Google Front End"
    - Bluestrike2 9y ago
      Site Reliability Engineering.[0] Google's SRE book is a pretty interesting read. 0. https://landing.google.com/sre/interview/ben-treynor.html https://landing.google.com/sre/interview/ben-treynor.html
  - packetslave 9y ago
    I think my favorite quote about my outage was "boy, it's a good thing you didn't decide to go take a crap right about then..."
- ceph_ 9y ago
  db31.lpp01 lol
  - packetslave 9y ago
    When they decommed it, I asked if I could have db31 to go Office Space on. They said no :-(

someonewhocar3s 9y ago

You thank them for preventing the system from going down. Not for taking it down. But yes, if you performed well and the mistake was not to blame on you, that's not your fault. Still, an easy-mistake (user error caused by software (UX) defect) is a tad harder to place. Of course procedure shouldn't allow easy-errors, but that doesn't quite excuse making one. Also, as a company Google failed for a good 6 minutes. Since the one executing the flawed procedure was the one best positioned to notice, they carry a tad more responsibility than the rest of the org. But really, mostly the executives look bad after such an event, as they have the supreme responsibility.

scoggs 9y ago

Oof! I'm not envious of the way you must have felt when you realized what had happened. We've all (probably) had the same sort of feeling on a much, much smaller scale and I've never been one to enjoy that heart-sinking-into-my-shoes feeling!

simonebrunozzi 9y ago

This was (probably still is) the best approach at AWS in the early days (source: I worked there from 2008 to 2014). Never blame the person, find out why he/she did the mistake and make sure it doesn't happen again.

sa46 9y ago

An interesting variation to this thought is: When is appropriate to fire someone for making a mistake? Some ideas: - gross negligence / malice - ethics - publicity damage / internet pitchforks Additionally, it doesn't seem like the forgive mistakes approach meshes well with the startup advice of "it's never too early to fire."

Renaud 9y ago
Incompetence should probably be number one. There is a big difference between someone making a mistake due to unclear or incomplete process and someone doing that mistake because they are not competent to perform that task. This then raises the issue that hiring and leaving incompetent people in place is itself proof of an organisational incompetence...

nfread 9y ago

They say worker hit real button rather than test button. Some folks don’t believe this b/c test button is hit on 1st of month. It is not the 1st of month so why was any button hit, especially one that is so major, how could be accident?