7 ms·

Do people get fired over things like this in the bigger companies?

by ionwake 10y ago

stevewilhelm 10y ago
Not for a single incident. If production outages become a frequent occurrence, the VP of engineering or Ops may not be around much longer.
hmate9 10y ago
Someone probably will get fired. Last quarter amazon earned $20.58bn from product sales. So 4 hours of that is $38,000,000.
- BinaryIdiot 10y ago
  I highly doubt search equates to all of their sales during a period. That would seem odd and would indicate no one uses their navigation at all.
  - jessriedel 10y ago
    Also, Amazon is established enough, and has few enough comparable competitors, that anyone whose search breaks is liable to just try again tomorrow. Still, some people who do go somewhere else, or are less likely to try Amazon in the future. I'd be surprised if this wasn't at least a multi-million dollar mistake.
  - cwilkes 10y ago
    I never do. Going to their homepage is like seeing the dog's breakfast. It is a complete mess of whatever passed an a/b test of increasing sales. It is like they reached a local maxima but it is on St Crappy's Peak. If I go to Amazon.com I just go to the search bar and refine from there. So I guess I use their "navigation" in the form of their groupings of features on the left side.
- deleted 10y ago
  [deleted]
- dspillett 10y ago
  Unless there was deliberate action (skipping procedure in a way that a person has been warned and disciplined about previously) there will be no firing. Any calculation based on the hours down needs to be reduced to account for people finding products by other means (already saved for later in their basket, direct links, searches via external search engines).
- mdjt 10y ago
  This is so inaccurate. There is no way that the distribution of product sales is constant over time of day, day of the week, month of the year, etc.
- mrdrozdov 10y ago
  Firing seems like a toxic move, especially without knowing what happened. I'd like to think that Amazon has multiple layers of alerts built into place for situations like this, so this was probably a complex bug (perhaps a perfect storm of faulty configurations like what happened with Google's Cloud offering not too long ago). I'd say that number is probably a ceiling on their losses, and that the actual number should be much lower. I'm super curious if non-tech news organizations will pick up this story, in which case the biggest hit could be from the PR. That being said, it doesn't seem like such a bad time to go down.
drhayes9 10y ago
I would hope not. I've broken things at big companies before and, mostly, everyone just wants it to work again. Mistakes happen sometimes.
umanwizard 10y ago
Not as bad as this, but my friend was an SDET contractor at amazon and bugs in his testing framework let a multimillion dollar bug to go to prod. Not only was he not fired, but he got a full-time offer at the end of his contract.
- kozhevnikov 10y ago
  And I'm sitting here being forced to pay for Professional Indemnity insurance.
- droffel 10y ago
  Fire someone who they effectively just spent multiple millions of dollars to train? I doubt it. Thats a huge learning experience there, and if anything, will make him more cautious in the future.
deleted 10y ago
[deleted]
witten 10y ago
Yes. One of my co-workers actually got fired from Amazon for causing downtime. And this was after following all procedures, documentation, etc.
electriclove 10y ago
Not in the government.
loliworkatamzn 10y ago
I once caused a 60 minute outage at Amazon and the next day my manager and my skip both begged me not to blame myself and freak out and quit. Mistakes happen. Sometimes they are really expensive.
- goatforce5 10y ago
  Mistakes happen and will always happen. It's how you respond to it that's important.
- partisan 10y ago
  your "skip"?
  - underwater 10y ago
    Skip level, as in manager's manager.
- amzn-336495 10y ago
  I guarantee they told your execs something completely different because Amazon. SOP is to tell you don't worry be happy while loading you up with blame coming down from your director/VP. If you quit, everything they tried to secretly pin on you now focuses on your managers so they are desperate to keep you from quitting. Once you get to OLR the next year you'll get an unpleasant surprise and probably fired. This is called burn and churn management and Amazon is a master of it.
  - deanCommie 10y ago
    Someone needs to change teams. Sounds like you have a toxic org.
    - amzn-336495 10y ago
      Changing teams is pointless. Every manager of every team at Amazon has burner employees whether anyone knows it or not. Part of being an employee is trying to determine if you are the burner. People are hired in as burner employees at Amazon. That is unless Amazon stopped doing company-wide stack ranking, which obviously it will never do since it legitimizes pass-the-buck and is so beneficial to the people making the rules.
xyzzy4 10y ago
I used to work in problem management at a big company. The answer is no unless it was deliberate.
e40 10y ago
I have no idea where I read it, but I think it was about Bezos and a situation just like this. The quote was something like "why would I fire the person that caused the outage? They just learned a very valuable lesson and got great on the job training, why would I fire them now?" When I read it I remember thinking how true it was. That person would be very unlikely to commit the same error, and might even work to prevent others from making that error.
- umanwizard 10y ago
  Also, when I was at Amazon, we were taught that "Person X broke the site" is the wrong way to think about things. There should be enough testing, redundancy, code review and so on to avoid issues given that humans are fallible. So the system failed, not a particular person, and the solution is to dive deep into why the system failed, and improve it. Obviously there are exceptions for willful extreme negligence or policy violations.
  - bonniemuffin 10y ago
    Definitely true -- if it's possible for a single person's change to break Amazon's search, then something is wrong with the system of testing, code review, deployment, etc. It's never just a single person's fault, even if they wrote the causative line of code.
    - goldenkey 10y ago
      Well with Amazon's cult-think, every single person is a "leader" yet also, no one is as important as the shitty middle manager above them. So someone will take the fall but it'll be spun into some type of reckoning consisting of: Ownership, Dive Deep, and Earn Trust [1] As long as you can add to your cult karma, you can do anything at Amazon! https://www.amazon.jobs/principles https://www.amazon.jobs/principles [1]
  - goldenkey 10y ago
    Is that included or in addition to the leadership commandments training cult? [1] [1] https://www.amazon.jobs/principles https://www.amazon.jobs/principles
wrsh07 10y ago
At Google, the answer is a firm no. Someone else commented on the "we just spent a million dollars training them, why would we fire them now?" quote which is commonly thrown around. People get fired for leaks and being intentionally malicious, not mistakes. Even expensive ones. [besides, a big outage is usually caused by multiple smaller issues if your systems are sufficiently reliable. Also, read the SRE book about "blameless postmortems"]
- jrcii 10y ago
  I was watching this talk on Google's SRE and the speaker explains in depth why the answer at Google is indeed no https://www.youtube.com/watch?v=H4vMcD7zKM0 https://www.youtube.com/watch?v=H4vMcD7zKM0 She explains the emphasis is more on learning from the problem and post-mortem than pointing fingers.

Do people get fired over things like this in the bigger companies?

by ionwake 10y ago

Do people get fired over things like this in the bigger companies?

Not for a single incident. If production outages become a frequent occurrence, the VP of engineering or Ops may not be around much longer.

hmate9 10y ago

Someone probably will get fired. Last quarter amazon earned $20.58bn from product sales. So 4 hours of that is $38,000,000.

BinaryIdiot 10y ago
I highly doubt search equates to all of their sales during a period. That would seem odd and would indicate no one uses their navigation at all.
- jessriedel 10y ago
  Also, Amazon is established enough, and has few enough comparable competitors, that anyone whose search breaks is liable to just try again tomorrow. Still, some people who do go somewhere else, or are less likely to try Amazon in the future. I'd be surprised if this wasn't at least a multi-million dollar mistake.
- cwilkes 10y ago
  I never do. Going to their homepage is like seeing the dog's breakfast. It is a complete mess of whatever passed an a/b test of increasing sales. It is like they reached a local maxima but it is on St Crappy's Peak. If I go to Amazon.com I just go to the search bar and refine from there. So I guess I use their "navigation" in the form of their groupings of features on the left side.
deleted 10y ago
[deleted]
dspillett 10y ago
Unless there was deliberate action (skipping procedure in a way that a person has been warned and disciplined about previously) there will be no firing. Any calculation based on the hours down needs to be reduced to account for people finding products by other means (already saved for later in their basket, direct links, searches via external search engines).
mdjt 10y ago
This is so inaccurate. There is no way that the distribution of product sales is constant over time of day, day of the week, month of the year, etc.
mrdrozdov 10y ago
Firing seems like a toxic move, especially without knowing what happened. I'd like to think that Amazon has multiple layers of alerts built into place for situations like this, so this was probably a complex bug (perhaps a perfect storm of faulty configurations like what happened with Google's Cloud offering not too long ago). I'd say that number is probably a ceiling on their losses, and that the actual number should be much lower. I'm super curious if non-tech news organizations will pick up this story, in which case the biggest hit could be from the PR. That being said, it doesn't seem like such a bad time to go down.

drhayes9 10y ago

I would hope not. I've broken things at big companies before and, mostly, everyone just wants it to work again. Mistakes happen sometimes.

umanwizard 10y ago

Not as bad as this, but my friend was an SDET contractor at amazon and bugs in his testing framework let a multimillion dollar bug to go to prod. Not only was he not fired, but he got a full-time offer at the end of his contract.

kozhevnikov 10y ago
And I'm sitting here being forced to pay for Professional Indemnity insurance.
droffel 10y ago
Fire someone who they effectively just spent multiple millions of dollars to train? I doubt it. Thats a huge learning experience there, and if anything, will make him more cautious in the future.

deleted 10y ago

[deleted]

witten 10y ago

Yes. One of my co-workers actually got fired from Amazon for causing downtime. And this was after following all procedures, documentation, etc.

electriclove 10y ago

Not in the government.

loliworkatamzn 10y ago

I once caused a 60 minute outage at Amazon and the next day my manager and my skip both begged me not to blame myself and freak out and quit. Mistakes happen. Sometimes they are really expensive.

goatforce5 10y ago
Mistakes happen and will always happen. It's how you respond to it that's important.
partisan 10y ago
your "skip"?
- underwater 10y ago
  Skip level, as in manager's manager.
amzn-336495 10y ago
I guarantee they told your execs something completely different because Amazon. SOP is to tell you don't worry be happy while loading you up with blame coming down from your director/VP. If you quit, everything they tried to secretly pin on you now focuses on your managers so they are desperate to keep you from quitting. Once you get to OLR the next year you'll get an unpleasant surprise and probably fired. This is called burn and churn management and Amazon is a master of it.
- deanCommie 10y ago
  Someone needs to change teams. Sounds like you have a toxic org.
  - amzn-336495 10y ago
    Changing teams is pointless. Every manager of every team at Amazon has burner employees whether anyone knows it or not. Part of being an employee is trying to determine if you are the burner. People are hired in as burner employees at Amazon. That is unless Amazon stopped doing company-wide stack ranking, which obviously it will never do since it legitimizes pass-the-buck and is so beneficial to the people making the rules.

xyzzy4 10y ago

I used to work in problem management at a big company. The answer is no unless it was deliberate.

e40 10y ago

I have no idea where I read it, but I think it was about Bezos and a situation just like this. The quote was something like "why would I fire the person that caused the outage? They just learned a very valuable lesson and got great on the job training, why would I fire them now?" When I read it I remember thinking how true it was. That person would be very unlikely to commit the same error, and might even work to prevent others from making that error.

umanwizard 10y ago
Also, when I was at Amazon, we were taught that "Person X broke the site" is the wrong way to think about things. There should be enough testing, redundancy, code review and so on to avoid issues given that humans are fallible. So the system failed, not a particular person, and the solution is to dive deep into why the system failed, and improve it. Obviously there are exceptions for willful extreme negligence or policy violations.
- bonniemuffin 10y ago
  Definitely true -- if it's possible for a single person's change to break Amazon's search, then something is wrong with the system of testing, code review, deployment, etc. It's never just a single person's fault, even if they wrote the causative line of code.
  - goldenkey 10y ago
    Well with Amazon's cult-think, every single person is a "leader" yet also, no one is as important as the shitty middle manager above them. So someone will take the fall but it'll be spun into some type of reckoning consisting of: Ownership, Dive Deep, and Earn Trust [1] As long as you can add to your cult karma, you can do anything at Amazon! https://www.amazon.jobs/principles https://www.amazon.jobs/principles [1]
- goldenkey 10y ago
  Is that included or in addition to the leadership commandments training cult? [1] [1] https://www.amazon.jobs/principles https://www.amazon.jobs/principles

wrsh07 10y ago

At Google, the answer is a firm no. Someone else commented on the "we just spent a million dollars training them, why would we fire them now?" quote which is commonly thrown around. People get fired for leaks and being intentionally malicious, not mistakes. Even expensive ones. [besides, a big outage is usually caused by multiple smaller issues if your systems are sufficiently reliable. Also, read the SRE book about "blameless postmortems"]

jrcii 10y ago
I was watching this talk on Google's SRE and the speaker explains in depth why the answer at Google is indeed no https://www.youtube.com/watch?v=H4vMcD7zKM0 https://www.youtube.com/watch?v=H4vMcD7zKM0 She explains the emphasis is more on learning from the problem and post-mortem than pointing fingers.