5 ms·

Years ago, not long after I started at Amazon, there was a huge Netflix outage[0]. It surfaced - or at least was widely speculated - that the cause was a pretty

Years ago, not long after I started at Amazon, there was a huge Netflix outage[0]. It surfaced - or at least was widely speculated - that the cause was a pretty green employee running a DROP TABLE command against a prod database instead of a dev environment. One morning when I came in and sat down at my desk, all of the old-timers were having coffee and discussing the fiasco. I was very happy to hear all of them talk about how mistakes happen, and the last person to be blamed for such an outage is the poor guy or gal that hit the ENTER button. Rather, blame falls (to various degrees) on: the engineers in their orbit who should be backing them up; the managers helping to onboard them; the chain of command; the entire system that is in place to prevent inappropriate access. One of my best early-in-career lessons was that it takes maturity to own up to your mistakes (no matter how bone-headed), and it also takes good managers and a good company to foster an environment in which you can own up to them without fear of losing your job. Any company that wants to hang a weight around an intern's neck for something like this is not a company I would want to support in any way. [0] https://netflixtechblog.com/a-closer-look-at-the-christmas-eve-outage-d7b409a529ee https://netflixtechblog.com/a-closer-look-at-the-christmas-e...

kazen44 6y ago
> One of my best early-in-career lessons was that it takes maturity to own up to your mistakes (no matter how bone-headed), and it also takes good managers and a good company to foster an environment in which you can own up to them without fear of losing your job. Any company that wants to hang a weight around an intern's neck for something like this is not a company I would want to support in any way. owning up to your mistakes also gives you an incredible amount of credibility and respect in my opinion. Mistakes happen, especially in complex systems. Owning up to mistakes and explaining your reasoning about your actions makes you and your compatriots better engineers. Excluding malicious action, most people make a (semi) critical error sometime in their career, especially if you work on the ops side of things, these can often be disasterous. Engineering who claim they never have made a mistake that usually either not working on anything that has value or are just lucky in my opinion. engineering who are afraid to say they made a mistake are a cultural issue aswell, because it delays troubleshooting during incidents. Something we tell employees during our onboarding in a technical comes down to this. - reason about a problem by yourself first - think about impact before you do a change, if in doubt, ask and doublecheck. - admit mistakes when you realise them, explain your reasoning and why you did the action. - learn from your mistakes, but accept that being error-free is simply not possible.
- ClumsyPilot 6y ago
  "Engineering who claim they never have made a mistake that usually either not working on anything that has value or are just lucky" Third option: they don't realise when they make a mistake, either because they are not smart enough or too full of themselves.
  - MikeDelta 6y ago
    Like the engineers who never have to refactor, because their code is good enough in the first go. The only way to guaranteed never have mistakes in your code, is to not have any code at all.
    - dustymcp 6y ago
      To be fair sometimes people hyper focus on refactoring rather than fixing business problems, which isnt really helpful either, have to find a decent middle ground.
      - MikeDelta 6y ago
        You are absolutely right, I forgot about the other extreme.
- mns 6y ago
  > owning up to your mistakes also gives you an incredible amount of credibility and respect in my opinion. Mistakes happen, especially in complex systems. Owning up to mistakes and explaining your reasoning about your actions makes you and your compatriots better engineers. If you are in the right company. I made a big mistake at one point. We had all of the people responsible for one of the payment methods away on holiday (this one payment method was managed by another team in another country, as it wasn't using our normal payment gateway and provider that my team was maintaining). Huge panic, someone needs to fix this, you can't use X any more for completing your order. I'm in the right team that's handling payments and fulfilment, only one at work at that point so I'm told to fix it. I do, I fix it, send it over to testing, get the green light, fix is deployed, everyone is happy. 2 hours later, we figure out the payments are working, but the orders are not being finalised and are still in unpaid. We realised that that single payment method done by this other team in another country was not using our standard payment processing workflow and it has a different way of actually getting the confirmation from the payment provider. This was quite a big company, we had around tens of thousands of Euros blocked in those 2 hours. I own up to it, I admin I made a mistake, go deeper (we did not have anything about this documented) fix it again, we unblock everything, all good. Until 1 month later I got fired (there were layoffs because of larger financial issues, but I was on the list because of the incident), it was the only time in my career this happened to me. In the same time, someone else in my team made a mistake, hid it from management, even though we knew about it internally, fixed it and bragged about fixing the issue (no mention of him being the one who caused it) and got somehow (didn't even know we had such a thing) employee of the month and big praise in the next department meeting from management. My lesson from this? Screw these companies and the people running them. I was asked to help, I did it, I made a mistake, fully aware of that, but then I'm the only one thrown under the bus for it.
anoplus 6y ago
The bottom line is, the PR damage done by SolarWinds' CEO to his company stands out the most, and he can and should fix it.
- dehrmann 6y ago
  The CEO blaming an intern is the big thing that looks bad, but it's also very telling of their culture and how they got in this situation in the first place.
  - sitkack 6y ago
    The CEO saying this, they are inadvertently signaling that they want a certain class of customer, a customer who agrees with and would make similar statements. Any customer that can see through the BS will immediately turn around on their heels. And SolarWinds will be happy that they just lost that problem customer. SolarWinds is looking for an Equifax not a Netflix.
  - josho 6y ago
    What’s left unsaid is that not just the CEO, but every manager down the line has made it clear that it’s okay to toss the blame down the line. Every individual contributor in that company has just learned that they need to cover their ass for any action that could possibly go wrong. The cultural outcome is that accountabilities will be spread across managers so that blame can’t be assigned to an individual.
  - deleted 6y ago
    [deleted]
justapassenger 6y ago
At well managed big tech companies, stepping on a landmine like that is not only not detrimental to your career, but it can actually help you (unless you didn’t do that by mistake, but by being reckless and on purpose skipping all the protection layers). You can own your accident and, depending on the complexity, create whole teams to properly fix it, so no one else can cause an outrage like that.
- sverhagen 6y ago
  My colleague typically asks this question in interviews, and I've occasionally borrowed it from him: tell me about the last problem you caused and how it got resolved. It's one of those questions designed to just get them talking and there aren't too many wrong answers, except maybe, you know, not being able to think of anything.
jedberg 6y ago
I was on the other side of that Netflix outage, as the main contact point between Netflix and AWS at the time. Amazon did give us a detailed rundown of what happened, but was very specific to not name names, nor did we ask. We all agreed it was an excellent learning opportunity for both AWS and Netflix. That outage is what drove us to rearchitect all of Netflix to be multi-region.
vyhd 6y ago
I am deeply ambivalent about my tenure at Amazon overall, but I unconditionally applaud the level of psychological safety in their dev culture: there is a shared understanding that any system failure caused by a single person is necessarily a failure of the process that governs changes to that system. If there were any singular behavior I wish every tech company would adopt, it is that level of perspective on what makes a system reliable.
- mynameisash 6y ago
  My experience there was overall very negative. My comment above, though, was actually a very bright spot, and I want to call out the positives that exist. At the time, I had several years in the industry under my belt but was new to big tech companies, and especially the tech that Amazon had built. I lacked confidence in myself but I was super conscious of actively seeking out the help of others. And it failed spectacularly - one SDE3 in my team with whom I was sort of paired would quite literally back away slowly when I was showing him an issue I needed his advice on. Big companies like that have such variation in teams, and I think I just got rather unlucky in my experience.

kazen44 6y ago

> One of my best early-in-career lessons was that it takes maturity to own up to your mistakes (no matter how bone-headed), and it also takes good managers and a good company to foster an environment in which you can own up to them without fear of losing your job. Any company that wants to hang a weight around an intern's neck for something like this is not a company I would want to support in any way. owning up to your mistakes also gives you an incredible amount of credibility and respect in my opinion. Mistakes happen, especially in complex systems. Owning up to mistakes and explaining your reasoning about your actions makes you and your compatriots better engineers. Excluding malicious action, most people make a (semi) critical error sometime in their career, especially if you work on the ops side of things, these can often be disasterous. Engineering who claim they never have made a mistake that usually either not working on anything that has value or are just lucky in my opinion. engineering who are afraid to say they made a mistake are a cultural issue aswell, because it delays troubleshooting during incidents. Something we tell employees during our onboarding in a technical comes down to this. - reason about a problem by yourself first - think about impact before you do a change, if in doubt, ask and doublecheck. - admit mistakes when you realise them, explain your reasoning and why you did the action. - learn from your mistakes, but accept that being error-free is simply not possible.

ClumsyPilot 6y ago
"Engineering who claim they never have made a mistake that usually either not working on anything that has value or are just lucky" Third option: they don't realise when they make a mistake, either because they are not smart enough or too full of themselves.
- MikeDelta 6y ago
  Like the engineers who never have to refactor, because their code is good enough in the first go. The only way to guaranteed never have mistakes in your code, is to not have any code at all.
  - dustymcp 6y ago
    To be fair sometimes people hyper focus on refactoring rather than fixing business problems, which isnt really helpful either, have to find a decent middle ground.
    - MikeDelta 6y ago
      You are absolutely right, I forgot about the other extreme.
mns 6y ago
> owning up to your mistakes also gives you an incredible amount of credibility and respect in my opinion. Mistakes happen, especially in complex systems. Owning up to mistakes and explaining your reasoning about your actions makes you and your compatriots better engineers. If you are in the right company. I made a big mistake at one point. We had all of the people responsible for one of the payment methods away on holiday (this one payment method was managed by another team in another country, as it wasn't using our normal payment gateway and provider that my team was maintaining). Huge panic, someone needs to fix this, you can't use X any more for completing your order. I'm in the right team that's handling payments and fulfilment, only one at work at that point so I'm told to fix it. I do, I fix it, send it over to testing, get the green light, fix is deployed, everyone is happy. 2 hours later, we figure out the payments are working, but the orders are not being finalised and are still in unpaid. We realised that that single payment method done by this other team in another country was not using our standard payment processing workflow and it has a different way of actually getting the confirmation from the payment provider. This was quite a big company, we had around tens of thousands of Euros blocked in those 2 hours. I own up to it, I admin I made a mistake, go deeper (we did not have anything about this documented) fix it again, we unblock everything, all good. Until 1 month later I got fired (there were layoffs because of larger financial issues, but I was on the list because of the incident), it was the only time in my career this happened to me. In the same time, someone else in my team made a mistake, hid it from management, even though we knew about it internally, fixed it and bragged about fixing the issue (no mention of him being the one who caused it) and got somehow (didn't even know we had such a thing) employee of the month and big praise in the next department meeting from management. My lesson from this? Screw these companies and the people running them. I was asked to help, I did it, I made a mistake, fully aware of that, but then I'm the only one thrown under the bus for it.

anoplus 6y ago

The bottom line is, the PR damage done by SolarWinds' CEO to his company stands out the most, and he can and should fix it.

dehrmann 6y ago
The CEO blaming an intern is the big thing that looks bad, but it's also very telling of their culture and how they got in this situation in the first place.
- sitkack 6y ago
  The CEO saying this, they are inadvertently signaling that they want a certain class of customer, a customer who agrees with and would make similar statements. Any customer that can see through the BS will immediately turn around on their heels. And SolarWinds will be happy that they just lost that problem customer. SolarWinds is looking for an Equifax not a Netflix.
- josho 6y ago
  What’s left unsaid is that not just the CEO, but every manager down the line has made it clear that it’s okay to toss the blame down the line. Every individual contributor in that company has just learned that they need to cover their ass for any action that could possibly go wrong. The cultural outcome is that accountabilities will be spread across managers so that blame can’t be assigned to an individual.
- deleted 6y ago
  [deleted]

justapassenger 6y ago

At well managed big tech companies, stepping on a landmine like that is not only not detrimental to your career, but it can actually help you (unless you didn’t do that by mistake, but by being reckless and on purpose skipping all the protection layers). You can own your accident and, depending on the complexity, create whole teams to properly fix it, so no one else can cause an outrage like that.

sverhagen 6y ago
My colleague typically asks this question in interviews, and I've occasionally borrowed it from him: tell me about the last problem you caused and how it got resolved. It's one of those questions designed to just get them talking and there aren't too many wrong answers, except maybe, you know, not being able to think of anything.

jedberg 6y ago

I was on the other side of that Netflix outage, as the main contact point between Netflix and AWS at the time. Amazon did give us a detailed rundown of what happened, but was very specific to not name names, nor did we ask. We all agreed it was an excellent learning opportunity for both AWS and Netflix. That outage is what drove us to rearchitect all of Netflix to be multi-region.

vyhd 6y ago

I am deeply ambivalent about my tenure at Amazon overall, but I unconditionally applaud the level of psychological safety in their dev culture: there is a shared understanding that any system failure caused by a single person is necessarily a failure of the process that governs changes to that system. If there were any singular behavior I wish every tech company would adopt, it is that level of perspective on what makes a system reliable.

mynameisash 6y ago
My experience there was overall very negative. My comment above, though, was actually a very bright spot, and I want to call out the positives that exist. At the time, I had several years in the industry under my belt but was new to big tech companies, and especially the tech that Amazon had built. I lacked confidence in myself but I was super conscious of actively seeking out the help of others. And it failed spectacularly - one SDE3 in my team with whom I was sort of paired would quite literally back away slowly when I was showing him an issue I needed his advice on. Big companies like that have such variation in teams, and I think I just got rather unlucky in my experience.