7 ms·

Call me old fashioned but the latest trend of showing "empathy" for a serious incident, then proceeding to dance around the aftermath of it, whilst people give

Call me old fashioned but the latest trend of showing "empathy" for a serious incident, then proceeding to dance around the aftermath of it, whilst people give themselves a pat on back in a retro/post-mortem, isn't the way to do it. People need to be blamed, and responsibility for actions taken (without covering asses)

q3k 5y ago
The point isn't to dance around the incident, but to not blame people. You can blame systems, design, engineering culture, processes, but don't blame people. Even if someone accidentally pressed the 'destroy prod' button, that's not the fault of that person, it's the fault of that button existing and being accessible in the first place. I have no empathy for Fastly-the-company. I hate the fact that the Internet is centralized around CDNs. I wish this idea of 'but we _must_ run a CDN for our 1QPM blog!' would die in a fire. But I can still empathize with the Fastly engineers handling this shitstorm right now.
- tyrex2017 5y ago
  I disagree. People implemented those systems, so if you are correct that it is the systems fault, then it is also a persons fault. People must be held accountable to have good incentives to reduce such outtages in the future. I do agree though that we should always be compassionate and realistic with other humans.
  - taurath 5y ago
    Loss of trust in a service provider and the afterwards loss of business is quite an incentive. Having someone drawn and quartered just provides an incentive to scapegoat.
  - q3k 5y ago
    > I disagree. People implemented those systems, so if you are correct that it is the systems fault, then it is also a persons fault. How do you make sure that mistakes don't happen, then? Do you blame and fire people who make mistakes, and hope that the next person put in the same spot doesn't make a mistake? Or do you figure out what caused that person to make the mistake and ensure there are processes in place so that next time this is less likely to happen? Extrinsic motivators like 'we will give you a bonus' or 'we will fire you' are surprisingly bad at getting people to not fuck things up.
    - colesantiago 5y ago
      This sort of culture worked at Netflix. Did they go down today?
      - stuff4ben 5y ago
        Lets hope you don't ever go into management. You clearly have no idea how to motivate and retain people or have any insight on how hard it is to hire good people to begin with. And no, I'm pretty certain this is not how Netflix's culture is.
        colesantiago 5y ago
        > pretty certain this is not how Netflix's culture is. > pReTtY CeRtAiN This, the wording in of itself shows you have absolutely no clue whatsoever at all of Netflix's culture.
        stuff4ben 5y ago
        Riiiight... Anyways, you kept complaining of being downvoted, here's a clue: you're being an ass and no one likes you or what you have to say because you're wrong. So go scurry back to reddit where you belong troll...
        colesantiago 5y ago
        > you're being an ass and no one likes you or what you have to say because you're wrong. So go scurry back to reddit where you belong troll... Okay? some proof please? This is not far off from a baseless character attack which isn't really effective when trying to convince me about your point on you knowing about Netflix's culture. If you really want a proper answer, the truth is, unfortunately for you I am in management (previously was an engineer) and have always known Netflix to have a stellar performance oriented (and fear driven) culture, their playbook operates like a sports team. Not for everyone, but that's the point and it works for them. Maybe you should look inward to yourself if you're so vexed with me to call me silly names, that you can't handle the truth or the culture about why some companies like Netflix adopts this. Peace.
  - philjohn 5y ago
    That's a sure fire way to get a CYA culture, and it's a reason why the most successful tech firms don't do it.
  - altacc 5y ago
    v1. "It's Bob's fault and so we fired Bob." v2. "The issue was caused by a previously unidentified pathway that caused a feedback loop and overloaded our servers in a cascading fashion (or whatever). We have implemented a fix for this and updated our testing and deployment processes to stop similar cascades." Which solves the problem long term? As an architect making product choices, v2 wins every time. (With the caveat that if the cause was something that reveals a fundamental problem with the larger processes/professionalism/culture of the company, especially to do with security concerns, then I'm not buying that product and migrating away if we already use it.
    - richardwhiuk 5y ago
      If an employee does something actively malicious, you should absolute remove them. This is very rare though - incompetence /broken systems is much more likely. Otherwise you develop internal process that's entirely scar tissue, and only stops your teams doing their jobs.
      - altacc 5y ago
        I feel it is somewhat obvious and goes without saying that malicious action results in personal responsibility & repercussions. However I don't have any evidence or past experience that malicious action by an internal employee is a likely scenario for most outages. It may well occur but most examples I've heard of seem apocryphal. The scar tissue: this is where good choices come in because it's certainly not a rule that a change as a result of an incident review is an impediment to work. These definitely occur, and sometimes linger after the root cause is phased out. But best practices often reduce cognitive & process overheads. A rough example is that there are still people out there FTPing code to servers, having to manually select which files from a directory to upload. Replacing this error prone process with a deployment pipeline leads to a massive reduction in the likelihood of errors and will actually speed up the deployment process. It's all about making the right choices, not knee-jerk protections, and sometimes the choice is to leave things as they are.
    - tyrex2017 5y ago
      As I replied to a sibling comment, I never thought about firing Bob. I think we can assign responsibilities without being mean or denegrate someone. I am critizing myself all the time for stuff. No hurt feelings there.
  - aetimmes 5y ago
    > People must be held accountable to have good incentives to reduce such outtages in the future. Holding specific people "accountable" for outages doesn't incentivize reducing outages; it incentivizes not getting caught for having caused the outage. As a result, post-mortems turn into finger-pointing games instead of finding and resolving the root cause of the issue, which costs the company more money in the long run when a political scapegoat is found but the actual bug in the code is not.
- atq2119 5y ago
  > don't blame people I feel like this requires some nuance. Don't blame an IC for introducing a bug or misconfiguration that led to the outage. Do consider blaming (and firing!) management if, during the postmortem, it turns out that it was in the way of fixing systemic problems. Ultimately, rule #1 should be: don't blame somebody unless malice or gross negligence is proven. Rule #2 should be the assumption that ICs will not have done either. Rule #3 is that sometimes, individual responsibility is required.
gurgus 5y ago
Blame culture isn't the way forward here. Do a post-mortem, work out root causes, work as a unit to ensure this doesn't happen again. Obviously if there are levels of gross negligence or misconduct discovered during post-mortem, that will need to be dealt with accordingly, but coming into this with an attitude of "we must find someone to blame and incur repercussions" isn't healthy at all. We are humans - don't forget that. edit: forgot some words.
- thrdbndndn 5y ago
  I think what you said is exactly why people have different opinions on this topic: what counts as "gross negligence" and what doesn't? Different people draw lines at different places.
  - q3k 5y ago
    There's, to me, no obvious clear cut line. But here are some indicators that make me consider someone was being grossly negligent and/or even malicious: - ignoring warnings - acting against known-to-them best practices - repeating a previous mistake But, again, these are just indicators, not a checklist. Interestingly, any of these can happen also due to stress, burnout and generally broken company/team culture. Including something like a CYA culture where if they don't do something fast, they will be blamed for it, and thus they need to move fast and break things.
- deleted 5y ago
  [deleted]
- colesantiago 5y ago
  > Do a post-mortem, work out root causes, work as a unit to ensure this doesn't happen again. And if this happens again? They advertised they had failover and mitigations for this in the RAREST of cases: > Notices will be posted here when we re-route traffic, upgrade hardware, or in the extremely rare case our network isn’t serving traffic. - status.fastly.com The extremely rare case happened for an hour, which is a very long time in internet time.
hardwaresofton 5y ago
I hear you, but I just want to point out that this rarely happens anywhere else. It's great if tech (and people in general) hold themselves to progressively higher standards than what is out there already, but I don't think tech needs to be that much better, I'd settle for just doing a good honest retro (without throwing anyone under the bus, and without covering their asses) A good leader will take the hit (and the repercussions) for their underlings, compensate customers where compensation can make it better (and offer to make it easy to use fallbacks if this happens again) -- and internally fix the problem so it can't happen again, without throwing anyone to the dogs.
OJFord 5y ago
I don't agree about the blame, but I do also find the empathy cringeworthy. Something's broken; someone's job is to fix it; they'll fix it; it will work again. /shrug/ Post-mortems make far more interesting submissions IMO, but I suppose people up-vote 'yes down for me too'.
darkcha0s 5y ago
I'm sure you've never made a mistake. The best way (in a team), to tackle mistakes, is to ensure the process in place corrects these mistakes. The only way to do that, is a post-mortem/learning from the mistake. If you blame it on some engineer who did it, that guy will eventually be replaced by some other guy, who may make the same mistake.
- fmajid 5y ago
  You also need to be proactive about other possible failure modes. Avoiding a culture of blame may or may not help. There needs to be a strong incentive for the organization to expend the resources to do so, and a mere "oops my bad" doesn't provide that without SLAs with teeth.
berkes 5y ago
We need to learn from our, and other mistakes, or else we keep repeating them. Nothing "old fashioned" about that. And we, especially companies, typically only learn if there is something at stake. Stock-price, a job, customers, liability etc. (Call me old fashioned, but what I learned from it, having no stake in the game, is we are truly demolishing the resilient, decentralised nature of the internet; or already have done so)
hyper_reality 5y ago
I recommend reading about "blameless postmortems" [1]. Our natural tendency is to look for who is responsible for an incident and point the finger of blame. Over time this leads to a cover-your-ass culture, whether you like it or not. Therefore such a tendency needs to be actively fought against to keep the focus on quality engineering and not politics. "An atmosphere of blame risks creating a culture in which incidents and issues are swept under the rug, leading to greater risk for the organization." [1] https://sre.google/sre-book/postmortem-culture/ https://sre.google/sre-book/postmortem-culture/
taurath 5y ago
Scapegoating in those situations happens more often than not. In an operations team all problems are systemic - having to do with decision makers throughout the process, sometimes acting on perverse incentives set up by others. Blame then gets diluted but still tends to fall upon the organization responsible rather than an individual, which is where it should be. Gross negligence is not so cut and dry.
erwald 5y ago
the attitude that "people need to be blamed" will never improve reliability in the long run. people come and go; systems and processes endure. blaming people is the best way to avoid making durable improvements to systems and processes.
- notyourday 5y ago
  Doctors that make too many mistakes resulting in too high of payouts can't get individual malpractice insurance. Doctors that can't get individual malpractice insurance go to hospitals. Hospitals that hire too many doctors that make too many mistakes can't get hospital level policy. Hospital has to fire those doctors. That's how the system adjusts. We do not have a system that adjusts to "oops"
mhandley 5y ago
The problem is a blame culture ensures the near-misses are never reported. Air safety discovered this many years back - a no-blame culture ensures anything safety-related can be reported without fear of repercussions. This allows you to discover near misses due to human error and ensure that the overall system gains resilience over time. If you blame people for mistakes, they cover the non-obvious ones up, and so you cannot protect against similar ones in future, so your reliability/safety ends up much lower in the long run. It's all about evolving a system that is resilient to human error - we will make mistakes, but the system overall should catch them before they become catastrophies. In air travel now, the remaining errors almost never have a single simple cause, except in airlines/countries that don't have an effective safety reporting culture.
austinjp 5y ago
"Call me old-fashioned but..." is a dog-whistle harking back to "better days" that never existed. Emapthy and responsiblity are not mutually exclusive.
colesantiago 5y ago
> People need to be blamed, and responsibility for actions taken (without covering asses) This. When people talk about "HugOps", "empathy" and all that when a worldwide incident affecting a huge amount of time critical customers (e.g. trading, hft, cargo, food delivery, etc.) is happening for an hour, it has catastrophic consequences. I hope the engineers also understand the other side and why we are paying huge sums of cash for their service.
- q3k 5y ago
  It's empathy towards people managing the incident, not towards the company. It's a sign of solidarity from SRE to SRE, not a sign of solidarity with a company.
  - colesantiago 5y ago
    oh right nice, and this somehow makes this all ok then? have a post mortem and all sunshine and rainbows. #noblameculture fucking hell, you guys have no idea of how this impacted other industries.
sophacles 5y ago
> People need to be blamed, and responsibility for actions taken (without covering asses) What i think this syntactically invalid sentence is trying to say is: People need to be blamed, and held responsible for actions taken. Why do people need to be blamed? Why do we need to make someone the scapegoat? What does being held responsible look like? Let say we find some sacrificial engineer to pin this on: * does the downtime magically disappear? * does the engineer suffering (say losing his job or whatever) make your downtime meaningful? You'll recoup your revenue somehow from it? * does the fact that there's a scapegoat mean that everyone else at fastly is perfect and it's ok to keep using them?

q3k 5y ago

The point isn't to dance around the incident, but to not blame people. You can blame systems, design, engineering culture, processes, but don't blame people. Even if someone accidentally pressed the 'destroy prod' button, that's not the fault of that person, it's the fault of that button existing and being accessible in the first place. I have no empathy for Fastly-the-company. I hate the fact that the Internet is centralized around CDNs. I wish this idea of 'but we _must_ run a CDN for our 1QPM blog!' would die in a fire. But I can still empathize with the Fastly engineers handling this shitstorm right now.

tyrex2017 5y ago
I disagree. People implemented those systems, so if you are correct that it is the systems fault, then it is also a persons fault. People must be held accountable to have good incentives to reduce such outtages in the future. I do agree though that we should always be compassionate and realistic with other humans.
- taurath 5y ago
  Loss of trust in a service provider and the afterwards loss of business is quite an incentive. Having someone drawn and quartered just provides an incentive to scapegoat.
- q3k 5y ago
  > I disagree. People implemented those systems, so if you are correct that it is the systems fault, then it is also a persons fault. How do you make sure that mistakes don't happen, then? Do you blame and fire people who make mistakes, and hope that the next person put in the same spot doesn't make a mistake? Or do you figure out what caused that person to make the mistake and ensure there are processes in place so that next time this is less likely to happen? Extrinsic motivators like 'we will give you a bonus' or 'we will fire you' are surprisingly bad at getting people to not fuck things up.
  - colesantiago 5y ago
    This sort of culture worked at Netflix. Did they go down today?
    - stuff4ben 5y ago
      Lets hope you don't ever go into management. You clearly have no idea how to motivate and retain people or have any insight on how hard it is to hire good people to begin with. And no, I'm pretty certain this is not how Netflix's culture is.
      - colesantiago 5y ago
        > pretty certain this is not how Netflix's culture is. > pReTtY CeRtAiN This, the wording in of itself shows you have absolutely no clue whatsoever at all of Netflix's culture.
        stuff4ben 5y ago
        Riiiight... Anyways, you kept complaining of being downvoted, here's a clue: you're being an ass and no one likes you or what you have to say because you're wrong. So go scurry back to reddit where you belong troll...
        colesantiago 5y ago
        > you're being an ass and no one likes you or what you have to say because you're wrong. So go scurry back to reddit where you belong troll... Okay? some proof please? This is not far off from a baseless character attack which isn't really effective when trying to convince me about your point on you knowing about Netflix's culture. If you really want a proper answer, the truth is, unfortunately for you I am in management (previously was an engineer) and have always known Netflix to have a stellar performance oriented (and fear driven) culture, their playbook operates like a sports team. Not for everyone, but that's the point and it works for them. Maybe you should look inward to yourself if you're so vexed with me to call me silly names, that you can't handle the truth or the culture about why some companies like Netflix adopts this. Peace.
- philjohn 5y ago
  That's a sure fire way to get a CYA culture, and it's a reason why the most successful tech firms don't do it.
- altacc 5y ago
  v1. "It's Bob's fault and so we fired Bob." v2. "The issue was caused by a previously unidentified pathway that caused a feedback loop and overloaded our servers in a cascading fashion (or whatever). We have implemented a fix for this and updated our testing and deployment processes to stop similar cascades." Which solves the problem long term? As an architect making product choices, v2 wins every time. (With the caveat that if the cause was something that reveals a fundamental problem with the larger processes/professionalism/culture of the company, especially to do with security concerns, then I'm not buying that product and migrating away if we already use it.
  - richardwhiuk 5y ago
    If an employee does something actively malicious, you should absolute remove them. This is very rare though - incompetence /broken systems is much more likely. Otherwise you develop internal process that's entirely scar tissue, and only stops your teams doing their jobs.
    - altacc 5y ago
      I feel it is somewhat obvious and goes without saying that malicious action results in personal responsibility & repercussions. However I don't have any evidence or past experience that malicious action by an internal employee is a likely scenario for most outages. It may well occur but most examples I've heard of seem apocryphal. The scar tissue: this is where good choices come in because it's certainly not a rule that a change as a result of an incident review is an impediment to work. These definitely occur, and sometimes linger after the root cause is phased out. But best practices often reduce cognitive & process overheads. A rough example is that there are still people out there FTPing code to servers, having to manually select which files from a directory to upload. Replacing this error prone process with a deployment pipeline leads to a massive reduction in the likelihood of errors and will actually speed up the deployment process. It's all about making the right choices, not knee-jerk protections, and sometimes the choice is to leave things as they are.
  - tyrex2017 5y ago
    As I replied to a sibling comment, I never thought about firing Bob. I think we can assign responsibilities without being mean or denegrate someone. I am critizing myself all the time for stuff. No hurt feelings there.
- aetimmes 5y ago
  > People must be held accountable to have good incentives to reduce such outtages in the future. Holding specific people "accountable" for outages doesn't incentivize reducing outages; it incentivizes not getting caught for having caused the outage. As a result, post-mortems turn into finger-pointing games instead of finding and resolving the root cause of the issue, which costs the company more money in the long run when a political scapegoat is found but the actual bug in the code is not.
atq2119 5y ago
> don't blame people I feel like this requires some nuance. Don't blame an IC for introducing a bug or misconfiguration that led to the outage. Do consider blaming (and firing!) management if, during the postmortem, it turns out that it was in the way of fixing systemic problems. Ultimately, rule #1 should be: don't blame somebody unless malice or gross negligence is proven. Rule #2 should be the assumption that ICs will not have done either. Rule #3 is that sometimes, individual responsibility is required.

gurgus 5y ago

Blame culture isn't the way forward here. Do a post-mortem, work out root causes, work as a unit to ensure this doesn't happen again. Obviously if there are levels of gross negligence or misconduct discovered during post-mortem, that will need to be dealt with accordingly, but coming into this with an attitude of "we must find someone to blame and incur repercussions" isn't healthy at all. We are humans - don't forget that. edit: forgot some words.

thrdbndndn 5y ago
I think what you said is exactly why people have different opinions on this topic: what counts as "gross negligence" and what doesn't? Different people draw lines at different places.
- q3k 5y ago
  There's, to me, no obvious clear cut line. But here are some indicators that make me consider someone was being grossly negligent and/or even malicious: - ignoring warnings - acting against known-to-them best practices - repeating a previous mistake But, again, these are just indicators, not a checklist. Interestingly, any of these can happen also due to stress, burnout and generally broken company/team culture. Including something like a CYA culture where if they don't do something fast, they will be blamed for it, and thus they need to move fast and break things.
deleted 5y ago
[deleted]
colesantiago 5y ago
> Do a post-mortem, work out root causes, work as a unit to ensure this doesn't happen again. And if this happens again? They advertised they had failover and mitigations for this in the RAREST of cases: > Notices will be posted here when we re-route traffic, upgrade hardware, or in the extremely rare case our network isn’t serving traffic. - status.fastly.com The extremely rare case happened for an hour, which is a very long time in internet time.

hardwaresofton 5y ago

I hear you, but I just want to point out that this rarely happens anywhere else. It's great if tech (and people in general) hold themselves to progressively higher standards than what is out there already, but I don't think tech needs to be that much better, I'd settle for just doing a good honest retro (without throwing anyone under the bus, and without covering their asses) A good leader will take the hit (and the repercussions) for their underlings, compensate customers where compensation can make it better (and offer to make it easy to use fallbacks if this happens again) -- and internally fix the problem so it can't happen again, without throwing anyone to the dogs.

OJFord 5y ago

I don't agree about the blame, but I do also find the empathy cringeworthy. Something's broken; someone's job is to fix it; they'll fix it; it will work again. /shrug/ Post-mortems make far more interesting submissions IMO, but I suppose people up-vote 'yes down for me too'.

darkcha0s 5y ago

I'm sure you've never made a mistake. The best way (in a team), to tackle mistakes, is to ensure the process in place corrects these mistakes. The only way to do that, is a post-mortem/learning from the mistake. If you blame it on some engineer who did it, that guy will eventually be replaced by some other guy, who may make the same mistake.

fmajid 5y ago
You also need to be proactive about other possible failure modes. Avoiding a culture of blame may or may not help. There needs to be a strong incentive for the organization to expend the resources to do so, and a mere "oops my bad" doesn't provide that without SLAs with teeth.

berkes 5y ago

We need to learn from our, and other mistakes, or else we keep repeating them. Nothing "old fashioned" about that. And we, especially companies, typically only learn if there is something at stake. Stock-price, a job, customers, liability etc. (Call me old fashioned, but what I learned from it, having no stake in the game, is we are truly demolishing the resilient, decentralised nature of the internet; or already have done so)

hyper_reality 5y ago

I recommend reading about "blameless postmortems" [1]. Our natural tendency is to look for who is responsible for an incident and point the finger of blame. Over time this leads to a cover-your-ass culture, whether you like it or not. Therefore such a tendency needs to be actively fought against to keep the focus on quality engineering and not politics. "An atmosphere of blame risks creating a culture in which incidents and issues are swept under the rug, leading to greater risk for the organization." [1] https://sre.google/sre-book/postmortem-culture/ https://sre.google/sre-book/postmortem-culture/

taurath 5y ago

Scapegoating in those situations happens more often than not. In an operations team all problems are systemic - having to do with decision makers throughout the process, sometimes acting on perverse incentives set up by others. Blame then gets diluted but still tends to fall upon the organization responsible rather than an individual, which is where it should be. Gross negligence is not so cut and dry.

erwald 5y ago

the attitude that "people need to be blamed" will never improve reliability in the long run. people come and go; systems and processes endure. blaming people is the best way to avoid making durable improvements to systems and processes.

notyourday 5y ago
Doctors that make too many mistakes resulting in too high of payouts can't get individual malpractice insurance. Doctors that can't get individual malpractice insurance go to hospitals. Hospitals that hire too many doctors that make too many mistakes can't get hospital level policy. Hospital has to fire those doctors. That's how the system adjusts. We do not have a system that adjusts to "oops"

mhandley 5y ago

The problem is a blame culture ensures the near-misses are never reported. Air safety discovered this many years back - a no-blame culture ensures anything safety-related can be reported without fear of repercussions. This allows you to discover near misses due to human error and ensure that the overall system gains resilience over time. If you blame people for mistakes, they cover the non-obvious ones up, and so you cannot protect against similar ones in future, so your reliability/safety ends up much lower in the long run. It's all about evolving a system that is resilient to human error - we will make mistakes, but the system overall should catch them before they become catastrophies. In air travel now, the remaining errors almost never have a single simple cause, except in airlines/countries that don't have an effective safety reporting culture.

austinjp 5y ago

"Call me old-fashioned but..." is a dog-whistle harking back to "better days" that never existed. Emapthy and responsiblity are not mutually exclusive.

colesantiago 5y ago

> People need to be blamed, and responsibility for actions taken (without covering asses) This. When people talk about "HugOps", "empathy" and all that when a worldwide incident affecting a huge amount of time critical customers (e.g. trading, hft, cargo, food delivery, etc.) is happening for an hour, it has catastrophic consequences. I hope the engineers also understand the other side and why we are paying huge sums of cash for their service.

q3k 5y ago
It's empathy towards people managing the incident, not towards the company. It's a sign of solidarity from SRE to SRE, not a sign of solidarity with a company.
- colesantiago 5y ago
  oh right nice, and this somehow makes this all ok then? have a post mortem and all sunshine and rainbows. #noblameculture fucking hell, you guys have no idea of how this impacted other industries.

sophacles 5y ago

> People need to be blamed, and responsibility for actions taken (without covering asses) What i think this syntactically invalid sentence is trying to say is: People need to be blamed, and held responsible for actions taken. Why do people need to be blamed? Why do we need to make someone the scapegoat? What does being held responsible look like? Let say we find some sacrificial engineer to pin this on: * does the downtime magically disappear? * does the engineer suffering (say losing his job or whatever) make your downtime meaningful? You'll recoup your revenue somehow from it? * does the fact that there's a scapegoat mean that everyone else at fastly is perfect and it's ok to keep using them?