55 ms·

Yeah so it's been mentioned in the comments already, but to everyone in Fastly right now: I feel for you. Something like this must be insanely stressful, and no

Yeah so it's been mentioned in the comments already, but to everyone in Fastly right now: I feel for you. Something like this must be insanely stressful, and not just during the outage. There will be (should be) a massive post-mortem. People will be losing sleep over this for days, weeks, months. :( Edit: There seems to be a major empathy outage in this thread. Disgusted but not surprised, unfortunately.

mothsonasloth 5y ago
Call me old fashioned but the latest trend of showing "empathy" for a serious incident, then proceeding to dance around the aftermath of it, whilst people give themselves a pat on back in a retro/post-mortem, isn't the way to do it. People need to be blamed, and responsibility for actions taken (without covering asses)
- q3k 5y ago
  The point isn't to dance around the incident, but to not blame people. You can blame systems, design, engineering culture, processes, but don't blame people. Even if someone accidentally pressed the 'destroy prod' button, that's not the fault of that person, it's the fault of that button existing and being accessible in the first place. I have no empathy for Fastly-the-company. I hate the fact that the Internet is centralized around CDNs. I wish this idea of 'but we _must_ run a CDN for our 1QPM blog!' would die in a fire. But I can still empathize with the Fastly engineers handling this shitstorm right now.
  - tyrex2017 5y ago
    I disagree. People implemented those systems, so if you are correct that it is the systems fault, then it is also a persons fault. People must be held accountable to have good incentives to reduce such outtages in the future. I do agree though that we should always be compassionate and realistic with other humans.
    - taurath 5y ago
      Loss of trust in a service provider and the afterwards loss of business is quite an incentive. Having someone drawn and quartered just provides an incentive to scapegoat.
    - q3k 5y ago
      > I disagree. People implemented those systems, so if you are correct that it is the systems fault, then it is also a persons fault. How do you make sure that mistakes don't happen, then? Do you blame and fire people who make mistakes, and hope that the next person put in the same spot doesn't make a mistake? Or do you figure out what caused that person to make the mistake and ensure there are processes in place so that next time this is less likely to happen? Extrinsic motivators like 'we will give you a bonus' or 'we will fire you' are surprisingly bad at getting people to not fuck things up.
      - colesantiago 5y ago
        This sort of culture worked at Netflix. Did they go down today?
        stuff4ben 5y ago
        Lets hope you don't ever go into management. You clearly have no idea how to motivate and retain people or have any insight on how hard it is to hire good people to begin with. And no, I'm pretty certain this is not how Netflix's culture is.
        colesantiago 5y ago
        > pretty certain this is not how Netflix's culture is. > pReTtY CeRtAiN This, the wording in of itself shows you have absolutely no clue whatsoever at all of Netflix's culture.
        stuff4ben 5y ago
        Riiiight... Anyways, you kept complaining of being downvoted, here's a clue: you're being an ass and no one likes you or what you have to say because you're wrong. So go scurry back to reddit where you belong troll...
  - atq2119 5y ago
    > don't blame people I feel like this requires some nuance. Don't blame an IC for introducing a bug or misconfiguration that led to the outage. Do consider blaming (and firing!) management if, during the postmortem, it turns out that it was in the way of fixing systemic problems. Ultimately, rule #1 should be: don't blame somebody unless malice or gross negligence is proven. Rule #2 should be the assumption that ICs will not have done either. Rule #3 is that sometimes, individual responsibility is required.
- gurgus 5y ago
  Blame culture isn't the way forward here. Do a post-mortem, work out root causes, work as a unit to ensure this doesn't happen again. Obviously if there are levels of gross negligence or misconduct discovered during post-mortem, that will need to be dealt with accordingly, but coming into this with an attitude of "we must find someone to blame and incur repercussions" isn't healthy at all. We are humans - don't forget that. edit: forgot some words.
  - thrdbndndn 5y ago
    I think what you said is exactly why people have different opinions on this topic: what counts as "gross negligence" and what doesn't? Different people draw lines at different places.
    - q3k 5y ago
      There's, to me, no obvious clear cut line. But here are some indicators that make me consider someone was being grossly negligent and/or even malicious: - ignoring warnings - acting against known-to-them best practices - repeating a previous mistake But, again, these are just indicators, not a checklist. Interestingly, any of these can happen also due to stress, burnout and generally broken company/team culture. Including something like a CYA culture where if they don't do something fast, they will be blamed for it, and thus they need to move fast and break things.
  - deleted 5y ago
    [deleted]
  - colesantiago 5y ago
    > Do a post-mortem, work out root causes, work as a unit to ensure this doesn't happen again. And if this happens again? They advertised they had failover and mitigations for this in the RAREST of cases: > Notices will be posted here when we re-route traffic, upgrade hardware, or in the extremely rare case our network isn’t serving traffic. - status.fastly.com The extremely rare case happened for an hour, which is a very long time in internet time.
- hardwaresofton 5y ago
  I hear you, but I just want to point out that this rarely happens anywhere else. It's great if tech (and people in general) hold themselves to progressively higher standards than what is out there already, but I don't think tech needs to be that much better, I'd settle for just doing a good honest retro (without throwing anyone under the bus, and without covering their asses) A good leader will take the hit (and the repercussions) for their underlings, compensate customers where compensation can make it better (and offer to make it easy to use fallbacks if this happens again) -- and internally fix the problem so it can't happen again, without throwing anyone to the dogs.
- OJFord 5y ago
  I don't agree about the blame, but I do also find the empathy cringeworthy. Something's broken; someone's job is to fix it; they'll fix it; it will work again. /shrug/ Post-mortems make far more interesting submissions IMO, but I suppose people up-vote 'yes down for me too'.
- darkcha0s 5y ago
  I'm sure you've never made a mistake. The best way (in a team), to tackle mistakes, is to ensure the process in place corrects these mistakes. The only way to do that, is a post-mortem/learning from the mistake. If you blame it on some engineer who did it, that guy will eventually be replaced by some other guy, who may make the same mistake.
  - fmajid 5y ago
    You also need to be proactive about other possible failure modes. Avoiding a culture of blame may or may not help. There needs to be a strong incentive for the organization to expend the resources to do so, and a mere "oops my bad" doesn't provide that without SLAs with teeth.
- berkes 5y ago
  We need to learn from our, and other mistakes, or else we keep repeating them. Nothing "old fashioned" about that. And we, especially companies, typically only learn if there is something at stake. Stock-price, a job, customers, liability etc. (Call me old fashioned, but what I learned from it, having no stake in the game, is we are truly demolishing the resilient, decentralised nature of the internet; or already have done so)
- hyper_reality 5y ago
  I recommend reading about "blameless postmortems" [1]. Our natural tendency is to look for who is responsible for an incident and point the finger of blame. Over time this leads to a cover-your-ass culture, whether you like it or not. Therefore such a tendency needs to be actively fought against to keep the focus on quality engineering and not politics. "An atmosphere of blame risks creating a culture in which incidents and issues are swept under the rug, leading to greater risk for the organization." [1] https://sre.google/sre-book/postmortem-culture/ https://sre.google/sre-book/postmortem-culture/
- taurath 5y ago
  Scapegoating in those situations happens more often than not. In an operations team all problems are systemic - having to do with decision makers throughout the process, sometimes acting on perverse incentives set up by others. Blame then gets diluted but still tends to fall upon the organization responsible rather than an individual, which is where it should be. Gross negligence is not so cut and dry.
- erwald 5y ago
  the attitude that "people need to be blamed" will never improve reliability in the long run. people come and go; systems and processes endure. blaming people is the best way to avoid making durable improvements to systems and processes.
  - notyourday 5y ago
    Doctors that make too many mistakes resulting in too high of payouts can't get individual malpractice insurance. Doctors that can't get individual malpractice insurance go to hospitals. Hospitals that hire too many doctors that make too many mistakes can't get hospital level policy. Hospital has to fire those doctors. That's how the system adjusts. We do not have a system that adjusts to "oops"
- mhandley 5y ago
  The problem is a blame culture ensures the near-misses are never reported. Air safety discovered this many years back - a no-blame culture ensures anything safety-related can be reported without fear of repercussions. This allows you to discover near misses due to human error and ensure that the overall system gains resilience over time. If you blame people for mistakes, they cover the non-obvious ones up, and so you cannot protect against similar ones in future, so your reliability/safety ends up much lower in the long run. It's all about evolving a system that is resilient to human error - we will make mistakes, but the system overall should catch them before they become catastrophies. In air travel now, the remaining errors almost never have a single simple cause, except in airlines/countries that don't have an effective safety reporting culture.
- austinjp 5y ago
  "Call me old-fashioned but..." is a dog-whistle harking back to "better days" that never existed. Emapthy and responsiblity are not mutually exclusive.
- colesantiago 5y ago
  > People need to be blamed, and responsibility for actions taken (without covering asses) This. When people talk about "HugOps", "empathy" and all that when a worldwide incident affecting a huge amount of time critical customers (e.g. trading, hft, cargo, food delivery, etc.) is happening for an hour, it has catastrophic consequences. I hope the engineers also understand the other side and why we are paying huge sums of cash for their service.
  - q3k 5y ago
    It's empathy towards people managing the incident, not towards the company. It's a sign of solidarity from SRE to SRE, not a sign of solidarity with a company.
    - colesantiago 5y ago
      oh right nice, and this somehow makes this all ok then? have a post mortem and all sunshine and rainbows. #noblameculture fucking hell, you guys have no idea of how this impacted other industries.
- sophacles 5y ago
  > People need to be blamed, and responsibility for actions taken (without covering asses) What i think this syntactically invalid sentence is trying to say is: People need to be blamed, and held responsible for actions taken. Why do people need to be blamed? Why do we need to make someone the scapegoat? What does being held responsible look like? Let say we find some sacrificial engineer to pin this on: * does the downtime magically disappear? * does the engineer suffering (say losing his job or whatever) make your downtime meaningful? You'll recoup your revenue somehow from it? * does the fact that there's a scapegoat mean that everyone else at fastly is perfect and it's ok to keep using them?
mnordhoff 5y ago
They shouldn't lose sleep over it, though.
- austinjp 5y ago
  Everyone's got responsibilities and aspirations. To be fair, I was thinking more of the jobbing engineer who's going to face anxiety about losing their job over this, but it extends to all levels. Having a fat bank balance helps get through periods without employment, but it's not just about money. There's anxiety, shame, embarrassment, the whole gamut. Going through a major incident at work is a shitty experience.
yvan 5y ago
Well, not much, I mean all our competitors are also using Fastly. I would be more worried if we were the only one using Fastly and everybody else was fine. But as we are all in the same boat, we lose the same :-)
- nebulous1 5y ago
  He's talking about Fastly themselves, not their customers
  - yvan 5y ago
    Arf, thanks for pointing at it, I misread. Sorry.
willejs 5y ago
#HugOps
strictfp 5y ago
Meh. Losing sleep sounds like an over-reaction. No system is foolproof. Of course Fastly should do what they can to prevent downtime, but it's still expected that they will go down. I would blame anyone who claimed otherwise or couldn't deal with it while not having a fallback.
- busymom0 5y ago
  I think the government websites being down (UK ones for example) are the bigger issue. Reddit/Stackoverflow etc being down isn't that big of a deal imo.
- kylen 5y ago
  I hear that you're suggesting that those involved shouldnt feel bad because its a systemic / just a job / etc. But the reality is that incidents like this can be very traumatic for those involved and thats not something they can control. If it was that simple to manage, depression and anxiety would not be a thing. Think its best to show a large amount of support and empathy for the individuals having a really bad day today, and how awful they may feel. Some will probably end up reading this thread (I know I would). And of course, still hold Fastly the business accountable for their response (but objectively, once we understand what the root cause was, and the long term solution).
  - paublyrne 5y ago
    I don't see how it's so traumatic for the engineers involved, unless the company culture in Fastly is really awful and there are punitive repercussions, or attempts to pin responsbility on individuals rather than systems, which I doubt. Many here have been responsible for web service outages albeit on much smaller scales, and in my experience it feels awful while it's happening but you quickly forget about it because so does everyone else.
    - Jare 5y ago
      > on much smaller scales > you quickly forget about it because so does everyone else This is definitely not the case here, and the experiences are bound to be very different.
H8crilA 5y ago
Imagine losing sleep over a corporate problem where you're just the next Joe Engineer, to be fired the second you're not needed. Have some perspective people.
- adkadskhj 5y ago
  I'm confused, why isn't being fired something to lose sleep over in your eyes? I get that you're implying that the job itself is not worth that much concern, but it seems you're ignoring that jobs bring in income, pay your mortgage, etc. If i lost my job tomorrow i'd be terrified.
  - ihumanable 5y ago
    People rarely get fired for outages. The comment you are replying to is saying that engineers shouldn’t stress out over an outage that only impacts a corporation. It’s a commentary on work / life balance and the all-too-common phenomenon of employees sacrificing for a company (in this case, feeling such personal stress that they would lose sleep) and contrasting it with the fact that most employers will fire you without a second thought if it’s what’s best for the business (they won’t lose any sleep). It’s a critique of the asymmetry that often exists and is frequently exploited by companies. This is often seen in statements like, “we are one big family so put in a few more hours for this launch” coupled with announcements like, “profit projections didn’t meet expectations so we are downsizing 5% of the work force.” You are family when they need you to work hard, and an expendable free market agent when your continued employment might risk hitting the quarterly goal. It is, of course, reasonable to lose sleep if you think your employment is in jeopardy. Very few companies, especially in the competitive SV market are firing engineers because of a single outage, even a bad one, because you just paid a bunch of money to train those engineers how to see this coming and fix it.
    - H8crilA 5y ago
      Yup, exactly, couldn't have written it better myself :)
jtdev 5y ago
Our fathers and mothers put man on the moon… we build shitty software that helps the technocrats sell more junk to the masses.
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
colesantiago 5y ago
Well, while engineers are getting paid $100K/yr to post #HugOps, I know someone in HFT and their dashboard uses the Fastly service, so this has had a huge impact on them for sure. Flag and downvote all you want, you know this is true.
- nsp 5y ago
  I suspect you'll have trouble convincing a forum of primarily engineers that a high frequency trader is more worthy of sympathy than an engineer. They're both pretty privileged jobs and HFT is not known for having tons of benefits to society
  - colesantiago 5y ago
    > I suspect you'll have trouble convincing a forum of primarily engineers that a high frequency trader is more worthy of sympathy than an engineer. At least HFT traders don't get paid to spy on their own customers with trackers littered everywhere, I find that very unethical that engineers get paid to even do that sort of thing, and every damn website has these trackers because engineers put them there. > They're both pretty privileged jobs and HFT is not known for having tons of benefits to society So HFT firms don't have their own foundations and grants to give to charities and organisations then?
  - notyourday 5y ago
    > I suspect you'll have trouble convincing a forum of primarily engineers that a high frequency trader is more worthy of sympathy than an engineer. Engineers are paid because their companies have customers. The it is pure madness that #hugops is the thing. I sincerely hope that Fastly's customers wack it $$ wise so hard that it actually affects #hugops engineering culture.
- Deathmax 5y ago
  And ignore the pre-agreed SLA targets and compensation for not meeting those targets that's in the contract they signed right? If you're going to say you're losing $X/minute of downtime, then either deal with it, architect around it, or negotiate the necessary SLA and compensation.
colesantiago 5y ago
Not my problem. Fastly should work as intended. The fault is theirs and they have said that they have failover, this worldwide outage caused by them just goes to show you that Fastly does not actually have a failover system in place. > "Fastly’s network has built-in redundancies and automatic failover routing to ensure optimal performance and uptime." - status.fastly.com Even their status page was down. Very embarrassing, Fastly did not work as advertised and mislead its customers. Edit: Offended flaggers circling around silencing misled Fastly customers. How pathetic.
- macintux 5y ago
  > this worldwide outage caused by them just goes to show you that Fastly does not actually have a failover system in place. I don’t know Fastly at all, but in my experience there’s no such thing as a foolproof failover system that covers all possible scenarios.
  - colesantiago 5y ago
    Even when they said this was a rare [0] case, they knew this case should be handled, but didn't handle it. > or in the extremely rare case our network isn’t serving traffic. reports also came in that this was a service configuration[1] issue, so not only there is no failover system, not even any validation automation was in place that could have prevented this. [0] https://status.fastly.com https://status.fastly.com [1] https://twitter.com/fastly/status/1402221348659814411 https://twitter.com/fastly/status/1402221348659814411
    - detaro 5y ago
      Systems failing is not evidence of systems not existing.
      - colesantiago 5y ago
        So why didn't the 'automatic failover' kick in during the outage? Where was it then? I don't see anything about 're-routing traffic' anywhere in the status page [0] [0] https://status.fastly.com/incidents/vpk0ssybt3bj https://status.fastly.com/incidents/vpk0ssybt3bj
southerntofu 5y ago
I feel for the Fastly workers, who managers are probably currently harassing to get things back online. I certainly don't feel any sympathy for Fastly administrators/managers who make business out of exploiting other people.
throwaway7747 5y ago
I have worked for one of their competitors (I'm not saying which) for quite a while. I've indirectly caused multiple outages that were maybe 1% this bad before, that didn't make the news only due to luck. Code that I owned (but did not write) was once a key cause of a severe outage that did make the news, and it would have been worse if I weren't coincidentally halfway through replacing that code with something more modern. I also had to do some very rapid work on internal failsafes around the time of the infamous Mirai botnet, to minimize service degradation in case it was pointed at us. It sucks. Working on CDN reliability is like working on wastewater management: the public forgets you exist until something breaks, when they start asking why you weren't doing your job. Fortunately, internal people at least seem to get it -- I hope this is the same as Fastly.
dm319 5y ago
Empathy is hard to find around here, maybe someone needs to study it. Is it a feature of people in tech? Don't remember much being on slashdot either.
- the_only_law 5y ago
  Just wait till they figure out how to make money off it.

mothsonasloth 5y ago

Call me old fashioned but the latest trend of showing "empathy" for a serious incident, then proceeding to dance around the aftermath of it, whilst people give themselves a pat on back in a retro/post-mortem, isn't the way to do it. People need to be blamed, and responsibility for actions taken (without covering asses)

q3k 5y ago
The point isn't to dance around the incident, but to not blame people. You can blame systems, design, engineering culture, processes, but don't blame people. Even if someone accidentally pressed the 'destroy prod' button, that's not the fault of that person, it's the fault of that button existing and being accessible in the first place. I have no empathy for Fastly-the-company. I hate the fact that the Internet is centralized around CDNs. I wish this idea of 'but we _must_ run a CDN for our 1QPM blog!' would die in a fire. But I can still empathize with the Fastly engineers handling this shitstorm right now.
- tyrex2017 5y ago
  I disagree. People implemented those systems, so if you are correct that it is the systems fault, then it is also a persons fault. People must be held accountable to have good incentives to reduce such outtages in the future. I do agree though that we should always be compassionate and realistic with other humans.
  - taurath 5y ago
    Loss of trust in a service provider and the afterwards loss of business is quite an incentive. Having someone drawn and quartered just provides an incentive to scapegoat.
  - q3k 5y ago
    > I disagree. People implemented those systems, so if you are correct that it is the systems fault, then it is also a persons fault. How do you make sure that mistakes don't happen, then? Do you blame and fire people who make mistakes, and hope that the next person put in the same spot doesn't make a mistake? Or do you figure out what caused that person to make the mistake and ensure there are processes in place so that next time this is less likely to happen? Extrinsic motivators like 'we will give you a bonus' or 'we will fire you' are surprisingly bad at getting people to not fuck things up.
    - colesantiago 5y ago
      This sort of culture worked at Netflix. Did they go down today?
      - stuff4ben 5y ago
        Lets hope you don't ever go into management. You clearly have no idea how to motivate and retain people or have any insight on how hard it is to hire good people to begin with. And no, I'm pretty certain this is not how Netflix's culture is.
        colesantiago 5y ago
        > pretty certain this is not how Netflix's culture is. > pReTtY CeRtAiN This, the wording in of itself shows you have absolutely no clue whatsoever at all of Netflix's culture.
        stuff4ben 5y ago
        Riiiight... Anyways, you kept complaining of being downvoted, here's a clue: you're being an ass and no one likes you or what you have to say because you're wrong. So go scurry back to reddit where you belong troll...
- atq2119 5y ago
  > don't blame people I feel like this requires some nuance. Don't blame an IC for introducing a bug or misconfiguration that led to the outage. Do consider blaming (and firing!) management if, during the postmortem, it turns out that it was in the way of fixing systemic problems. Ultimately, rule #1 should be: don't blame somebody unless malice or gross negligence is proven. Rule #2 should be the assumption that ICs will not have done either. Rule #3 is that sometimes, individual responsibility is required.
gurgus 5y ago
Blame culture isn't the way forward here. Do a post-mortem, work out root causes, work as a unit to ensure this doesn't happen again. Obviously if there are levels of gross negligence or misconduct discovered during post-mortem, that will need to be dealt with accordingly, but coming into this with an attitude of "we must find someone to blame and incur repercussions" isn't healthy at all. We are humans - don't forget that. edit: forgot some words.
- thrdbndndn 5y ago
  I think what you said is exactly why people have different opinions on this topic: what counts as "gross negligence" and what doesn't? Different people draw lines at different places.
  - q3k 5y ago
    There's, to me, no obvious clear cut line. But here are some indicators that make me consider someone was being grossly negligent and/or even malicious: - ignoring warnings - acting against known-to-them best practices - repeating a previous mistake But, again, these are just indicators, not a checklist. Interestingly, any of these can happen also due to stress, burnout and generally broken company/team culture. Including something like a CYA culture where if they don't do something fast, they will be blamed for it, and thus they need to move fast and break things.
- deleted 5y ago
  [deleted]
- colesantiago 5y ago
  > Do a post-mortem, work out root causes, work as a unit to ensure this doesn't happen again. And if this happens again? They advertised they had failover and mitigations for this in the RAREST of cases: > Notices will be posted here when we re-route traffic, upgrade hardware, or in the extremely rare case our network isn’t serving traffic. - status.fastly.com The extremely rare case happened for an hour, which is a very long time in internet time.
hardwaresofton 5y ago
I hear you, but I just want to point out that this rarely happens anywhere else. It's great if tech (and people in general) hold themselves to progressively higher standards than what is out there already, but I don't think tech needs to be that much better, I'd settle for just doing a good honest retro (without throwing anyone under the bus, and without covering their asses) A good leader will take the hit (and the repercussions) for their underlings, compensate customers where compensation can make it better (and offer to make it easy to use fallbacks if this happens again) -- and internally fix the problem so it can't happen again, without throwing anyone to the dogs.
OJFord 5y ago
I don't agree about the blame, but I do also find the empathy cringeworthy. Something's broken; someone's job is to fix it; they'll fix it; it will work again. /shrug/ Post-mortems make far more interesting submissions IMO, but I suppose people up-vote 'yes down for me too'.
darkcha0s 5y ago
I'm sure you've never made a mistake. The best way (in a team), to tackle mistakes, is to ensure the process in place corrects these mistakes. The only way to do that, is a post-mortem/learning from the mistake. If you blame it on some engineer who did it, that guy will eventually be replaced by some other guy, who may make the same mistake.
- fmajid 5y ago
  You also need to be proactive about other possible failure modes. Avoiding a culture of blame may or may not help. There needs to be a strong incentive for the organization to expend the resources to do so, and a mere "oops my bad" doesn't provide that without SLAs with teeth.
berkes 5y ago
We need to learn from our, and other mistakes, or else we keep repeating them. Nothing "old fashioned" about that. And we, especially companies, typically only learn if there is something at stake. Stock-price, a job, customers, liability etc. (Call me old fashioned, but what I learned from it, having no stake in the game, is we are truly demolishing the resilient, decentralised nature of the internet; or already have done so)
hyper_reality 5y ago
I recommend reading about "blameless postmortems" [1]. Our natural tendency is to look for who is responsible for an incident and point the finger of blame. Over time this leads to a cover-your-ass culture, whether you like it or not. Therefore such a tendency needs to be actively fought against to keep the focus on quality engineering and not politics. "An atmosphere of blame risks creating a culture in which incidents and issues are swept under the rug, leading to greater risk for the organization." [1] https://sre.google/sre-book/postmortem-culture/ https://sre.google/sre-book/postmortem-culture/
taurath 5y ago
Scapegoating in those situations happens more often than not. In an operations team all problems are systemic - having to do with decision makers throughout the process, sometimes acting on perverse incentives set up by others. Blame then gets diluted but still tends to fall upon the organization responsible rather than an individual, which is where it should be. Gross negligence is not so cut and dry.
erwald 5y ago
the attitude that "people need to be blamed" will never improve reliability in the long run. people come and go; systems and processes endure. blaming people is the best way to avoid making durable improvements to systems and processes.
- notyourday 5y ago
  Doctors that make too many mistakes resulting in too high of payouts can't get individual malpractice insurance. Doctors that can't get individual malpractice insurance go to hospitals. Hospitals that hire too many doctors that make too many mistakes can't get hospital level policy. Hospital has to fire those doctors. That's how the system adjusts. We do not have a system that adjusts to "oops"
mhandley 5y ago
The problem is a blame culture ensures the near-misses are never reported. Air safety discovered this many years back - a no-blame culture ensures anything safety-related can be reported without fear of repercussions. This allows you to discover near misses due to human error and ensure that the overall system gains resilience over time. If you blame people for mistakes, they cover the non-obvious ones up, and so you cannot protect against similar ones in future, so your reliability/safety ends up much lower in the long run. It's all about evolving a system that is resilient to human error - we will make mistakes, but the system overall should catch them before they become catastrophies. In air travel now, the remaining errors almost never have a single simple cause, except in airlines/countries that don't have an effective safety reporting culture.
austinjp 5y ago
"Call me old-fashioned but..." is a dog-whistle harking back to "better days" that never existed. Emapthy and responsiblity are not mutually exclusive.
colesantiago 5y ago
> People need to be blamed, and responsibility for actions taken (without covering asses) This. When people talk about "HugOps", "empathy" and all that when a worldwide incident affecting a huge amount of time critical customers (e.g. trading, hft, cargo, food delivery, etc.) is happening for an hour, it has catastrophic consequences. I hope the engineers also understand the other side and why we are paying huge sums of cash for their service.
- q3k 5y ago
  It's empathy towards people managing the incident, not towards the company. It's a sign of solidarity from SRE to SRE, not a sign of solidarity with a company.
  - colesantiago 5y ago
    oh right nice, and this somehow makes this all ok then? have a post mortem and all sunshine and rainbows. #noblameculture fucking hell, you guys have no idea of how this impacted other industries.
sophacles 5y ago
> People need to be blamed, and responsibility for actions taken (without covering asses) What i think this syntactically invalid sentence is trying to say is: People need to be blamed, and held responsible for actions taken. Why do people need to be blamed? Why do we need to make someone the scapegoat? What does being held responsible look like? Let say we find some sacrificial engineer to pin this on: * does the downtime magically disappear? * does the engineer suffering (say losing his job or whatever) make your downtime meaningful? You'll recoup your revenue somehow from it? * does the fact that there's a scapegoat mean that everyone else at fastly is perfect and it's ok to keep using them?

mnordhoff 5y ago

They shouldn't lose sleep over it, though.

austinjp 5y ago
Everyone's got responsibilities and aspirations. To be fair, I was thinking more of the jobbing engineer who's going to face anxiety about losing their job over this, but it extends to all levels. Having a fat bank balance helps get through periods without employment, but it's not just about money. There's anxiety, shame, embarrassment, the whole gamut. Going through a major incident at work is a shitty experience.

yvan 5y ago

Well, not much, I mean all our competitors are also using Fastly. I would be more worried if we were the only one using Fastly and everybody else was fine. But as we are all in the same boat, we lose the same :-)

nebulous1 5y ago
He's talking about Fastly themselves, not their customers
- yvan 5y ago
  Arf, thanks for pointing at it, I misread. Sorry.

willejs 5y ago

#HugOps

strictfp 5y ago

Meh. Losing sleep sounds like an over-reaction. No system is foolproof. Of course Fastly should do what they can to prevent downtime, but it's still expected that they will go down. I would blame anyone who claimed otherwise or couldn't deal with it while not having a fallback.

busymom0 5y ago
I think the government websites being down (UK ones for example) are the bigger issue. Reddit/Stackoverflow etc being down isn't that big of a deal imo.
kylen 5y ago
I hear that you're suggesting that those involved shouldnt feel bad because its a systemic / just a job / etc. But the reality is that incidents like this can be very traumatic for those involved and thats not something they can control. If it was that simple to manage, depression and anxiety would not be a thing. Think its best to show a large amount of support and empathy for the individuals having a really bad day today, and how awful they may feel. Some will probably end up reading this thread (I know I would). And of course, still hold Fastly the business accountable for their response (but objectively, once we understand what the root cause was, and the long term solution).
- paublyrne 5y ago
  I don't see how it's so traumatic for the engineers involved, unless the company culture in Fastly is really awful and there are punitive repercussions, or attempts to pin responsbility on individuals rather than systems, which I doubt. Many here have been responsible for web service outages albeit on much smaller scales, and in my experience it feels awful while it's happening but you quickly forget about it because so does everyone else.
  - Jare 5y ago
    > on much smaller scales > you quickly forget about it because so does everyone else This is definitely not the case here, and the experiences are bound to be very different.

H8crilA 5y ago

Imagine losing sleep over a corporate problem where you're just the next Joe Engineer, to be fired the second you're not needed. Have some perspective people.

adkadskhj 5y ago
I'm confused, why isn't being fired something to lose sleep over in your eyes? I get that you're implying that the job itself is not worth that much concern, but it seems you're ignoring that jobs bring in income, pay your mortgage, etc. If i lost my job tomorrow i'd be terrified.
- ihumanable 5y ago
  People rarely get fired for outages. The comment you are replying to is saying that engineers shouldn’t stress out over an outage that only impacts a corporation. It’s a commentary on work / life balance and the all-too-common phenomenon of employees sacrificing for a company (in this case, feeling such personal stress that they would lose sleep) and contrasting it with the fact that most employers will fire you without a second thought if it’s what’s best for the business (they won’t lose any sleep). It’s a critique of the asymmetry that often exists and is frequently exploited by companies. This is often seen in statements like, “we are one big family so put in a few more hours for this launch” coupled with announcements like, “profit projections didn’t meet expectations so we are downsizing 5% of the work force.” You are family when they need you to work hard, and an expendable free market agent when your continued employment might risk hitting the quarterly goal. It is, of course, reasonable to lose sleep if you think your employment is in jeopardy. Very few companies, especially in the competitive SV market are firing engineers because of a single outage, even a bad one, because you just paid a bunch of money to train those engineers how to see this coming and fix it.
  - H8crilA 5y ago
    Yup, exactly, couldn't have written it better myself :)

jtdev 5y ago

Our fathers and mothers put man on the moon… we build shitty software that helps the technocrats sell more junk to the masses.

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

colesantiago 5y ago

Well, while engineers are getting paid $100K/yr to post #HugOps, I know someone in HFT and their dashboard uses the Fastly service, so this has had a huge impact on them for sure. Flag and downvote all you want, you know this is true.

nsp 5y ago
I suspect you'll have trouble convincing a forum of primarily engineers that a high frequency trader is more worthy of sympathy than an engineer. They're both pretty privileged jobs and HFT is not known for having tons of benefits to society
- colesantiago 5y ago
  > I suspect you'll have trouble convincing a forum of primarily engineers that a high frequency trader is more worthy of sympathy than an engineer. At least HFT traders don't get paid to spy on their own customers with trackers littered everywhere, I find that very unethical that engineers get paid to even do that sort of thing, and every damn website has these trackers because engineers put them there. > They're both pretty privileged jobs and HFT is not known for having tons of benefits to society So HFT firms don't have their own foundations and grants to give to charities and organisations then?
- notyourday 5y ago
  > I suspect you'll have trouble convincing a forum of primarily engineers that a high frequency trader is more worthy of sympathy than an engineer. Engineers are paid because their companies have customers. The it is pure madness that #hugops is the thing. I sincerely hope that Fastly's customers wack it $$ wise so hard that it actually affects #hugops engineering culture.
Deathmax 5y ago
And ignore the pre-agreed SLA targets and compensation for not meeting those targets that's in the contract they signed right? If you're going to say you're losing $X/minute of downtime, then either deal with it, architect around it, or negotiate the necessary SLA and compensation.

colesantiago 5y ago

Not my problem. Fastly should work as intended. The fault is theirs and they have said that they have failover, this worldwide outage caused by them just goes to show you that Fastly does not actually have a failover system in place. > "Fastly’s network has built-in redundancies and automatic failover routing to ensure optimal performance and uptime." - status.fastly.com Even their status page was down. Very embarrassing, Fastly did not work as advertised and mislead its customers. Edit: Offended flaggers circling around silencing misled Fastly customers. How pathetic.

macintux 5y ago
> this worldwide outage caused by them just goes to show you that Fastly does not actually have a failover system in place. I don’t know Fastly at all, but in my experience there’s no such thing as a foolproof failover system that covers all possible scenarios.
- colesantiago 5y ago
  Even when they said this was a rare [0] case, they knew this case should be handled, but didn't handle it. > or in the extremely rare case our network isn’t serving traffic. reports also came in that this was a service configuration[1] issue, so not only there is no failover system, not even any validation automation was in place that could have prevented this. [0] https://status.fastly.com https://status.fastly.com [1] https://twitter.com/fastly/status/1402221348659814411 https://twitter.com/fastly/status/1402221348659814411
  - detaro 5y ago
    Systems failing is not evidence of systems not existing.
    - colesantiago 5y ago
      So why didn't the 'automatic failover' kick in during the outage? Where was it then? I don't see anything about 're-routing traffic' anywhere in the status page [0] [0] https://status.fastly.com/incidents/vpk0ssybt3bj https://status.fastly.com/incidents/vpk0ssybt3bj

southerntofu 5y ago

I feel for the Fastly workers, who managers are probably currently harassing to get things back online. I certainly don't feel any sympathy for Fastly administrators/managers who make business out of exploiting other people.

throwaway7747 5y ago

I have worked for one of their competitors (I'm not saying which) for quite a while. I've indirectly caused multiple outages that were maybe 1% this bad before, that didn't make the news only due to luck. Code that I owned (but did not write) was once a key cause of a severe outage that did make the news, and it would have been worse if I weren't coincidentally halfway through replacing that code with something more modern. I also had to do some very rapid work on internal failsafes around the time of the infamous Mirai botnet, to minimize service degradation in case it was pointed at us. It sucks. Working on CDN reliability is like working on wastewater management: the public forgets you exist until something breaks, when they start asking why you weren't doing your job. Fortunately, internal people at least seem to get it -- I hope this is the same as Fastly.

dm319 5y ago

Empathy is hard to find around here, maybe someone needs to study it. Is it a feature of people in tech? Don't remember much being on slashdot either.

the_only_law 5y ago
Just wait till they figure out how to make money off it.