7 ms·

Google had a comparable outage several years ago. https://status.cloud.google.com/incident/cloud-networking/19009 https://status.cloud.google.com/incident/clou

by vitus 5y ago

Google had a comparable outage several years ago. https://status.cloud.google.com/incident/cloud-networking/19009 https://status.cloud.google.com/incident/cloud-networking/19... This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust). I'd be curious to see what systemic changes happen at FB as a result, if any.

silicon2401 5y ago
> leadership read out emails from customers telling us how we let them down and lost their trust). That's amazing. I would never have expected my feedback to a company to actually be read, let alone taken seriously. Hopefully more companies do this than I thought.
- spaetzleesser 5y ago
  From my experience this is more done to make leadership feel better and deflect blame from their leadership.
- variant 5y ago
  I just read your comment out loud if it helps.
hoppyhoppy2 5y ago
Facebook also had a nearly 24-hour outage in 2019. https://www.nytimes.com/2019/03/14/technology/facebook-whatsapp-outage.html https://www.nytimes.com/2019/03/14/technology/facebook-whats... (or http://archive.today/O7ycB http://archive.today/O7ycB )
vitus 5y ago
To expand on why this made me think of the Google outage: It was a global backbone isolation, caused by configuration changes (as they all are...). It was detected fairly early on, but recovery was difficult because internal tools / debugging workflows were also impacted, and even after the problem was identified, it still took time to back out the change. "But wait, a global backbone isolation? Google wasn't totally down," you might say. That's because Google has two (primary) backbones (B2 and B4), and only B4 was isolated, so traffic spilled over onto B2 (which has much less capacity), causing heavy congestion.
- tantalor 5y ago
  But the FB outage was not a configuration change. > a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network
  - vitus 5y ago
    From yesterday's post: "Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. ... Our services are now back online and we’re actively working to fully return them to regular operations. We want to make clear that there was no malicious activity behind this outage — its root cause was a faulty configuration change on our end." Ultimately, that faulty command changed router configuration globally. The Google outage was triggered by a configuration change due to an automation system gone rogue. But hey, it too was triggered by a human issuing a command at some point.
    - quietbritishjim 5y ago
      I'm inclined to believe the later post as they've had more time to assess the details. I think the point of the earlier post is really to say "we weren't hacked!" but they didn't want to use exactly that language.
  - narrator 5y ago
    This is kind of like Chernobyl where they were testing to see how hot they could run the reactor to see how much power it could generate. Then things went sideways.
    - formerly_proven 5y ago
      The Chernobyl test was not a test to drive the reactor to the limits, but actually a test to verify that the inertia of the main turbines is big enough to drive the coolant pumps for X amount of time in the case of grid failure.
cratermoon 5y ago
> I'd be curious to see what systemic changes happen at FB as a result, if any. If history is any guide, Facebook will decide some division charged with preventing problems was an ineffective waste of money, shut it down, and fire a bunch of people.
AdamHominem 5y ago
> This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust). Bullshit. I'd believe this if it was not completely impossible for 99.999999% of google "customers" to contact anyone at the company. Or for the decade and a half of personal and professional observations of people getting fucked over by google and having absolutely nobody they could contact to try and resolve the situation. You googlers can't even disdain yourselves to talk to other workers at the company who are in a caste lower than you. The fundamental problem googlers have is that they all think they're so smart/good at what they do, it just doesn't seem to occur that they could have possibly screwed something up, or something could go wrong or break, or someone might need help in a way your help page authors didn't anticipate...and people might need to get ahold of an actual human to say "shit's broke, yo." Or worse, none of you give a shit. The company certainly doesn't. When you've got near monopoly and have your fingers in every single aspect of the internet, you don't need to care about fucking your customers over. I cannot legitimately name a single google product that I, or anyone I know, likes or wants to use. We just don't have a choice because of your market dominance.
- ro_bit 5y ago
  > You googlers can't even disdain yourselves to talk to other workers at the company who are in a caste lower than you. We must know different googlers then. It's good to avoid painting a group with the same brush
- colechristensen 5y ago
  Maps, Mail, Drive, Scholar, and Search are all the best or near the best available. That doesn’t mean I like every one of them or I wouldn’t prefer others, but as far as I can tell the competition doesn’t exist that works better. GCP and Pixel phones are a toss-up between them and competitors. It isn’t market dominance, nobody has made anything better.
  - astrange 5y ago
    Search is famously kind of bad the last few years, but even Maps isn’t that great. (Data errors I’ve seen this week: the aerial imagery over Brisbane Australia is from ~2010 but labeled 2021, the coastline near Barentsburg in Svalbard is wrong and doesn’t match any other map.)
agwa 5y ago
> a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust Is that normal at Google? Making people feel bad for an outage doesn't seem consistent with the "blameless postmortem" culture promoted in the SRE book[1]. [1] https://sre.google/sre-book/postmortem-culture/ https://sre.google/sre-book/postmortem-culture/
- fires10 5y ago
  I don't think Google really cares about listening to their users. I have spent more than 6 hours trying to get simple warranty issues resolved. I wish they had to feel the pain of their actions and decisions.
- UncleMeat 5y ago
  "Blameless Postmortem" does not mean "No Consequences", even if people often want to interpret it that way. If an organization determines that a disconnect between ground work and a customer's experience is a contributing factor to poor decision making then they might conclude that making engineers more emotionally invested in their customers could be a viable path forward.
  - bob1029 5y ago
    Relentless customer service is never going to screw you over in my experience... It pains me that we have to constantly play these games of abstraction between engineer and customer. You are presumably working a job which involves some business and some customer. It is not a fucking daycare. If any of my customers are pissed about their experience, I want to be on the phone with them as soon as humanly possible and I want to hear it myself. Yes, it is a dreadful experience to get bitched at, but it also sharpens your focus like you wouldn't believe when you can't just throw a problem to the guy behind you. By all means, put the support/enhancement requests through a separate channel+buffer so everyone can actually get work done during the day. But, at no point should an engineer ever be allowed to feel like they don't have to answer to some customer. If you are terrified a junior dev is going to say a naughty phrase to a VIP, then invent an internal customer for them to answer to, and diligently proxy the end customer's sentiment for the engineer's benefit.
spaetzleesser 5y ago
“ mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust “ The same leadership that demanded tighter and tighter deadlines and discouraged thinking things through?
grumple 5y ago
The most remarkable thing about this is learning that anyone at Google read an email from a customer. Given the automated responses to complaints of account shutdowns, or complaints about app store rejections, etc, this is pretty surprising.
- lvs 5y ago
  I'd love to get a read receipt each time someone at Google has actually read my feedback. Then it might be possible to determine whether I'm just shaking my fists at the heavens or not.

by vitus 5y ago

silicon2401 5y ago

> leadership read out emails from customers telling us how we let them down and lost their trust). That's amazing. I would never have expected my feedback to a company to actually be read, let alone taken seriously. Hopefully more companies do this than I thought.

spaetzleesser 5y ago
From my experience this is more done to make leadership feel better and deflect blame from their leadership.
variant 5y ago
I just read your comment out loud if it helps.

hoppyhoppy2 5y ago

Facebook also had a nearly 24-hour outage in 2019. https://www.nytimes.com/2019/03/14/technology/facebook-whatsapp-outage.html https://www.nytimes.com/2019/03/14/technology/facebook-whats... (or http://archive.today/O7ycB http://archive.today/O7ycB )

vitus 5y ago

To expand on why this made me think of the Google outage: It was a global backbone isolation, caused by configuration changes (as they all are...). It was detected fairly early on, but recovery was difficult because internal tools / debugging workflows were also impacted, and even after the problem was identified, it still took time to back out the change. "But wait, a global backbone isolation? Google wasn't totally down," you might say. That's because Google has two (primary) backbones (B2 and B4), and only B4 was isolated, so traffic spilled over onto B2 (which has much less capacity), causing heavy congestion.

tantalor 5y ago
But the FB outage was not a configuration change. > a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network
- vitus 5y ago
  From yesterday's post: "Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. ... Our services are now back online and we’re actively working to fully return them to regular operations. We want to make clear that there was no malicious activity behind this outage — its root cause was a faulty configuration change on our end." Ultimately, that faulty command changed router configuration globally. The Google outage was triggered by a configuration change due to an automation system gone rogue. But hey, it too was triggered by a human issuing a command at some point.
  - quietbritishjim 5y ago
    I'm inclined to believe the later post as they've had more time to assess the details. I think the point of the earlier post is really to say "we weren't hacked!" but they didn't want to use exactly that language.
- narrator 5y ago
  This is kind of like Chernobyl where they were testing to see how hot they could run the reactor to see how much power it could generate. Then things went sideways.
  - formerly_proven 5y ago
    The Chernobyl test was not a test to drive the reactor to the limits, but actually a test to verify that the inertia of the main turbines is big enough to drive the coolant pumps for X amount of time in the case of grid failure.

cratermoon 5y ago

> I'd be curious to see what systemic changes happen at FB as a result, if any. If history is any guide, Facebook will decide some division charged with preventing problems was an ineffective waste of money, shut it down, and fire a bunch of people.

AdamHominem 5y ago

> This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust). Bullshit. I'd believe this if it was not completely impossible for 99.999999% of google "customers" to contact anyone at the company. Or for the decade and a half of personal and professional observations of people getting fucked over by google and having absolutely nobody they could contact to try and resolve the situation. You googlers can't even disdain yourselves to talk to other workers at the company who are in a caste lower than you. The fundamental problem googlers have is that they all think they're so smart/good at what they do, it just doesn't seem to occur that they could have possibly screwed something up, or something could go wrong or break, or someone might need help in a way your help page authors didn't anticipate...and people might need to get ahold of an actual human to say "shit's broke, yo." Or worse, none of you give a shit. The company certainly doesn't. When you've got near monopoly and have your fingers in every single aspect of the internet, you don't need to care about fucking your customers over. I cannot legitimately name a single google product that I, or anyone I know, likes or wants to use. We just don't have a choice because of your market dominance.

ro_bit 5y ago
> You googlers can't even disdain yourselves to talk to other workers at the company who are in a caste lower than you. We must know different googlers then. It's good to avoid painting a group with the same brush
colechristensen 5y ago
Maps, Mail, Drive, Scholar, and Search are all the best or near the best available. That doesn’t mean I like every one of them or I wouldn’t prefer others, but as far as I can tell the competition doesn’t exist that works better. GCP and Pixel phones are a toss-up between them and competitors. It isn’t market dominance, nobody has made anything better.
- astrange 5y ago
  Search is famously kind of bad the last few years, but even Maps isn’t that great. (Data errors I’ve seen this week: the aerial imagery over Brisbane Australia is from ~2010 but labeled 2021, the coastline near Barentsburg in Svalbard is wrong and doesn’t match any other map.)

agwa 5y ago

> a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust Is that normal at Google? Making people feel bad for an outage doesn't seem consistent with the "blameless postmortem" culture promoted in the SRE book[1]. [1] https://sre.google/sre-book/postmortem-culture/ https://sre.google/sre-book/postmortem-culture/

fires10 5y ago
I don't think Google really cares about listening to their users. I have spent more than 6 hours trying to get simple warranty issues resolved. I wish they had to feel the pain of their actions and decisions.
UncleMeat 5y ago
"Blameless Postmortem" does not mean "No Consequences", even if people often want to interpret it that way. If an organization determines that a disconnect between ground work and a customer's experience is a contributing factor to poor decision making then they might conclude that making engineers more emotionally invested in their customers could be a viable path forward.
- bob1029 5y ago
  Relentless customer service is never going to screw you over in my experience... It pains me that we have to constantly play these games of abstraction between engineer and customer. You are presumably working a job which involves some business and some customer. It is not a fucking daycare. If any of my customers are pissed about their experience, I want to be on the phone with them as soon as humanly possible and I want to hear it myself. Yes, it is a dreadful experience to get bitched at, but it also sharpens your focus like you wouldn't believe when you can't just throw a problem to the guy behind you. By all means, put the support/enhancement requests through a separate channel+buffer so everyone can actually get work done during the day. But, at no point should an engineer ever be allowed to feel like they don't have to answer to some customer. If you are terrified a junior dev is going to say a naughty phrase to a VIP, then invent an internal customer for them to answer to, and diligently proxy the end customer's sentiment for the engineer's benefit.

spaetzleesser 5y ago

“ mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust “ The same leadership that demanded tighter and tighter deadlines and discouraged thinking things through?

grumple 5y ago

The most remarkable thing about this is learning that anyone at Google read an email from a customer. Given the automated responses to complaints of account shutdowns, or complaints about app store rejections, etc, this is pretty surprising.

lvs 5y ago
I'd love to get a read receipt each time someone at Google has actually read my feedback. Then it might be possible to determine whether I'm just shaking my fists at the heavens or not.