7 ms·

Is it just me or is this lacking any acknowledgment of the impact it had on GCE and all of the third parties that were impacted by this. They make it sound lik

Is it just me or is this lacking any acknowledgment of the impact it had on GCE and all of the third parties that were impacted by this. They make it sound like a few people could not watch YouTube videos and even fewer people had some email disruption but this outage had a lot more impact than that. As just one example, a huge number of Shopify sites were impacted by this as were I am sure a number of other SaaS businesses that use GCE. I realize this is not the complete post mortem but it fails to even acknowledge the full impact of this disruption.

gamegod 7y ago
The network became congested, and our networking systems correctly triaged the traffic overload and dropped larger, less latency-sensitive traffic in order to preserve smaller latency-sensitive traffic flows... Overall, YouTube measured a 10% drop in global views during the incident... So what I'm hearing is that while Google Cloud Pub/Sub was down for hours, crippling my SaaS business, Google was prioritizing traffic to cat videos. It's good to know Google considers GCP traffic neither important, nor urgent.
- ldng 7y ago
  Cat videos would be generating more revenue?
- enneff 7y ago
  10% of YouTube views is a significant portion of ALL internet traffic. They weren’t prioritising YouTube, they used that as an example to describe the magnitude of the outage. It is easy to measure YouTube views, somewhat harder to measure the effect on a service as complex as GCP. I am sure they’ll have more to say about the effect on GCP services once they have more detailed analysis.
- jsty 7y ago
  Considering this was seemingly a mostly North America-affecting networking issue, and the 10% reduction in views was global, it doesn't sound like YouTube got much of a priority - quite a lot of the videos that were actually view-able in affected regions during the outage may simply have been served from edge caches. Disclaimer: no inside knowledge, the above is pure supposition
  - gamegod 7y ago
    You're right, but I wanted to point out their poor wording. They shouldn't downplay the huge impact on GCP customers in one paragraph, and then gloat about YouTube being fine in the next. It's undermines their GCP business in a big way too - It makes you think that if they had to choose, they would throw their GCP customers under the bus to preserve their own other services. The value proposition of GCP is greatly diminished then in comparison to a dedicated cloud provider like DigitalOcean, who has no other competing interests. This changed the way I view some of these cloud providers. eg. If Google had to prioritize ad network traffic over GCP, there's no question the ad network would get priority. But why not just go with a different provider who doesn't have to make that compromise?
    - ariwilson 7y ago
      Do you think Amazon wouldn't prioritize shopping over AWS? Or Microsoft with Xbox over Azure?
- ars 7y ago
  > Google was prioritizing traffic to cat videos. It was not. Youtube was unavailable to me, but gmail worked sporadically.
dabei 7y ago
Absolutely. I guess they want to avoid drawing attention to the full impact on businesses built on Google Cloud, before having a concerted effort across PR, Legal, Eng.
notatoad 7y ago
I understand that this isn't a postmortem, but it's posted on the GCP blog, not on blog.google so it feels like it should be aimed at GCP customers rather than YouTube viewers. If I were a GCP customers and I read this I'd be pretty mad. It doesn't just minimize the impact, it completely ignores that they have external customers.

gamegod 7y ago

The network became congested, and our networking systems correctly triaged the traffic overload and dropped larger, less latency-sensitive traffic in order to preserve smaller latency-sensitive traffic flows... Overall, YouTube measured a 10% drop in global views during the incident... So what I'm hearing is that while Google Cloud Pub/Sub was down for hours, crippling my SaaS business, Google was prioritizing traffic to cat videos. It's good to know Google considers GCP traffic neither important, nor urgent.

ldng 7y ago
Cat videos would be generating more revenue?
enneff 7y ago
10% of YouTube views is a significant portion of ALL internet traffic. They weren’t prioritising YouTube, they used that as an example to describe the magnitude of the outage. It is easy to measure YouTube views, somewhat harder to measure the effect on a service as complex as GCP. I am sure they’ll have more to say about the effect on GCP services once they have more detailed analysis.
jsty 7y ago
Considering this was seemingly a mostly North America-affecting networking issue, and the 10% reduction in views was global, it doesn't sound like YouTube got much of a priority - quite a lot of the videos that were actually view-able in affected regions during the outage may simply have been served from edge caches. Disclaimer: no inside knowledge, the above is pure supposition
- gamegod 7y ago
  You're right, but I wanted to point out their poor wording. They shouldn't downplay the huge impact on GCP customers in one paragraph, and then gloat about YouTube being fine in the next. It's undermines their GCP business in a big way too - It makes you think that if they had to choose, they would throw their GCP customers under the bus to preserve their own other services. The value proposition of GCP is greatly diminished then in comparison to a dedicated cloud provider like DigitalOcean, who has no other competing interests. This changed the way I view some of these cloud providers. eg. If Google had to prioritize ad network traffic over GCP, there's no question the ad network would get priority. But why not just go with a different provider who doesn't have to make that compromise?
  - ariwilson 7y ago
    Do you think Amazon wouldn't prioritize shopping over AWS? Or Microsoft with Xbox over Azure?
ars 7y ago
> Google was prioritizing traffic to cat videos. It was not. Youtube was unavailable to me, but gmail worked sporadically.

dabei 7y ago

Absolutely. I guess they want to avoid drawing attention to the full impact on businesses built on Google Cloud, before having a concerted effort across PR, Legal, Eng.

notatoad 7y ago

I understand that this isn't a postmortem, but it's posted on the GCP blog, not on blog.google so it feels like it should be aimed at GCP customers rather than YouTube viewers. If I were a GCP customers and I read this I'd be pretty mad. It doesn't just minimize the impact, it completely ignores that they have external customers.