6 ms·

So the bigger news here is GCP had a global outage. How'd that happen? It mentions a pipeline deployment introduced a bug (https://status.cloud.google.com/inci

So the bigger news here is GCP had a global outage. How'd that happen? It mentions a pipeline deployment introduced a bug (https://status.cloud.google.com/incidents/LuGcJVjNTeC5Sb9pSJ9o https://status.cloud.google.com/incidents/LuGcJVjNTeC5Sb9pSJ...) but why weren't they doing staged deployments to region(s) at a time?

politelemon 5y ago
This is now 'par for the course' with GCP. A recent example, Firefox's Jan 2022 outage was due to an unannounced change deployed by GCP to their load balancers. https://hacks.mozilla.org/2022/02/retrospective-and-technical-details-on-the-recent-firefox-outage/ https://hacks.mozilla.org/2022/02/retrospective-and-technica... I've read several GCP related stories over the years so my opinion of them has fallen quite a bit. For some reason GCP does not get the flak they deserve, and their role in outages tend to be downplayed, while AWS and Azure are scrutinized and raked over the coals for smaller incidents.
- toomuchtodo 5y ago
  They don’t get flak because most folks are in AWS or Azure. You notice when AWS or Azure hiccups because of this. GCP, not so much (besides Spotify and Snap, although I believe Snap’s 5yr $2B GCP commit deal recently ended). https://info.flexera.com/CM-REPORT-State-of-the-Cloud https://info.flexera.com/CM-REPORT-State-of-the-Cloud https://thenewstack.io/long-running-study-finds-uptake-of-private-clouds-from-azure-aws-and-gcp/ https://thenewstack.io/long-running-study-finds-uptake-of-pr...
  - politelemon 5y ago
    Ah, good point about higher usage therefore flak, I hadn't considered that. Thanks.
    - MetricExpansion 5y ago
      Bjarne Stroustrup’s classic quote “There are only two kinds of languages: the ones people complain about and the ones nobody uses” certainly applies here.
  - luhn 5y ago
    Worth noting that Snap also has a commitment with AWS, they are not exclusively running on GCP.
    - northerdome 5y ago
      Also worth noting that AWS is not exactly the most stable either.
- charcircuit 5y ago
  >unannounced change deployed by GCP to their load balancers Google doesn't need to announce every change they make. It was Firefox's fault for not supporting the protocol correctly.
  - acdha 5y ago
    Most outages have more than one cause. It’s true that Firefox had a nasty bug but it’s also likely that they’d have resolved the problem faster had a clear notification that GCP had upgraded their infrastructure right before the outage started, especially if that had an easy rollback option.
  - politelemon 5y ago
    Changing a default behaviour, yes they do. If you feel they don't, then you might be happy with GCP's services. It does not inspire any confidence with me.
    - charcircuit 5y ago
      It was known in advance that HTTP/3 support would eventually be enabled for the load balancers. If you wanted to force support for it or force it to not support it there was an override setting. Mozilla did not override the behaviour meaning that they did not mind HTTP/3 to be advertised as being supported.
      - dastbe 5y ago
        so this is my "worked in aws" informed take, but you absolutely do not force default behavior change on anyone unless it is absolutely necessary, and advertising http/3 isn't necessary.
        e1g 5y ago
        AWS had a similar oopsie with their Load Balancer: introduced a new flag to strip some headers, enabled it by default for all existing public ALBs, and the effect was that all custom headers with an underscore in the name (“x-access_token”) started getting stripped. Took down a few APIs https://forums.aws.amazon.com/thread.jspa?messageID=923182 https://forums.aws.amazon.com/thread.jspa?messageID=923182
londons_explore 5y ago
Traffic Directors config is global. I assume it was some kind of config system bug.

politelemon 5y ago

This is now 'par for the course' with GCP. A recent example, Firefox's Jan 2022 outage was due to an unannounced change deployed by GCP to their load balancers. https://hacks.mozilla.org/2022/02/retrospective-and-technical-details-on-the-recent-firefox-outage/ https://hacks.mozilla.org/2022/02/retrospective-and-technica... I've read several GCP related stories over the years so my opinion of them has fallen quite a bit. For some reason GCP does not get the flak they deserve, and their role in outages tend to be downplayed, while AWS and Azure are scrutinized and raked over the coals for smaller incidents.

toomuchtodo 5y ago
They don’t get flak because most folks are in AWS or Azure. You notice when AWS or Azure hiccups because of this. GCP, not so much (besides Spotify and Snap, although I believe Snap’s 5yr $2B GCP commit deal recently ended). https://info.flexera.com/CM-REPORT-State-of-the-Cloud https://info.flexera.com/CM-REPORT-State-of-the-Cloud https://thenewstack.io/long-running-study-finds-uptake-of-private-clouds-from-azure-aws-and-gcp/ https://thenewstack.io/long-running-study-finds-uptake-of-pr...
- politelemon 5y ago
  Ah, good point about higher usage therefore flak, I hadn't considered that. Thanks.
  - MetricExpansion 5y ago
    Bjarne Stroustrup’s classic quote “There are only two kinds of languages: the ones people complain about and the ones nobody uses” certainly applies here.
- luhn 5y ago
  Worth noting that Snap also has a commitment with AWS, they are not exclusively running on GCP.
  - northerdome 5y ago
    Also worth noting that AWS is not exactly the most stable either.
charcircuit 5y ago
>unannounced change deployed by GCP to their load balancers Google doesn't need to announce every change they make. It was Firefox's fault for not supporting the protocol correctly.
- acdha 5y ago
  Most outages have more than one cause. It’s true that Firefox had a nasty bug but it’s also likely that they’d have resolved the problem faster had a clear notification that GCP had upgraded their infrastructure right before the outage started, especially if that had an easy rollback option.
- politelemon 5y ago
  Changing a default behaviour, yes they do. If you feel they don't, then you might be happy with GCP's services. It does not inspire any confidence with me.
  - charcircuit 5y ago
    It was known in advance that HTTP/3 support would eventually be enabled for the load balancers. If you wanted to force support for it or force it to not support it there was an override setting. Mozilla did not override the behaviour meaning that they did not mind HTTP/3 to be advertised as being supported.
    - dastbe 5y ago
      so this is my "worked in aws" informed take, but you absolutely do not force default behavior change on anyone unless it is absolutely necessary, and advertising http/3 isn't necessary.
      - e1g 5y ago
        AWS had a similar oopsie with their Load Balancer: introduced a new flag to strip some headers, enabled it by default for all existing public ALBs, and the effect was that all custom headers with an underscore in the name (“x-access_token”) started getting stripped. Took down a few APIs https://forums.aws.amazon.com/thread.jspa?messageID=923182 https://forums.aws.amazon.com/thread.jspa?messageID=923182

londons_explore 5y ago

Traffic Directors config is global. I assume it was some kind of config system bug.