5 ms·

Multi region support is interesting, but haven't most of GCPs outages affected all of their regions?

by plandis 10y ago

hurstdog 10y ago
Checking specifically for GCS at https://status.cloud.google.com/summary https://status.cloud.google.com/summary ... There are two outages there, one that only affected a few projects and one that affected only service in the central US (we have regions over much of the world). Anecdotally, from watching and working with other services internally I don't think most of our outages affect all regions. We actually spend a significant amount of engineering effort ensuring that we're as decoupled as possible. Disclaimer: I'm an Engineering Manager on Google Cloud Storage.
- jonathanoliver 10y ago
  Is there some documentation anywhere talks about how Google Cloud (any product) creates isolation between various regions while at the same time exposing a simple "regionless" programming model?
  - hurstdog 10y ago
    Probably the best reference is the SRE book, specifically the chapters on loadbalancing and distributed consensus protocols. Other than that, the general approach is to minimize global control planes and dependencies in our software stack. In the case of GCS, we do have a single namespace which means we need to look up the locations of data early in the request. Once we know the locations of data we can route the request to the right datacenter to serve it. That global location table is highly replicated and cached, of course. When outages happen, most are caused by changes to the stack, so we also are careful to roll out code or configuration slowly and carefully, slowly increasing the blast radius after it's been proven safe. For example rolling out new binaries first to a few canary instances in one zone, then to a few instances in many regions, then to a full region, then to the world, all spread over a few days. Disclaimer: I'm an Engineering Manager on Google Cloud Storage.
- brianwawok 10y ago
  On the flip side, many many outages lately have hit AN entire region, making the availability zone bit not so useful. For example the last US Central load balancer issue, it took out the entire central region for anyone using it. Not sure what happened there, but think a deploy should be rolled out to 1 availability zone at a time for hosted things (like load balancer)
jonathanoliver 10y ago
That's what I was thinking. The various outages I've seen (at least big enough to get mentioned here) seem to affect things globally. The AWS model is traditionally having things isolated between regions. While that makes programming slightly more complicated, for me it creates an added layer of safety.
boulos 10y ago
To be clear, GCS has had multi-region support (the "Standard" storage class), we're just making it explicit in the name. To add to hurstdog's response, I'd say: No, most outages for GCP have not been global. In particular, most Compute Engine outages are single zone and caused by either a networking configuration change (often affecting VM <=> Internet connectivity, like https://status.cloud.google.com/incident/compute/16004 https://status.cloud.google.com/incident/compute/16004) or sometimes a software rollout to our VM infrastructure. Much like with GCS (and any service at Google), the most common source of outages is a rollout. While we strive to offer zonal and regional services where appropriate, some like GCS and PubSub do have real value as "global" APIs. Trust me though, hurstdog spends a lot of his time struggling with that balance ;). Disclosure: I work on Google Cloud.

Multi region support is interesting, but haven't most of GCPs outages affected all of their regions?

by plandis 10y ago

Multi region support is interesting, but haven't most of GCPs outages affected all of their regions?

Checking specifically for GCS at https://status.cloud.google.com/summary https://status.cloud.google.com/summary ... There are two outages there, one that only affected a few projects and one that affected only service in the central US (we have regions over much of the world). Anecdotally, from watching and working with other services internally I don't think most of our outages affect all regions. We actually spend a significant amount of engineering effort ensuring that we're as decoupled as possible. Disclaimer: I'm an Engineering Manager on Google Cloud Storage.

jonathanoliver 10y ago
Is there some documentation anywhere talks about how Google Cloud (any product) creates isolation between various regions while at the same time exposing a simple "regionless" programming model?
- hurstdog 10y ago
  Probably the best reference is the SRE book, specifically the chapters on loadbalancing and distributed consensus protocols. Other than that, the general approach is to minimize global control planes and dependencies in our software stack. In the case of GCS, we do have a single namespace which means we need to look up the locations of data early in the request. Once we know the locations of data we can route the request to the right datacenter to serve it. That global location table is highly replicated and cached, of course. When outages happen, most are caused by changes to the stack, so we also are careful to roll out code or configuration slowly and carefully, slowly increasing the blast radius after it's been proven safe. For example rolling out new binaries first to a few canary instances in one zone, then to a few instances in many regions, then to a full region, then to the world, all spread over a few days. Disclaimer: I'm an Engineering Manager on Google Cloud Storage.
brianwawok 10y ago
On the flip side, many many outages lately have hit AN entire region, making the availability zone bit not so useful. For example the last US Central load balancer issue, it took out the entire central region for anyone using it. Not sure what happened there, but think a deploy should be rolled out to 1 availability zone at a time for hosted things (like load balancer)

jonathanoliver 10y ago

That's what I was thinking. The various outages I've seen (at least big enough to get mentioned here) seem to affect things globally. The AWS model is traditionally having things isolated between regions. While that makes programming slightly more complicated, for me it creates an added layer of safety.

boulos 10y ago

To be clear, GCS has had multi-region support (the "Standard" storage class), we're just making it explicit in the name. To add to hurstdog's response, I'd say: No, most outages for GCP have not been global. In particular, most Compute Engine outages are single zone and caused by either a networking configuration change (often affecting VM <=> Internet connectivity, like https://status.cloud.google.com/incident/compute/16004 https://status.cloud.google.com/incident/compute/16004) or sometimes a software rollout to our VM infrastructure. Much like with GCS (and any service at Google), the most common source of outages is a rollout. While we strive to offer zonal and regional services where appropriate, some like GCS and PubSub do have real value as "global" APIs. Trust me though, hurstdog spends a lot of his time struggling with that balance ;). Disclosure: I work on Google Cloud.