3 ms·

This is a cool feature. However, for me it is helpful to know how often GCE tends to have a zone fail, but not the whole region. Personally, I've been using

This is a cool feature. However, for me it is helpful to know how often GCE tends to have a zone fail, but not the whole region. Personally, I've been using GCP to run cocalc.com since 2014. In the last year I remember two significant outages to our site, which were 100% the fault of Google: (1) Last week, the GCE network went down completely for over an hour -- this killed the entire region (not just zone!) where cocalc is deployed -- see ://status.cloud.google.com/incident/cloud-networking/18010 (2) Last year, the GCE network went down completely for the entire world (!), and again this made cocalc not work. In both cases, when the outage happened, having cocalc be hosted in multiple zones (but one regions, or in (2) one cloud), would not have been enough. I haven't had to deal with any other significant GCE outages that I can remember that weren't at least partly my fault. For what it is worth, I used to host cocalc both on premise and on GCE, but can no longer afford to do that.

jkaplowitz 8y ago
Last week's outage does look bad, and clearly would have severely impacted certain dynamic patterns of scaling, but "went down completely" does not match what's documented on that link: it prevented the creation of new GCE instances in us-east4 which required allocating/attaching new external IP addresses. Of particular relevance to this thread, if you had a GKE cluster spun up in that region, that cluster would have continued unaffected based on that description. GCP does have outages just like AWS does, but in recent years the impact is usually something constrained to certain products and use cases. (Disclosure: I worked for GCP 2013-2015 but haven't worked for Google since then.)
alexbeloi 8y ago
Somebody on HN once mentioned maintaining duplicate/redundant deployment scripts for two different cloud providers. If one goes down you 'just' redeploy onto the other cloud provider. So you would be down for only as long as it takes to run a deployment and update dns. Minimal additional recurring costs: * duplicate storage (e.g. GCS and S3) * engineering cost of maintaining/testing multiple deployments (non-trivial) This feels very much like a five 9's level optimization that I would not find easy to justify. It might also be worth investigating how Netflix handles this issue.

jkaplowitz 8y ago

Last week's outage does look bad, and clearly would have severely impacted certain dynamic patterns of scaling, but "went down completely" does not match what's documented on that link: it prevented the creation of new GCE instances in us-east4 which required allocating/attaching new external IP addresses. Of particular relevance to this thread, if you had a GKE cluster spun up in that region, that cluster would have continued unaffected based on that description. GCP does have outages just like AWS does, but in recent years the impact is usually something constrained to certain products and use cases. (Disclosure: I worked for GCP 2013-2015 but haven't worked for Google since then.)

alexbeloi 8y ago

Somebody on HN once mentioned maintaining duplicate/redundant deployment scripts for two different cloud providers. If one goes down you 'just' redeploy onto the other cloud provider. So you would be down for only as long as it takes to run a deployment and update dns. Minimal additional recurring costs: * duplicate storage (e.g. GCS and S3) * engineering cost of maintaining/testing multiple deployments (non-trivial) This feels very much like a five 9's level optimization that I would not find easy to justify. It might also be worth investigating how Netflix handles this issue.