5 ms·

Question to Google employees: Why do you guys suffer global outages? This is your 2nd major global outage in less than 5 years. I’m sorry to say this, but it i

Question to Google employees: Why do you guys suffer global outages? This is your 2nd major global outage in less than 5 years. I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. I need to see some blog posts about how you guys are rethinking whatever design can lead to this - twice - or you are never getting a cent of money under my control. You have the most feature rich cloud (particularly your networking products), but down time like this is unacceptable.

toomuchtodo 8y ago
> I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. It's the opposite really: the expectation that service providers have no unexpected downtime is unrealistic, and it's strange this idea persists.
- Twirrim 8y ago
  (disclaimer: I work for another cloud provider) I agree, in general, outages are almost inevitable, but global outages shouldn't occur. It suggests at least a couple of things: 1) Bad software deployments, without proper validation. A message elsewhere in this post on HN suggest that problems have been occurring for at least 5 days, which makes me think this is the most likely situation. If this is the case, presumably given this is multiple days in to the issue, rolling back isn't an option. That doesn't say good things about their testing or deployment stories, and possibly their monitoring of the product? Even if the deployment validation processes failed to catch it, you'd really hope alarming would have caught it. or: 2) Regions aren't isolated from each other. Cross-region dependencies are bad, for all sorts of obvious reasons.
  - toomuchtodo 8y ago
    That shouldn't, but they do. S3 goes down [1]. The AWS global console goes down, right after Prime Day outages [2]. Lots of Google Cloud services go down [3, current thread]. Tens of Azure services go down hard [4]. Are software development and release processes improving to mitigate these outages? We don't know. You have to trust the marketing. Will regions ever be fully isolated? We don't know. Will AWS IAM and console ever not be global services? We don't know. Blah blah blah "We'll do better in the future". Right. Sure. Some service credits will get handed out and everyone will forget until the next outage. Disclaimer: Not a software engineer, but have worked in ops most of my career. You will have downtime, I assure you. It is unavoidable, even at global scale. You will never abstract and silo everything per region. [1] https://www.theregister.co.uk/2017/03/01/aws_s3_outage/ https://www.theregister.co.uk/2017/03/01/aws_s3_outage/ [2] https://www.cnbc.com/2018/07/16/aws-hits-snag-after-amazon-prime-day-downtime.html https://www.cnbc.com/2018/07/16/aws-hits-snag-after-amazon-p... [3] https://www.cnet.com/news/google-cloud-issues-causes-outages-for-snapchat-spotify-and-others/ https://www.cnet.com/news/google-cloud-issues-causes-outages... [4] https://www.datacenterknowledge.com/uptime/microsoft-blames-severe-weather-azure-cloud-outage https://www.datacenterknowledge.com/uptime/microsoft-blames-...
    - ignoramous 8y ago
      Can't speak for Google, but Facebook and Salesforce chose Cells for HA. http://highscalability.com/blog/2012/5/9/cell-architectures.html http://highscalability.com/blog/2012/5/9/cell-architectures....
      - toomuchtodo 8y ago
        Doesn't look like it was all that helpful to Facebook (as of 1542038976). Facebook.com errors out currently. > Facebook Platform Appears to be down > A check of https://developers.facebook.com/status/dashboard/ https://developers.facebook.com/status/dashboard/ returns an error and I'm unable to login with facebook to some of my mobile apps. https://news.ycombinator.com/item?id=18434262 https://news.ycombinator.com/item?id=18434262
    - doodliego 8y ago
      Look how frequent and detailed Amazon's update logs are in that first Register article. Multiple updates throughout the day going into some detail.
      - deleted 8y ago
        [deleted]
  - Latteland 8y ago
    When I was at google, the big outages were almost always bad routing to the service - it was never that the service couldn't handle the load, and bad service instances were kind of hidden. my service did have some problems on new releases, but because we had multiple instances we could just redirect traffic to the instances we hadn't updated, so they stayed up.
- manigandham 8y ago
  The major issue is that outages are global instead of regional, effectively making it impossible to design around using the typical region/zone redundancy.
- rodgerd 8y ago
  The pitch from cloud vendors always includes the idea that the cloud is more reliable than any in-house shop can achieve. So the expectation is set by the vendors.
- tw04 8y ago
  Because they sell themselves as being far more reliable than internal IT. If they weren't selling on uptime, people probably wouldn't be quite so critical of downtime.
  - toomuchtodo 8y ago
    As a technology practitioner, it is your failing if you believe them.
    - tw04 8y ago
      Let me know the next time you hear about the CIO of a fortune 500 asking his technology practitioners to validate what he read in Gartner and heard from Diane Greene.
      - toomuchtodo 8y ago
        My advice would be to find opportunities to get paid to tell people the right answer, not to implement the wrong answer against your better judgement. Hot job market right now, more jobs than talent, all that jazz. If you're stuck implementing a suboptimal solution, that's not your fault, and not the intent of my above comment.
        engineeringwoke 8y ago
        Lots of wisdom in this comment
tehlike 8y ago
2 outage in 5 years sounds pretty low, to be honest. Disclaimer: google employee in ads, who worked on many many fires throughout the years, but talking from my personal perspective and not from my employer. I am sure we are striving to have 0, but realistically, i have seen many that says things happen. Learn, and improve.
- noselasd 8y ago
  The issue people have with it is that it's global, not regional, indicating that there are dependencies in the entire architecture that people does not expect to be there.
  - blazespin 8y ago
    Yes, hello, canaries anyone
    - tehlike 8y ago
      Plenty of bugs happen despite canaries.
      - hexchain 8y ago
        Just like YouTube a few weeks ago?
        tehlike 8y ago
        Yt, ads, a bunch of services. Sure.
  - deleted 8y ago
    [deleted]
  - B-Con 8y ago
    There are many other possible causes for global outages, that specific one is not high on my list of likely culprits.
_wmd 8y ago
It is a natural effect of building massive yet flat homogeneous systems, failures tend to be greatly amplified. Most of what you can read of Google's approach will teach you their ideal computing environment is a single planetary resource, pushing any natural segmentation and partitioning out of view.
hartem_ 8y ago
I’d be curious to know what alternatives are you considering at this point?
- hacknat 8y ago
  Azure and AWS.
origami777 8y ago
Most feature rich cloud? I think that title belongs to AWS.
- jkaplowitz 8y ago
  You're right in terms of breadth officially covered. But if you look at the features where they both officially have support, there are many examples where the GCP version is more reliable and usable than the AWS version. Even GKE is an example of this, despite the outage in node pool creation that we're discussing here. Way better than EKS. (Disclosure: I worked for Google, including GCP, for a few years ending in 2015. I don't work or speak for them now and have no inside info on this outage.)
  - illumin8 8y ago
    I think you're going to have to back up a claim like this with some facts. GKE being the exception, since it was launched a couple years before EKS. AWS clearly has way more services, and the features are way deeper than GCP. Just compare virtual machines and managed databases, AWS has about 2-3x more types of VMs (VMs with more than 4TB of RAM, FPGAs, AMD Epyc, etc.), and in databases, more than just MySQL and PostgreSQL. When you start looking at features you get features that you just can't get in GCP, like 16 read-replicas, point in time recovery, backtrack, etc. Disclaimer: I work for AWS but my opinions are my own.
    - jkaplowitz 8y ago
      Each platform has features the other platform doesn't, even though AWS has more. Some of GCP's unique compelling features include live VM migration that makes it less relevant when a host has to reboot, the new life that has recently been put into Google App Engine (both flexible environment and the second generation standard environment runtimes), the global load balancer with a single IP and no pre-warming, and Cloud Spanner. In terms of feature coverage breadth I started my previous comment by agreeing that AWS was ahead, and I still reaffirm that. But if you randomly select a feature that they both have to a level which purports to meet a given customer requirement, the GCP offering will frequently have advantages over the AWS equivalent. Examples besides GKE: BigQuery is better regarded than Amazon Redshift, with less maintenance hassle. And EC2 instance, disk, and network performance is way more variable than GCE which generally delivers what it promises. One bit of praise for AWS: when Amazon does document something, the doc is easier to find and understand, and one is less likely to find something out of date in a way that doesn't work. But GCP is more likely to have documented the thing in the first place, especially in the case of system-imposed limits. To be clear, I want there to be three or four competitive and widely used cloud options. I just think GCP is now often the best of the major players in the cases where its scope meets customer needs.
asdfasgasdgasdg 8y ago
Not to minimize here (well, yes, a little), but this was a UI-only outage, from what I can tell. You could still create the pools from the command-line. It doesn't seem unreasonable to have a single, global UI server, as long as the API gateway is distributed and not subject to global outages.
- rlancer 8y ago
  Was certainly not UI only
  - asdfasgasdgasdg 8y ago
    OK. Perhaps I misunderstood. In the status page, it says: Affected customers can use gcloud command [1] in order to create new Node Pools. [1] https://cloud.google.com/sdk/gcloud/reference/container/node-pools/create https://cloud.google.com/sdk/gcloud/reference/container/node... That led me to believe that only the web UI was affected.
tomcam 8y ago
I believe this is a fair question. I’d really like to understand what Google thinks about this.
chumboslice 8y ago
2 outages in 5 years. 5. Years. Nothing to see here, move along.
- talonx 8y ago
  2 "global" outages. If it had been limited to a service, or a region, there would be nothing to see.
  - dvdgsng 8y ago
    it was limited to GKE, wasn't it?
    - talonx 8y ago
      Global here refers to the geographical spread of the service, GKE in this case, measured in regions, not the number of services. Edit: I saw your point a bit late. It was limited to GKE, which makes my initial comment about "service" incorrect, and it was global, which keeps my comment about "region" correct. On a related note, an SRE from GKE posted on Slack that GCE was out of resources and so GKE faced resource exhaustion as well [1][2] - so it _might_ have been a multi-service outage. 1.https://googlecloud-community.slack.com/messages/C0B9GKTKJ/convo/C0B9GKTKJ-1541981221.238400/ https://googlecloud-community.slack.com/messages/C0B9GKTKJ/c... 2. https://googlecloud-community.slack.com/archives/C0B9GKTKJ/p1541981256238500 https://googlecloud-community.slack.com/archives/C0B9GKTKJ/p...
illumin8 8y ago
Google has a global SDN (software-defined network) that gives them some unique and beneficial capabilities, like being able to onboard traffic in the closest CDN POP and letting it ride over the Google backbone to the region your systems are running in. The problem is that running a global SDN like this means if you do something wrong, you can have outages that impact multiple regions simultaneously. This is why AWS has strict regional isolation and will never create cross-region dependencies (outside of some truly global services like IAM and Route 53 that have sufficient redundancy that they should (hopefully) never go down). Disclaimer: I work for AWS, but my opinions are my own.

toomuchtodo 8y ago

> I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. It's the opposite really: the expectation that service providers have no unexpected downtime is unrealistic, and it's strange this idea persists.

Twirrim 8y ago
(disclaimer: I work for another cloud provider) I agree, in general, outages are almost inevitable, but global outages shouldn't occur. It suggests at least a couple of things: 1) Bad software deployments, without proper validation. A message elsewhere in this post on HN suggest that problems have been occurring for at least 5 days, which makes me think this is the most likely situation. If this is the case, presumably given this is multiple days in to the issue, rolling back isn't an option. That doesn't say good things about their testing or deployment stories, and possibly their monitoring of the product? Even if the deployment validation processes failed to catch it, you'd really hope alarming would have caught it. or: 2) Regions aren't isolated from each other. Cross-region dependencies are bad, for all sorts of obvious reasons.
- toomuchtodo 8y ago
  That shouldn't, but they do. S3 goes down [1]. The AWS global console goes down, right after Prime Day outages [2]. Lots of Google Cloud services go down [3, current thread]. Tens of Azure services go down hard [4]. Are software development and release processes improving to mitigate these outages? We don't know. You have to trust the marketing. Will regions ever be fully isolated? We don't know. Will AWS IAM and console ever not be global services? We don't know. Blah blah blah "We'll do better in the future". Right. Sure. Some service credits will get handed out and everyone will forget until the next outage. Disclaimer: Not a software engineer, but have worked in ops most of my career. You will have downtime, I assure you. It is unavoidable, even at global scale. You will never abstract and silo everything per region. [1] https://www.theregister.co.uk/2017/03/01/aws_s3_outage/ https://www.theregister.co.uk/2017/03/01/aws_s3_outage/ [2] https://www.cnbc.com/2018/07/16/aws-hits-snag-after-amazon-prime-day-downtime.html https://www.cnbc.com/2018/07/16/aws-hits-snag-after-amazon-p... [3] https://www.cnet.com/news/google-cloud-issues-causes-outages-for-snapchat-spotify-and-others/ https://www.cnet.com/news/google-cloud-issues-causes-outages... [4] https://www.datacenterknowledge.com/uptime/microsoft-blames-severe-weather-azure-cloud-outage https://www.datacenterknowledge.com/uptime/microsoft-blames-...
  - ignoramous 8y ago
    Can't speak for Google, but Facebook and Salesforce chose Cells for HA. http://highscalability.com/blog/2012/5/9/cell-architectures.html http://highscalability.com/blog/2012/5/9/cell-architectures....
    - toomuchtodo 8y ago
      Doesn't look like it was all that helpful to Facebook (as of 1542038976). Facebook.com errors out currently. > Facebook Platform Appears to be down > A check of https://developers.facebook.com/status/dashboard/ https://developers.facebook.com/status/dashboard/ returns an error and I'm unable to login with facebook to some of my mobile apps. https://news.ycombinator.com/item?id=18434262 https://news.ycombinator.com/item?id=18434262
  - doodliego 8y ago
    Look how frequent and detailed Amazon's update logs are in that first Register article. Multiple updates throughout the day going into some detail.
    - deleted 8y ago
      [deleted]
- Latteland 8y ago
  When I was at google, the big outages were almost always bad routing to the service - it was never that the service couldn't handle the load, and bad service instances were kind of hidden. my service did have some problems on new releases, but because we had multiple instances we could just redirect traffic to the instances we hadn't updated, so they stayed up.
manigandham 8y ago
The major issue is that outages are global instead of regional, effectively making it impossible to design around using the typical region/zone redundancy.
rodgerd 8y ago
The pitch from cloud vendors always includes the idea that the cloud is more reliable than any in-house shop can achieve. So the expectation is set by the vendors.
tw04 8y ago
Because they sell themselves as being far more reliable than internal IT. If they weren't selling on uptime, people probably wouldn't be quite so critical of downtime.
- toomuchtodo 8y ago
  As a technology practitioner, it is your failing if you believe them.
  - tw04 8y ago
    Let me know the next time you hear about the CIO of a fortune 500 asking his technology practitioners to validate what he read in Gartner and heard from Diane Greene.
    - toomuchtodo 8y ago
      My advice would be to find opportunities to get paid to tell people the right answer, not to implement the wrong answer against your better judgement. Hot job market right now, more jobs than talent, all that jazz. If you're stuck implementing a suboptimal solution, that's not your fault, and not the intent of my above comment.
      - engineeringwoke 8y ago
        Lots of wisdom in this comment

tehlike 8y ago

2 outage in 5 years sounds pretty low, to be honest. Disclaimer: google employee in ads, who worked on many many fires throughout the years, but talking from my personal perspective and not from my employer. I am sure we are striving to have 0, but realistically, i have seen many that says things happen. Learn, and improve.

noselasd 8y ago
The issue people have with it is that it's global, not regional, indicating that there are dependencies in the entire architecture that people does not expect to be there.
- blazespin 8y ago
  Yes, hello, canaries anyone
  - tehlike 8y ago
    Plenty of bugs happen despite canaries.
    - hexchain 8y ago
      Just like YouTube a few weeks ago?
      - tehlike 8y ago
        Yt, ads, a bunch of services. Sure.
- deleted 8y ago
  [deleted]
- B-Con 8y ago
  There are many other possible causes for global outages, that specific one is not high on my list of likely culprits.

_wmd 8y ago

It is a natural effect of building massive yet flat homogeneous systems, failures tend to be greatly amplified. Most of what you can read of Google's approach will teach you their ideal computing environment is a single planetary resource, pushing any natural segmentation and partitioning out of view.

hartem_ 8y ago

I’d be curious to know what alternatives are you considering at this point?

hacknat 8y ago
Azure and AWS.

origami777 8y ago

Most feature rich cloud? I think that title belongs to AWS.

jkaplowitz 8y ago
You're right in terms of breadth officially covered. But if you look at the features where they both officially have support, there are many examples where the GCP version is more reliable and usable than the AWS version. Even GKE is an example of this, despite the outage in node pool creation that we're discussing here. Way better than EKS. (Disclosure: I worked for Google, including GCP, for a few years ending in 2015. I don't work or speak for them now and have no inside info on this outage.)
- illumin8 8y ago
  I think you're going to have to back up a claim like this with some facts. GKE being the exception, since it was launched a couple years before EKS. AWS clearly has way more services, and the features are way deeper than GCP. Just compare virtual machines and managed databases, AWS has about 2-3x more types of VMs (VMs with more than 4TB of RAM, FPGAs, AMD Epyc, etc.), and in databases, more than just MySQL and PostgreSQL. When you start looking at features you get features that you just can't get in GCP, like 16 read-replicas, point in time recovery, backtrack, etc. Disclaimer: I work for AWS but my opinions are my own.
  - jkaplowitz 8y ago
    Each platform has features the other platform doesn't, even though AWS has more. Some of GCP's unique compelling features include live VM migration that makes it less relevant when a host has to reboot, the new life that has recently been put into Google App Engine (both flexible environment and the second generation standard environment runtimes), the global load balancer with a single IP and no pre-warming, and Cloud Spanner. In terms of feature coverage breadth I started my previous comment by agreeing that AWS was ahead, and I still reaffirm that. But if you randomly select a feature that they both have to a level which purports to meet a given customer requirement, the GCP offering will frequently have advantages over the AWS equivalent. Examples besides GKE: BigQuery is better regarded than Amazon Redshift, with less maintenance hassle. And EC2 instance, disk, and network performance is way more variable than GCE which generally delivers what it promises. One bit of praise for AWS: when Amazon does document something, the doc is easier to find and understand, and one is less likely to find something out of date in a way that doesn't work. But GCP is more likely to have documented the thing in the first place, especially in the case of system-imposed limits. To be clear, I want there to be three or four competitive and widely used cloud options. I just think GCP is now often the best of the major players in the cases where its scope meets customer needs.

asdfasgasdgasdg 8y ago

Not to minimize here (well, yes, a little), but this was a UI-only outage, from what I can tell. You could still create the pools from the command-line. It doesn't seem unreasonable to have a single, global UI server, as long as the API gateway is distributed and not subject to global outages.

rlancer 8y ago
Was certainly not UI only
- asdfasgasdgasdg 8y ago
  OK. Perhaps I misunderstood. In the status page, it says: Affected customers can use gcloud command [1] in order to create new Node Pools. [1] https://cloud.google.com/sdk/gcloud/reference/container/node-pools/create https://cloud.google.com/sdk/gcloud/reference/container/node... That led me to believe that only the web UI was affected.

tomcam 8y ago

I believe this is a fair question. I’d really like to understand what Google thinks about this.

chumboslice 8y ago

2 outages in 5 years. 5. Years. Nothing to see here, move along.

talonx 8y ago
2 "global" outages. If it had been limited to a service, or a region, there would be nothing to see.
- dvdgsng 8y ago
  it was limited to GKE, wasn't it?
  - talonx 8y ago
    Global here refers to the geographical spread of the service, GKE in this case, measured in regions, not the number of services. Edit: I saw your point a bit late. It was limited to GKE, which makes my initial comment about "service" incorrect, and it was global, which keeps my comment about "region" correct. On a related note, an SRE from GKE posted on Slack that GCE was out of resources and so GKE faced resource exhaustion as well [1][2] - so it _might_ have been a multi-service outage. 1.https://googlecloud-community.slack.com/messages/C0B9GKTKJ/convo/C0B9GKTKJ-1541981221.238400/ https://googlecloud-community.slack.com/messages/C0B9GKTKJ/c... 2. https://googlecloud-community.slack.com/archives/C0B9GKTKJ/p1541981256238500 https://googlecloud-community.slack.com/archives/C0B9GKTKJ/p...

illumin8 8y ago

Google has a global SDN (software-defined network) that gives them some unique and beneficial capabilities, like being able to onboard traffic in the closest CDN POP and letting it ride over the Google backbone to the region your systems are running in. The problem is that running a global SDN like this means if you do something wrong, you can have outages that impact multiple regions simultaneously. This is why AWS has strict regional isolation and will never create cross-region dependencies (outside of some truly global services like IAM and Route 53 that have sufficient redundancy that they should (hopefully) never go down). Disclaimer: I work for AWS, but my opinions are my own.