4 ms·

At the mean time, there are 7 GCP services currently under outage. --A frustrated engineer

swuecho 7y ago
I feel better, because I have only one. wonder why gcloud is so bad? not even as good as digital ocean from a user perspective.
- singron 7y ago
  Digital ocean has ~300 employees (circa 2017 according to wikipedia). Google has far more than that working on Cloud. Google Cloud is also largely layered on top of their existing internal infrastructure. Google cloud also has more services, more regions, and more zones. My opinion is that there is just way more that can go wrong.
- verdverm 7y ago
  I move people from AWS to GCP for cost, simplicity, and developer experience. What do you find so bad about GCP?
  - deleted 7y ago
    [deleted]
verdverm 7y ago
To be fair, these were not outages, rather degraded services where a feature or two was not working. None of my applications had any errors, nor were my users impacted. You can find the details here: https://status.cloud.google.com/ https://status.cloud.google.com/
- broth 7y ago
  You also have to account for the incidents that customers face that don't show up in the status dashboard until a certain influx of support tickets raise the flag. For example, right now we're facing an issue where one of our GCP HTTPS load balancers is suddenly failing to connect to any backend service for reasons unknown. Not sure if there was any correlation to the eight reported service disruptions on the status dashboard.
  - verdverm 7y ago
    Yeah, there are still per customer issues that don't make the status page. GCP does a good job with their RCAs, in-depth with future mitigation strategies. They are also really good about keeping the status updated. I had an experience on both GCP and AWS of disks failing. For AWS we had to manually reset the machine, for GCP, I only know because I looked at the audit logs. GCP auto-migrates on hardware issues.
  - FrankPetrilli 7y ago
    Sounds like something GCP support would love to help you troubleshoot. The disruptions I'm aware of have no correlation to load balancer traffic paths. What's the actual reason listed for the 502s in Stackdriver? That's a great first step.
    - broth 7y ago
      All Stackdriver logs for the load balancer have the following for each request as the status details: failed_to_connect_to_backend We've tried the following with no luck: - Restarted GCE instances in backend service - Recreated the backend service - Recreated the HTTPS load balancer - Switched the balancing mode of the backend service to rate instead of utilization since monitoring was stating the backend service CPU utilization was 500% (another red flag) We are, however, able to directly connect to the GCE instances within the backend service via external IP. Have not heard much back from Google support.
      - broth 7y ago
        Update This ended up being due to SNI getting enabled on our HTTPS binding in IIS when using a Let's Encrypt cert. Apparently, the GCP HTTPS LB does not like SNI.
        gcardone_ 7y ago
        SNI is supported by GCP load balancers, see https://cloud.google.com/load-balancing/docs/ssl-certificates https://cloud.google.com/load-balancing/docs/ssl-certificate...

I feel better, because I have only one. wonder why gcloud is so bad? not even as good as digital ocean from a user perspective.

singron 7y ago
Digital ocean has ~300 employees (circa 2017 according to wikipedia). Google has far more than that working on Cloud. Google Cloud is also largely layered on top of their existing internal infrastructure. Google cloud also has more services, more regions, and more zones. My opinion is that there is just way more that can go wrong.
verdverm 7y ago
I move people from AWS to GCP for cost, simplicity, and developer experience. What do you find so bad about GCP?
- deleted 7y ago
  [deleted]

verdverm 7y ago

To be fair, these were not outages, rather degraded services where a feature or two was not working. None of my applications had any errors, nor were my users impacted. You can find the details here: https://status.cloud.google.com/ https://status.cloud.google.com/

broth 7y ago
You also have to account for the incidents that customers face that don't show up in the status dashboard until a certain influx of support tickets raise the flag. For example, right now we're facing an issue where one of our GCP HTTPS load balancers is suddenly failing to connect to any backend service for reasons unknown. Not sure if there was any correlation to the eight reported service disruptions on the status dashboard.
- verdverm 7y ago
  Yeah, there are still per customer issues that don't make the status page. GCP does a good job with their RCAs, in-depth with future mitigation strategies. They are also really good about keeping the status updated. I had an experience on both GCP and AWS of disks failing. For AWS we had to manually reset the machine, for GCP, I only know because I looked at the audit logs. GCP auto-migrates on hardware issues.
- FrankPetrilli 7y ago
  Sounds like something GCP support would love to help you troubleshoot. The disruptions I'm aware of have no correlation to load balancer traffic paths. What's the actual reason listed for the 502s in Stackdriver? That's a great first step.
  - broth 7y ago
    All Stackdriver logs for the load balancer have the following for each request as the status details: failed_to_connect_to_backend We've tried the following with no luck: - Restarted GCE instances in backend service - Recreated the backend service - Recreated the HTTPS load balancer - Switched the balancing mode of the backend service to rate instead of utilization since monitoring was stating the backend service CPU utilization was 500% (another red flag) We are, however, able to directly connect to the GCE instances within the backend service via external IP. Have not heard much back from Google support.
    - broth 7y ago
      Update This ended up being due to SNI getting enabled on our HTTPS binding in IIS when using a Let's Encrypt cert. Apparently, the GCP HTTPS LB does not like SNI.
      - gcardone_ 7y ago
        SNI is supported by GCP load balancers, see https://cloud.google.com/load-balancing/docs/ssl-certificates https://cloud.google.com/load-balancing/docs/ssl-certificate...