4 ms·

You also have to account for the incidents that customers face that don't show up in the status dashboard until a certain influx of support tickets raise the fl

by broth 7y ago

You also have to account for the incidents that customers face that don't show up in the status dashboard until a certain influx of support tickets raise the flag. For example, right now we're facing an issue where one of our GCP HTTPS load balancers is suddenly failing to connect to any backend service for reasons unknown. Not sure if there was any correlation to the eight reported service disruptions on the status dashboard.

verdverm 7y ago
Yeah, there are still per customer issues that don't make the status page. GCP does a good job with their RCAs, in-depth with future mitigation strategies. They are also really good about keeping the status updated. I had an experience on both GCP and AWS of disks failing. For AWS we had to manually reset the machine, for GCP, I only know because I looked at the audit logs. GCP auto-migrates on hardware issues.
FrankPetrilli 7y ago
Sounds like something GCP support would love to help you troubleshoot. The disruptions I'm aware of have no correlation to load balancer traffic paths. What's the actual reason listed for the 502s in Stackdriver? That's a great first step.
- broth 7y ago
  All Stackdriver logs for the load balancer have the following for each request as the status details: failed_to_connect_to_backend We've tried the following with no luck: - Restarted GCE instances in backend service - Recreated the backend service - Recreated the HTTPS load balancer - Switched the balancing mode of the backend service to rate instead of utilization since monitoring was stating the backend service CPU utilization was 500% (another red flag) We are, however, able to directly connect to the GCE instances within the backend service via external IP. Have not heard much back from Google support.
  - broth 7y ago
    Update This ended up being due to SNI getting enabled on our HTTPS binding in IIS when using a Let's Encrypt cert. Apparently, the GCP HTTPS LB does not like SNI.
    - gcardone_ 7y ago
      SNI is supported by GCP load balancers, see https://cloud.google.com/load-balancing/docs/ssl-certificates https://cloud.google.com/load-balancing/docs/ssl-certificate...

by broth 7y ago

verdverm 7y ago

Yeah, there are still per customer issues that don't make the status page. GCP does a good job with their RCAs, in-depth with future mitigation strategies. They are also really good about keeping the status updated. I had an experience on both GCP and AWS of disks failing. For AWS we had to manually reset the machine, for GCP, I only know because I looked at the audit logs. GCP auto-migrates on hardware issues.

FrankPetrilli 7y ago

Sounds like something GCP support would love to help you troubleshoot. The disruptions I'm aware of have no correlation to load balancer traffic paths. What's the actual reason listed for the 502s in Stackdriver? That's a great first step.

broth 7y ago
All Stackdriver logs for the load balancer have the following for each request as the status details: failed_to_connect_to_backend We've tried the following with no luck: - Restarted GCE instances in backend service - Recreated the backend service - Recreated the HTTPS load balancer - Switched the balancing mode of the backend service to rate instead of utilization since monitoring was stating the backend service CPU utilization was 500% (another red flag) We are, however, able to directly connect to the GCE instances within the backend service via external IP. Have not heard much back from Google support.
- broth 7y ago
  Update This ended up being due to SNI getting enabled on our HTTPS binding in IIS when using a Let's Encrypt cert. Apparently, the GCP HTTPS LB does not like SNI.
  - gcardone_ 7y ago
    SNI is supported by GCP load balancers, see https://cloud.google.com/load-balancing/docs/ssl-certificates https://cloud.google.com/load-balancing/docs/ssl-certificate...